Optimizing the comparison feature using Clickhouse

[alisman]

When we create a comparison session from a chart, we first need to group the sampleids by the data. Each chart type has it's own mechanism for creating those groups. In case of clinical data charts, it has to fetch all the clinical data for that type in order to make the groups. This is time consuming. All these various mechanisms need to be studied and optimized. (It occurs in the study view, NOT the newly opened comparison page).

In the new opened comparison page, we await the creation of the session in the opening page. When it resolves, we can then load the session in the new page. The main bottlenecks here are the sample endpoint and then the enrichments endpoint.

We have a prototype that uses Clickhouse to improve the enrichments endpoint by 10x.

Note that caching solves the clinical data and samples endpoint latency, at least for single study queries. The enrichments endpoint is the highest priority.