Create merged full grapher configs for explorer views in the chart_configs table

[danyx23]

Motivation

We are building two related things in the 2024.6 cycle: production ready multidimensional datapages and dynamic thumbnails for explorers/Mdims. In both of these projects, it will be very useful to be able to easily get the full grapher config for a particular view.

In the case of an explorer, this works only for grapher based or indicator based explorers, not CSV based explorers. But for those, we'll have to take into account not just the indicator level grapher config (this one we already have in the chart_configs table), but also the view modifications that the explorer config adds for this particular view.

Similarly, multidimensional indicators have different views that often map to a single indicator but can also map to multiple indicators. In the case of a single indicator, the indicator level grapher config is merged with view specific overrides from the mdim config. In the multidimensional indicator case, only the view specific overrides apply.

Solution

We want a "final package" of the resulting grapher config for every one of these views that should exist in the grapher_configs table and have a UUID.

We probably also want a relationship table between the explorers and the chart_configs tables and one between multi_dim_data_pages and chart_configs. Each of these should have one entry per view and link to the chart_config for that view. The patch config should contain everything that the explorer view config or the mdim view config add, the full one should combine the defaults, the indicator level full admin config (if it exist, or the indicator level full etl config otherwise) and the patch config of the view.

Both explorers and mdims should probably have a simplified view mapping (a json stucture that enumerates views by dimension values and maps them to a UUID). This json file should be stored in R2 by slug.

This would then allow us e.g. in the thumbnail renderer or in the code to embed an mdim view inside an article to just take the full url with query params, get the simplified view mapping and then just interpret the query params that are needed in there to find the UUID to retrieve. Then it can get the chart config by UUID and render the thumbnail or do whatever else it needs to do - but, crucially, it would not have to be able to understand the full explorer program with all it's override options or the mdim config in full.

[danyx23]

Some more thoughts on this:

• Data transforms are a bit of a headache. Explorers can do this but this means running the explorer code so that the transforms get applied. If we want this to be possible with just a grapher config then Grapher would have to be able to do this
• The columns section in explorers can manipulate the display settings of indicators which e.g. overrides the names of indicators used in the legend. Mdims currently can't do this but there was a request for this feature already. Supporting this is probably quite useful but again, lives outside of the grapher config

With the above caveats it may not be the best approach to implement this issue and instead bite the bullet of having to interpret the full explorer/mdim config to arrive at the specific view (instead of having a pre-baked single chart config that you can fetch by UUID). Needs some more discussion.

[danyx23]

@rakyi did the part of this that covered MDims. The part of this issue that touches explorers was done differently (by actually running explorers on CF workers). This means that there isn't a really clear motivation to still do what is described in this issue for explorers.

One possible benefit would be that the exact entities for every explorer view could be easily enumerated, something that the current explorer code that runs for indexing for Algolia doesn't do. This alone is not worth the hassle though (and what is described here would only work for a subset of explorers).