Hierarchical Configuration Environments

[sbrugman]

Description

Imagine using Kedro for running pipelines both locally and remote, with mock and warehouse data.
Development takes place on mock data (shared between the environments), but computational configuration differs (e.g. spark local vs cluster mode). On the warehouse data, again the data definitions are shared, but some properties vary, e.g. database names.

The two shared configurations above make that the configuration could be far more concise if we would be able to inherit from multiple environments:

graph TD;
    base["Base (base)"];
    dev["Dev"];
    local["Local (default)"];
    remote["Remote"];
    cluster["Data Warehouse"];
    acc["Acceptance"];
    prd["Production"];
    
    base --> dev --> local;
    dev --> remote;
    base --> cluster --> acc;
    cluster --> prd;

Loading

The bottom four environments are used by the Kedro user.
Alternatively, we could reduce the number of environments:

graph TD;
    base["Base (base)"];
    dev["Dev + Remote"];
    local["Local (default)"];
    cluster["Data Warehouse + Acceptance"];
    prd["Production"];
    
    base --> dev --> local;
    base --> cluster --> prd;

Loading

This relies on the local environment overwriting the configuration set for remote in the combined dev + remote env.

Is this something that could be supported?

Context

As mentioned above, this would simplify the config and avoid duplication of entries.

Possible Implementation

Configuration the base environment takes place in settings.py. The base_env argument could be simply accepting a dict as well as a string to specify the base per environment.

Possible Alternatives

Something might be achieved through the advanced features of the OmegaConfigLoader that I am not aware of.

[noklam]

Sorry the graph doesn't rendered on my phone.

Does that mean local = base + local + dev
And prod = base + dev + remote?

[jccalvojackson]

this is a very common use case whenever you have a series independent binary (for example) options.
For example, you want to run something either locally or remotely and on top of that you want to either do a smoke or full run.
it would be great to just write a smoke and remote besides the base (local + full run) instead of having to write a remote_smoke, remote_full, local_smoke (on top of the base)

[astrojuanlu]

@MatthiasRoels mentioned the need for something like this, although excess of layers makes it complex

[astrojuanlu]

We were introduced to another group of users that want this cc @jonascristens

[jonascristens]

Including @philiproeleveld as well