Support reading storage_config.object_store.named_stores from a separate file

[ringerc]

Is your feature request related to a problem? Please describe.

In a complex environment it may be necessary to configure Loki with swappable or pluggable storage backends.

While -config.expand-env=true provides a means of extracting credentials from the Loki configuration into workload environment variables for runtime substitution, it doesn't work well when the storage backend type may vary, and/or the set of keys and values that must be specified for that backend. See "Describe alternatives you've considered" below.

Describe the solution you'd like

Thanos supports a separate object store configuration file: https://thanos.io/v0.6/thanos/storage.md/#object-storage

It would make sense for Loki to support the same, especially since it's moving to adopt the Thanos object store client.

In a k8s environment this might be a separate secret, mounted into the workload filesystem.

For Loki it would make sense to do this for named storage configurations, and might look something like this:

common:
  storage:
    # requires -use-thanos-objstore CLI option; this could be supported for the old named stores too
    # if required, but why bother?
    object_storage:
      named_stores_config_file: /etc/loki/object-storage-config.yaml     #<--- new functionality, include file

schema_config:
  configs:
  - from: 2020-05-15
    store: tsdb
    object_store: loki-store     #<--- named store reference so it doesn't have to be updated for provider type
    schema: v13
    index:
      prefix: index_
      period: 24h

where /etc/loki/object-storage-config.yaml might be something like this

s3:
  # name of the object store; must match schema_configs.configs[].object_store
  loki-store:
    endpoint: MY_S3_ENDPOINT
    region: MY_AWS_REGION
    bucket_name: MY_BUCKET_NAME

The Secret would typically be bound as a k8s Secret and mounted into the workload as a file.

Describe alternatives you've considered

Currently the only means to do this are:

• use -config.expand-env=true and substitute a yaml blob with the required storage configuration as an env-var, typically bound from a separate k8s Secret key or similar; or
• append a long series of CLI arguments to explicitly supply each storage option required, e.g. -common.storage.object-store.s3.region=${S3_REGION} -common.storage.object-store.s3.secret-access-key=${S3_ACCESS_KEY} etc. Then (on k8s) bind the env-vars to the workload environment so they'll be substituted before pod execution.
• write a custom configuration generator (such as a k8s operator, or an initContainer to compose the config before starting Loki) to dynamically create a Loki configuration

None of these are very good.

Appending options on the CLI is undesirable because it exposes them in procfs, etc, and it's very cumbersome.

Adding an env-var to expand a yaml snippet of storage configuration means the yaml configuration for loki must be an invalid yaml document, breaking various checkers and linters. e.g.

common:
  storage:
  # requires -use-thanos-objstore
    object_storage:
      ${THANOS_STORAGE_YAML_SNIPPET}

The content of the yaml snippet defining the storage backend, credentials, etc must then also be pre-indented and cannot have a yaml --- leader. And the main Loki configuration is no longer a valid yaml document, so it cannot be parsed and checked, manipulated with yaml-aware tools, etc. It's generally difficult to work with and validate.

While a custom k8s operator etc is an option, it shouldn't really need to be the norm for something like object storage configuration. It's a big hammer and lots of extra moving parts for a small job.

For k8s an initContainer is probably the least-bad workaround, where the YAML configuration can be composed from snippets using a yaml-aware helper; even a yq command. But it really makes more sense to separate out the object storage configuration into a separate file, especially now that Loki supports named stores so the provider itself can be abstracted./

Additional context

An alternative might be to support a more general mechanism for configuration overlays in Loki, like the OpenTelemetry Collector's support for multiple --config arguments. This can become complicated in terms of the semantics for merging vs replacing though.