Add emptyDir volume for /usr/share/opensearch/config to keystore init container to support readOnlyRootFilesystem: true

[DrissiReda]

What is the bug?

I get multiple errors linked to the read-only file system that I don't see how to circumvent

How can one reproduce the bug?

Deploy any cluster with securityContext.readOnlyRootFilesystem set to true.

What is the expected behavior?

I should be able to mount some paths as a R/W emptyDir where OpenSearch can create its files.

However I cannot, for example it tries to create a opensearch.keystore.tmp file in /usr/share/opensearch/config path, but if I mount an emptyDir to that path, I lose default configuration provided by the opensearch image

The complete relevant log:

Likely root cause: java.nio.file.FileSystemException: /usr/share/opensearch/config/opensearch.keystore.tmp: Read-only file system

What is your host/environment?

Kubernetes v1.29.10 rk2
Opensearch version 2.17.0
Operator version 2.7.0

Do you have any screenshots?

Not needed as the problem is straightforward.

Do you have any additional context?

Unrelated maybe but I was unable to change log4j.properties to print all logs to console. If opensearch doesn't need to write logs to file, it doesn't need R/W access to /usr/share/opensearch/logs.

[swoehrl-mw]

Hi @DrissiReda. Does the exception you report happen in the keystore init container or in the main opensearch container?
I would expect in the init container as there the keystore is modified.
If so, we might need to add an extra emptyDir volume just for the init container. I haven't tested it but I would expect there it should not be a problem to mount the volume to /usr/share/opensearch/config.

[DrissiReda]

Mounting manually an emptydir there works and that’s how I made it work. Then I removed the emptydir manually and registered the keystore as secret mounted to opensearch container. Didn’t get much issues afterwards. 

[DrissiReda]

@swoehrl-mw in case you consider doing it. Maybe also think about mentioning usage of keystore init container as a solution for readOnlyRootFilesystem users. And allow for empty keystore[0].keyMappings to avoid opensearch errors on unrecognized keys. This also works for the bootstrap pod by the way.

[swoehrl-mw]

Should be rather easy to implement, for anyone who wants to tackle that: Extend the cluster builder to add an emptydir volume and mount it to the keystore init container at /usr/share/opensearch/config.