Panic runtime error: invalid memory address or nil pointer dereference

[matteotumiati]

I have a chartmuseum instance deployed into Azure App Service using Docker container (image v0.14.0).
The App Service is referencing a Blob Storage where all the charts are saved, however recently we started facing this error:

2022-03-07T08:16:58.299Z INFO  - Initiating warmup request to container ...chartmuseum-dev_0_e197abc6 for site ...chartmuseum-dev
2022-03-07T08:17:04.326906800Z 2022-03-07T08:17:04.326Z	�[31mERROR�[0m	Could not save entry in cache store	{"error": "read: connection reset by peer", "repo": ""}
2022-03-07T08:17:04.329260028Z panic: runtime error: invalid memory address or nil pointer dereference
2022-03-07T08:17:04.357637073Z [signal SIGSEGV: segmentation violation code=0x1 addr=0x30 pc=0x1aa7201]
2022-03-07T08:17:04.357660274Z
2022-03-07T08:17:04.357673374Z goroutine 1 [running]:
2022-03-07T08:17:04.357677274Z helm.sh/chartmuseum/pkg/chartmuseum/server/multitenant.(*MultiTenantServer).getIndexFile(0xc00001f6c0, 0xc00000da58, {0x0, 0xc00026ad00})
2022-03-07T08:17:04.357681374Z 	/go/src/github.com/helm/chartmuseum/pkg/chartmuseum/server/multitenant/index.go:43 +0x61
2022-03-07T08:17:04.357685674Z helm.sh/chartmuseum/pkg/chartmuseum/server/multitenant.(*MultiTenantServer).primeCache(0xc00001f6c0)
2022-03-07T08:17:04.357689574Z 	/go/src/github.com/helm/chartmuseum/pkg/chartmuseum/server/multitenant/cache.go:86 +0x58
2022-03-07T08:17:04.357693374Z helm.sh/chartmuseum/pkg/chartmuseum/server/multitenant.NewMultiTenantServer({0xc00045a1c8, 0xc00001c370, {0x2478b58, 0xc00000d1b8}, {0x245f3f0, 0xc00045a1e8}, 0x0, {0x0, 0x0}, {0x20103d3, ...}, ...})
2022-03-07T08:17:04.357697274Z 	/go/src/github.com/helm/chartmuseum/pkg/chartmuseum/server/multitenant/server.go:163 +0x3b6
2022-03-07T08:17:04.357700974Z helm.sh/chartmuseum/pkg/chartmuseum.NewServer({{0x2478b58, 0xc00000d1b8}, {0x245f3f0, 0xc00045a1e8}, 0x0, 0xc00045a1c8, {0x0, 0x0}, {0x0, 0x0}, ...})
2022-03-07T08:17:04.357704874Z 	/go/src/github.com/helm/chartmuseum/pkg/chartmuseum/server.go:122 +0x5b8
2022-03-07T08:17:04.357708474Z main.cliHandler(0xc0006a66e0)
2022-03-07T08:17:04.357712074Z 	/go/src/github.com/helm/chartmuseum/cmd/chartmuseum/main.go:120 +0xc15
2022-03-07T08:17:04.357715674Z github.com/urfave/cli.HandleAction({0x1c61f20, 0x2119728}, 0xc0002e4380)
2022-03-07T08:17:04.357719374Z 	/go/pkg/mod/github.com/urfave/[email protected]/app.go:526 +0x50
2022-03-07T08:17:04.357722974Z github.com/urfave/cli.(*App).Run(0xc0002e4380, {0xc00004c200, 0x1, 0x1})
2022-03-07T08:17:04.357726674Z 	/go/pkg/mod/github.com/urfave/[email protected]/app.go:286 +0x625
2022-03-07T08:17:04.357730274Z main.main()
2022-03-07T08:17:04.357733874Z 	/go/src/github.com/helm/chartmuseum/cmd/chartmuseum/main.go:53 +0x1b0
2022-03-07T08:17:05.482Z ERROR - Container ...chartmuseum-dev_0_e197abc6 for site ...chartmuseum-dev has exited, failing site start
2022-03-07T08:17:05.485Z ERROR - Container ...chartmuseum-dev_0_e197abc6 didn't respond to HTTP pings on port: 8080, failing site start. See container logs for debugging.
2022-03-07T08:17:05.492Z INFO  - Stopping site ...chartmuseum-dev because it failed during startup.

It means that up until few days ago everything was fine, I was able to publish/download charts, then something started to break unexpectedly and now I cannot even read whatever is there.

If I look at the index-cache.yaml file, it is empty. The site cannot load and I only see an error message coming from the server:

[cbuto]

👋 @matteotumiati can you share your ChartMuseum configuration (redacted if necessary)? Do you have Redis setup?

[matteotumiati]

These are all the environment variables set:

"ALLOW_OVERWRITE": true
"AZURE_STORAGE_ACCESS_KEY" : "redacted"
"AZURE_STORAGE_ACCOUNT": "redacted"
"BASIC_AUTH_PASS": "redacted"
"BASIC_AUTH_USER": "redacted"
"STORAGE": "microsoft"
"STORAGE_MICROSOFT_CONTAINER": "redacted"
"WEBSITES_PORT": "8080"

I don't have Redis unfortunately, but I tried attaching it during the weekend. I wasn't lucky enough to make it work though 😢

I tried other things in the meanwhile, like applying these settings:

"CACHE_INTERVAL": "1m"
"DEPTH": 0
"DISABLE_STATEFILES": true

But I have to be honest I don't know what these values mean. However with these settings and scaling the App Service Plan in Azure to a P3v2 instance, then it started working again.

Is it possible that when you have 10k or more charts, then you need to scale resources? Is it because everything is loaded in memory therefore I may need more RAM?

[cbuto]

Interesting, so the error you posted above only can occur when an external cache is configured (Redis), Is that error happening without Redis configured or is there another error you came across before configuring Redis?

Just based on the error before the panic, it looks like your ChartMuseum container was unable to connect to Redis for some reason:

{"error": "read: connection reset by peer", "repo": ""}

[matteotumiati]

I don't know if I have the logs before trying to configure Redis, unfortunately.
How do I know if the connection to Redis is working? Shall I see something into Redis itself?

[cbuto]

There should definitely be data stored in Redis if it’s configured.

You mentioned in the issue:

It means that up until few days ago everything was fine, I was able to publish/download charts, then something started to break unexpectedly and now I cannot even read whatever is there.

but it looks like the logs you provided are related to using Redis, did your configuration change to cause that error?

[matteotumiati]

That's correct.

Everything was working fine initially, but at some point something broke without any explanation because no-one touched that environment and our team was not able to access CM anymore.

We then tried to recover customizing some parameters and then by including Redis, all unsuccessful.

In the end I was able to set some other parameters (the ones I added above) and scaled to a bigger instance (with more memory/CPU) to make it work again. I have this feeling though that the configuration I applied or scaling is just a temporary workaround and not something solid I should consider for production.

[cbuto]

Ah thanks for the additional context!

You can read about CACHE_INTERVAL in the docs. Setting that option can definitely help with performance when you have a large number of charts.

DEPTH=0 is the default for that configuration so that won’t really have much affect in terms of performance. By disabling statefiles, you might get some performance gains but lose any index caching unless you are using Redis.

But without logs/debug logs it will be hard to pinpoint exactly what’s going on. As for the issues with Redis, I was able to sucessfully use redis locally. Did you have any networking contraints between chartmuseum and Redis? (policies, firewalls, security groups, etc)

[matteotumiati]

I do not, everything is deployed quite simply from Azure and I'm only using PaaS services.
I guess given Redis is not really supported in Azure then I'll have to investigate a bit more this part.

Do you mind if I keep this issue open while I try a bit more with Redis and see if it replicates again?

[cbuto]

@matteotumiati were you able to reproduce this issue?

[cbuto]

@matteotumiati any updates here?