simple OOM Monitor - we gracefully quit before we hit our OOM limit #1215
Conversation
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
|
SERVER_OOM_ENABLED=true SERVER_OOM_THRESHOLD_BYTES=27574104 SERVER_OOM_SIGNAL="SIGTERM" SERVER_OOM_CHECK_INTERVAL="20s" task start-dev |
| default: | ||
| runtime.GC() | ||
| var memStats runtime.MemStats | ||
| runtime.ReadMemStats(&memStats) |
There was a problem hiding this comment.
This concerns me -- I don't think we can necessarily get a true estimate of memory usage from this. More more information: https://www.datadoghq.com/blog/go-memory-metrics/
Has there been any investigation into whether the memory reported by ReadMemStats matches the system memory (divided by two, according to the DataDog article) when we're running in a Docker image?
From what I understand, not only may the count not be accurate, but we also have to multiply threshold bytes by 2 to get the actual threshold.
There was a problem hiding this comment.
this is 100% a tuning thing, we need to experiment and find out what the correct limit is, memory is always very difficult to calculate on linux boxes, even top and the various memory tools do not give an accurate picture, shared libraries are shared etc. The strategy here is to set a conservative limit and increase it. What we have now is OOMs which are a disaster and very difficult to mitigate - all of our code can just stop running right in the middle of executing which is the worst possible failure. We are running redundant engines right? So restarting an engine should be a normal part of our operations and not lead to any downtime.
Description
What's Changed