Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Documentation feedback: /docs/sources/mimir/manage/run-production-environment/planning-capacity.md #10433

Open
bjorns163 opened this issue Jan 14, 2025 · 0 comments

Comments

@bjorns163
Copy link

I've been looking at your capacity planning documentation, but it not very clear to me how this works.

I've got a few tenants already sending data to my system. And when I add a new tenant I'm trying to see how munch extra resources i'll need.

For the already created tenants, I don't have their number of active series.
Can I fall back to what I see in the /distributor/all_user_stats page? And do I use the ingest rate value or the series value?

Image

Say it was series, it gives me a result

distributor: CPU=61.3Core(s) MEM=61.31GB

But this doesn't take into account how many replica's ill be running, and what would be the recommended limit value for a CPU?
based on the large values file it's set to 2 CPU this would mean I need 31 replica's?

Say it was rate values, it gives me a result of:

distributor: CPU=2.4Core(s) MEM=2.39GB

So two replicas would be enough?

Taking a look at the scaling dashboard
It's telling me I need 7 replicas based on cpu or 11 based on memory.

Image

Currently, I have running 5 replicas.


distributor:
  replicas: 5

  resources:
    limits:
      cpu: 2.5
      memory: 5.7Gi
    requests:
      cpu: 1.2
      memory: 2Gi

Image

Looking at their load, it doesn't seem like I need to scale up:

Image

To summaries my questions.

  • What are the recommended requests/limit values per replica. To determine the number of replicas.
  • How do I see the current sum(prometheus_tsdb_head_series) across all tenants. Is <mimir_domain>/distributor/all_user_stats a good source.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant