High network usage in GKE deployment

[lukedirtwalker]

Hi there,

We have a custom GKE deployment of 3 mimir instances running in monolith mode.
Each node is run in a different zone, with no specific multi-zone setups. We have a scraping rate of around 2MB/s but on the mimir nodes we see inter-zone traffic of around 60-70MB/s which is quite high and we are a bit surprised of this much usage.

Given our data rate is about 2MB/s I thought we would see maybe an additional 4MB/s to distribute to the other nodes. Additionally there might be some overhead from memberlist, but I really don't understand why it's that high.

Our configuration:

target: "all,alertmanager"
multitenancy_enabled: false
no_auth_tenant: anapaya
server:
    log_format: json
blocks_storage:
    backend: gcs
    gcs:
    bucket_name: mimir-storage-1
ruler:
    alertmanager_url: dnssrv+http://mimir.monitoring.svc.cluster.local:8080/alertmanager
ruler_storage:
    backend: local
    local:
    directory: /etc/ruler
alertmanager_storage:
    backend: local
    local:
    path: /etc/alertmanager
limits:
    ingestion_rate: 500000
    ingestion_burst_size: 500000
    max_global_series_per_user: 0
    max_global_series_per_metric: 0
activity_tracker:
    filepath: ""

Running with the config flags:

-config.file /etc/mimir/mimir.yaml -memberlist.join dnssrv+mimir.monitoring.svc.cluster.local:7946

We run mimir 2.2.0.

Is that network usage just normal? Or did we misconfigure something? Is there a way to reduce it?

Thanks for any hints.

[pstibrany]

Hello, you may want to enable grpc_compression for various grpc client used by Mimir. For example push request arrive to distributor as snappy-compressed protobuf messages, but once decoded by distributors, they are passed to ingesters uncompressed. Using -ingester.client.grpc-compression=snappy (or equivalent YAML config) will compress them again, at the cost of additional CPU usage by distributors and ingesters.

Note that there are multiple grpc_compression flags:

• -ingester.client.grpc-compression
• -query-frontend.grpc-client-config.grpc-compression
• -query-scheduler.grpc-client-config.grpc-compression
• -ruler.client.grpc-compression
• -alertmanager.alertmanager-client.grpc-compression
• -querier.frontend-client.grpc-compression

Depending on which client is configured.

If you search for grpc_compression at https://grafana.com/docs/mimir/v2.3.x/operators-guide/configure/reference-configuration-parameters/ you will find them.

[lukedirtwalker]

Thanks, this helped substantially. We are now at around 12MB/s, which is, while still being high, around the area that is acceptable.

Thanks a lot for the quick response.