Abnormal bandwidth usage of nats cluster service [v2.10.22 || v2.10.23]

[SucceedGetRich]

Observed behavior

I used AWS servers, using servers in different countries. There were servers in four countries in total, three servers in each country, and a total of 12 servers to build the nats cluster.The number of copies is 3, the message volume is not large, but the bandwidth usage is abnormal.

Expected behavior

There are more than thirty streams in total, and there are about twenty consumers every month. If twelve clusters are used as a cluster, the upstream and downstream bandwidth should not be so high.

Server and client version

Using the latest nats-server version

Host environment

No response

Steps to reproduce

No response

[wallyqs]

Is the topology 12 servers clustered together with routes over different regions? Do you see any readloop processing errors or Slow Consumer errors in the logs? Have you tried modifying the max_outstanding_catchup setting from JetStream to something lower max_outstanding_catchup=32mb?

[SucceedGetRich]

拓扑结构中 12 台服务器是否通过不同区域的路由聚集在一起？您是否在日志中看到任何 readloop 处理错误或 Slow Consumer 错误？您是否尝试过将设置max_outstanding_catchup从 JetStream 修改为更低的值max_outstanding_catchup=32mb？

Can a nats cluster specify a machine as a cluster?

[SucceedGetRich]

拓扑结构中 12 台服务器是否通过不同区域的路由聚集？您是否在日志中存在任何 readloop 处理错误或慢速消费者错误？您是否尝试过设置max_outstanding_catchup从 JetStream 修改互连的值max_outstanding_catchup=32mb？

max_outstanding_catchup defaults to 32mb

[wallyqs]

it might be helpful to share the output of nats server ls and nats server report js using the system account

[wallyqs]

One way this could happen is if many consumers tried to issue an uncapped Fetch with a large batch size not using MaxBytes when consuming the messages from the streams