Excessive memory retention using either mimalloc v2 or v3 on macOS #1025
Comments
|
Yikes -- thanks for the report. Strange that it happens with both v2 and v3, and not on ubuntu. Thanks for the repo -- if I find time I will try it out and see. Can you try the following environment settings on the latest
Also, as an experiment, trying, |
|
@daanx Thanks for the quick reply! I've tested these options on my macOS. Nothing has been changed in the demo other than changing some value to make it rebuild indefinitely. Here's some stats I yanked off from With branch With branch With branch With branch With branch |
|
Good to see v3 does much better as v2; I guess because it doesn't occur on Linux it must be something system specific, like an allocation in a thread that is about to be terminated (reinitializing the heap and leave it orphaned). Not sure. I'll try to repro when I find some time. |
|
I tried to compile from your repo (awesome that you constructed this! maybe we can use it a standard benchmark in the future). Can you help? |
Turned out the rust lang team had changed this API in the latest nightly release. I've pushed a new edit to add a You can also check out the active rust toolchain in the repo directory with command |
|
Thanks -- I got it running locally; but it doesn't quire reproduce. I set it to 100 iterations using the latest When I set 
Here I see lots of low use pages (the red Maybe I need the larger workload that you observed of 800MiB+ -- let me know how to do that. ps. with no options on a release build (with latest |
|
I can also reproduce the problem by following the steps in this demo.
Result: |
Unfortunately, this is not expected. I forgot to mention to install node modules in the directory
The benchmark contains heavy operation of reading files from disk (in directory
We don't normally write log files. The only file writing operation is called at the end of each rebuild. This is driven by the same blocking thread of tokio. |
|
I got it running now -- it still uses much less memory but I can see it grow now, from about 240MiB to 480MiB after 100 iterations. One thing I noticed was that every iteration an extra thread(s) is created, it starts at 339 threads and increases to 526 threads after 100 iterations. I will test on Linux too to see if that happens there too. I guess that extra thread is the issue? edit: on linux the threads stay constant at 199 (but uses more memory and goes much faster?) Question: how can I configure the project to use the standard macOS allocator? |
I did some tests on my local machine and found out the thread count and the memory consumption are indeed related and intertwined. In my opinion, it's strongly linked to these two configurations passed to tokio-rs:
Tuning either of the option will lower the memory usage. For example, if I set Even for mimalloc v2.1.7, which consumes more memory than v3 (dev3-bin), memory usage is way better than it was: The result for mimalloc-v3 (dev3-bin): Memory still goes up pretty quick on larger workload though. To adjust these two options, you may go to let rt = tokio::runtime::Builder::new_multi_thread()
.enable_all()
.max_blocking_thread(8)
.thread_keep_alive(Duration::from_millis(1_000))
.build()
.unwrap();
IIRC, mine test on linux gets stable at around 550~MB. I think this is totally fine if it's not increasing.
Just comment out these two lines at the top of #[global_allocator]
static GLOBAL: MiMalloc = MiMalloc; |
|
Thanks! It is good to see that v3 does much better as it was essentially redesigned to deal with threadpools better. Maybe the creation of the new threads that I see on macOS is the root cause of the memory growth (where there as an increasing amount of threads that are never terminated -- maybe due to mimalloc? maybe not?). |
|
Thanks for the reply!
That's weird. If I set For each rebuild, it consumed and retained So... do you know if there's any recommend way to debug this? For example, checking out the thread of which piece of memory was being hold off from releasing. Or do you know if there's anything should be checked out for me locally? |
Rspack is using mimalloc for speeding up the memory allocation. It's is a tool for transforming and bundling basically JavaScript files. According to our test to resolve this issue, we found out, on macOS, there's some strange memory retention as rspack rebuilds. I tried to create a minimal reproducible demo but unluckily they all failed. So I have to put on my local testing demo for reproduction using Rust:
https://github.com/h-a-n-a/rspack-allocation-test
In this demo, an rspack compiler was created to compile JavaScript in 10000 directory. For each build and rebuild, Rspack would trigger tokio-rs to spawn(if not already spawned) a few green threads to drive asynchronous tasks. Then, Rspack would trigger a series of JavaScript module transformations, then optimizations. Finally, assets generated in each build or rebuild will be emitted to the dist file.
During the compilation, the initial memory on macOS would be around 600 MB, and after a few rebuilds, the memory will skyrocket to 1 GB and more. This does not happen on my ubuntu-22.04 or when I was using macOS's system allocator. This does happen on both mimalloc v2 and v3.
I've added some details to help reproduce the issue in the repo and will try my best to create a minimal reproducible demo. Please bear with me.
Looking forward to hearing from you.
Cheers!
The text was updated successfully, but these errors were encountered: