ZFS 2.3.0 ignores zfs_arc_max, exhausts system memory #17052
Comments
|
@maru-sama The fact that ARC is allowed to use almost all of RAM in 2.3 is not a bug, but feature. Once there appear memory pressure from the kernel and other consumers, it should shrink. |
Hello, thanks for the reply. I removed my initial comment since the behaviour I described is standard (as you said) and the original poster is seeing a different issue. |
|
@brian-maloney As I see, most of your ARC size is reported as non-evictable metadata, which means something is actively referencing it. It might be metadata blocks backing dnodes, referenced by kernel via inodes. ZFS should start inodes pruning process for kernel once the percent of non-evictable metadata goes above the threshold and there is a need of eviction. So either the pruning does not start, or it does not work right, or I wonder if it just can't since those files are actually opened and kernel doesn't want them to go. |
|
@brian-maloney Do you use docker? |
I do have Docker running on the system, but the application that triggers the memory exhaustion (duplicacy) is not running in a container. So if the mere presence of containers on the system is enough to trigger the issue, then that might be of use, but most of the time (all except for 10-15 minutes per day) the ARC max is honored, it's only while duplicacy is running that I see the problem behavior. To @amotin's comment, it's likely that duplicacy does hold many files open while computing chunks to check the backup storage location for. I am not that familiar with the implementation. |
|
I did some testing with this. The dangerous interaction seems to occur only during the first phase of the backup operation, which is a depth-first listing of all files on the system (as seen in ListLocalFiles). I'm not sure what exactly about this implementation would be causing this behavior, but I did do a During this first phase, I am hopeful that these are enough clues to help someone more knowledgable than myself on ZFS internals and the 2.3.0 changes either suggest a code fix or a workaround. All of my attempts to tune so far have been completely ineffective. Thanks in advance for any advice you can provide! |
@amotin A quick test shows that, even on 2.3, setting |
|
@shodanshok Yes, |
|
@amotin :
Does that mean that the 50 % RAM limit for ARC that was in place for ZFS 2.2 and before does not apply anymore? Where can I read more about this change? I havent found a note in the change log. |
Right.
It was changed here: #15437 . It was discussed among developers and accompanied by several other changes to better coexist with Linux kernel, including some fixes to Linux kernel itself. |
Just to (hopefully) rule out any issues related to docker, I stopped the docker service (and all containers) and ran another backup. The same issue is present, so I think it's probably not docker-related. |
|
Yikes! Maybe FreeBSD does it better, but we all know that ZFS eviction can substantially lag memory demands on Linux. The linked discussion doesn't seem to reflect a level of consideration and conservatism befitting a system widely regarded for its reliability. It would be better to see more presentation of evidence than "things are probably better now", or at least a giant red flag in the release announcement that this change has been made. |
|
@ahesford "but we all know" is not an argument. We discussed it with many developers multiple times, and nobody could give good arguments why it should not work. But since the 50% limit was introduced so early in ZoL life time when things were definitely broken, nobody knew what to expect in practice. Now I can say that we have more than hundred thousands of Linux-based TrueNAS systems in a wild with this change, that work happily. I don't believe the problem reported here has anything to do with 50% change, since ARC buffers are just held by something (I suppose dnodes). If you experience some problems -- please report in separate issue. If just don't like it, you are welcome to set |
|
The inability to free memory on time, particular with VMs, was raised in the very PR that proposed this change, and the response was "it's probably better now". Filesystem behavior changes should always be extremely conservative, and "it's probably better" is anything but conservative. Again, at the very least, release notes ought to call aggressive behavior changes to user attention. Allowing the ARC to eat 95% of system RAM by default, when the very people discussing the change acknowledge that this has been known to cause problems in the past, is without question an aggressive behavior change. Whether the default change is a direct cause of this issue, or only exacerbates it because it allows the ARC to grow unreasonably large, is immaterial---this issue would not have had the reported impact without an unflagged change in default behavior. |
@ahesford - as the poster of the issue, I'm not sure that's the case. Before and after the change you are referring to, I had I feel like this issue is being hijacked by people who are concerned about the change to the default max size when from what I can see, the default max size isn't relevant to this issue at all. Maybe someone can explain how the default is relevant? It's not like I was seeing the ARC grow to 50% of my system's memory using 2.2.6 - during backups it stayed around 12.5% of my system's memory as mandated by my Thanks! |
|
Sorry, I only intended to respond to the later comments in this thread, not hijack the discussion. |
System information
Describe the problem you're observing
Possibly a duplicate of #16325, but I didn't see the behavior with the versions listed in that bug.
After upgrading from kernel
6.6.63with ZFS2.2.6to kernel6.12.13with ZFS2.3.0, I am experiencing issues with ARC memory utilization when my system's daily backups occur. My system has 16GB of RAM and I havezfs_arc_maxset to2147483648(2 GiB). When the backup occurs now, the ARC grows to a large size and the system locks up due to memory exhaustion.I can help the system get through the backup using
/proc/sys/vm/drop_cachesto clear the cache manually. I'm attempting to tune other ZFS settings to see if there's another way around it but I've never seen it exceedzfs_arc_maxby this much before.Describe how to reproduce the problem
This is reproducible any time I run a backup task.
Include any warning/errors/backtraces from the system logs
Here's an
arc_summary -drun exhibiting the issue:The text was updated successfully, but these errors were encountered: