Dotnet on OSX is limited to 10240 file handles

[LukeButters]

Description

Dotnet on mac is limited to only 10240 file handles, this occurs because:

runtime/src/coreclr/pal/src/init/pal.cpp

Lines 964 to 971 in eab352d

	#ifdef __APPLE__
	// Based on compatibility note in setrlimit(2) manpage for OSX,
	// trim the limit to OPEN_MAX.
	if (rlp.rlim_cur > OPEN_MAX)
	{
	rlp.rlim_cur = OPEN_MAX;
	}
	#endif

on OS X15.2 OPEN_MAX is defined as:

100:#define OPEN_MAX                10240   /* max open files per process - todo, make a config option? */

I got that from /Library/Developer/CommandLineTools/SDKs/MacOSX15.2.sdk/System/Library/Frameworks/Kernel.framework/Versions/A/Headers/sys/syslimits.h, another mac user got the same line from sys/resource.h.

The man page for setrlimit mentions:

COMPATIBILITY
setrlimit() now returns with errno set to EINVAL in places that historically succeeded. It no longer accepts "rlim_cur = RLIM_INFINITY" for RLIM_NOFILE. Use "rlim_cur = min(OPEN_MAX, rlim_max)".

Compiling my own version where those three lines are commented out, does result in a dotnet where I can open well over 10k files (my test ran to 250K files before I stopped it).

It would be useful if that artificial limit did not apply to the released version of dotnet.

Reproduction Steps

var directory = "/tmp/openfiles/";
        if(Directory.Exists(directory)) Directory.Delete(directory, true);
        
        Directory.CreateDirectory(directory);

        var fhs = new List<FileStream>();
        for (int i = 0; i < 30000; i++)
        {
            var file = File.Open(directory + "/" + i + ".txt", FileMode.OpenOrCreate, FileAccess.ReadWrite, FileShare.ReadWrite);
            fhs.Add(file);
            
            if(i % 1000 == 0) Console.WriteLine($"File {i} opened");
        }

Expected behavior

Can open more than 10240 files.

Actual behavior

fails to open more than 10kish files.

Regression?

no

Known Workarounds

none

Configuration

.net 8.0.13

Other information

No response

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/area-system-io
See info in area-owners.md if you want to be subscribed.

[LukeButters]

It is possible to compile a version that works around this on the mac for v8.0.13 by applying the attached diff then compiling with:
./build.sh clr -arch arm64 -c release
and then copying what is built into the existing copy of dotnet:
sudo cp -a artifacts/bin/coreclr/osx.arm64.Release/* /usr/local/share/dotnet/shared/Microsoft.NETCore.App/8.0.13/

Doing that does prevent things like dotnet-dump from running, I think because of unsigned code.

[LukeButters]

An easier workaround is to just intercept the sys call:
Set the contents of file hackopenfilelimits.c to:

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/resource.h>

// From https://opensource.apple.com/source/dyld/dyld-97.1/include/mach-o/dyld-interposing.h.auto.html
#define DYLD_INTERPOSE(_replacment, _replacee)                                    \
    __attribute__ ((used)) static struct                                          \
    {                                                                             \
        const void* replacment;                                                   \
        const void* replacee;                                                     \
    } _interpose_##_replacee __attribute__ ((section ("__DATA,__interpose"))) = { \
        (const void*) (unsigned long) &_replacment, (const void*) (unsigned long) &_replacee};



int my_setrlimit(int resource, const struct rlimit *rlp)
{
    if (RLIMIT_NOFILE == resource)
    {
        struct rlimit rlp2;
        rlp2.rlim_max = rlp->rlim_max;
        rlp2.rlim_cur = rlp->rlim_max;

        // printf ("Back to max");

        return setrlimit(resource, &rlp2);
    }
    return setrlimit(resource, rlp);
}

DYLD_INTERPOSE (my_setrlimit, setrlimit)

Compile it:

clang -dynamiclib hackopenfilelimits.c -o hackopenfilelimits.dylib

Include the following env var in any dotnet apps:

export DYLD_INSERT_LIBRARIES=`pwd`/hackopenfilelimits.dylib

[adamsitnik]

This code pre-dates me (last touched 8 years ago!), but my understanding is that we increase the default limit to the max allowed limit:

runtime/src/coreclr/pal/src/init/pal.cpp

Lines 944 to 945 in eab352d

	Calls setrlimit(2) to increase the maximum number of file descriptors
	this process can open.

runtime/src/coreclr/pal/src/init/pal.cpp

Lines 965 to 970 in eab352d

	// Based on compatibility note in setrlimit(2) manpage for OSX,
	// trim the limit to OPEN_MAX.
	if (rlp.rlim_cur > OPEN_MAX)
	{
	rlp.rlim_cur = OPEN_MAX;
	}

@janvorli @jkotas PTAL

[janvorli]

@LukeButters based on our experience it seems that Apple has changed the behavior since 2017 when I have added the limitation to OPEN_MAX, as it was really failing. See dotnet/coreclr#14054 description for details.

I'd be happy to accept a PR removing that artificial clipping if you prepared it.

[LukeButters]

I had a look at what getrlimit is returning on my mac.

For context:

ulimit -a
-t: cpu time (seconds)              unlimited
-f: file size (blocks)              unlimited
-d: data seg size (kbytes)          unlimited
-s: stack size (kbytes)             8176
-c: core file size (blocks)         0
-v: address space (kbytes)          unlimited
-l: locked-in-memory size (kbytes)  unlimited
-u: processes                       10666
-n: file descriptors                245760

The result of getrlimit:

rlim_max: 9223372036854775807
rlim_cur: 245760

This means the current behaviour results in the rlim_cur (soft limit) is set below the current value :(.

I was also concerned about setting the value to RLIM_INFINITY which on my machine is: 9223372036854775807 (when printed as %llu). So I tried setting rlim_cur to infinity, and it was succesful specifically setrlimit returned a result of 0.

[jkotas]

Should we simply disable this method for apple platforms?

[LukeButters]

By default macos has a crazy low soft limit of 256!

 ulimit -a
-t: cpu time (seconds)              unlimited
-f: file size (blocks)              unlimited
-d: data seg size (kbytes)          unlimited
-s: stack size (kbytes)             8176
-c: core file size (blocks)         0
-v: address space (kbytes)          unlimited
-l: locked-in-memory size (kbytes)  unlimited
-u: processes                       10666
-n: file descriptors                256

If a concern exists around backwards compatibility, we could on apple machines set the current limit to:

rlp.rlim_cur = max(rlp.rlim_cur, min(rlp.rlim_max, OPEN_MAX));

so if the user has already set a soft limit that exceeds OPEN_MAX we use what the user has set. If the user has a soft limit below rlim_max, we increase the soft limit up to the smaller of rlim_max and OPEN_MAX.

Doing that means:

• The soft limit is raised, since the default is way to low to realistically work.
• The soft limit is not lowered below what a user has specified.
• Backwards compatibility is maintained since soft limits are still raised, but we never raise them above what the doco says to raise them to.
• If the OS was ever to actually reject soft limits above OPEN_MAX (each on unsupported versions from 2017) the user is able to workaround the issue by using ulimit to lower the soft limit. (By default the limit is crazy low so this is a very unlikely scenario).