Crash with XIO: fatal IO error 24 (Zu viele offene Dateien) on X server ":1"

[lem-solutions]

Operating System Info

Other

Other OS

Debian Trixie

OBS Studio Version

Git

OBS Studio Version (Other)

No response

OBS Studio Log URL

https://obsproject.com/logs/smBwzmgjuiFsLdQD

OBS Studio Crash Log URL

https://obsproject.com/logs/smBwzmgjuiFsLdQD

Expected Behavior

OBS should not exhaust the open file limit and crash shortly after launch.

Current Behavior

OBS hangs and crashes after a brief moment after launch. It seems to exhaust the OS open file limit and then choke on that.

cli output 1:
obs cli output.txt

cli output 2:
obs cli output 2.txt

Steps to Reproduce

1. Install OBS (I tested both the Debian package and compiling master)
2. Run OBS
3. Wait ~5 seconds
4. ???
5. Profit!

Since there isn't really anything special that I'm doing that's triggering this, I doubt anyone will be able to reproduce this, but it's happening consistently to me.

Anything else we should know?

I Tried finding the syscall that first returns EMFILE(the ERRNO that indicates the open file limit has been reached) with gdb and came up with this:

backtrace.txt

(note that this is a socket syscall, so OBS/libdbus doesn't try to excessively open regular files, but creates an excessive number of sockets)

Not sure if that's useful though. I'd think it might be helpful to know what the QEvent* passed to QObject::event is, but I don't know how to figure that out.

If you guys can tell me how to get more info that'd be useful to you please let me know.

[kkartaltepe]

Can you provide the list of open files from obs after this crash occurs but before obs is terminated. Such as running obs in gdb and then performing an lsof -p $OBS_PID.

[lem-solutions]

Sure: lsof output.txt

[kkartaltepe]

Sorry It looks like its need to be re-run with lsof +E -p $OBS_PID to identify the sockets.

[lem-solutions]

lsof +E.txt

[kkartaltepe]

Thanks all the excessive sockets are from dbus, however the stack trace above would indicate an issue in qt. Its possible the sockets are being opened elsewhere but we really only ever touch dbus directly in obs for screencapture portals and screensaver inhibiting.

[lem-solutions]

I'm not very familiar with cmake, would it be possible to compile libdbus and the qt libraries with debug info and have cmake use those instead of the system ones? If so how do I tell cmake to use them?

[kkartaltepe]

If you can compile obs with this patch:

diff --git a/libobs/cmake/os-linux.cmake b/libobs/cmake/os-linux.cmake
index 1af874f4f..0729e44c4 100644
--- a/libobs/cmake/os-linux.cmake
+++ b/libobs/cmake/os-linux.cmake
@@ -76,10 +76,10 @@ else()
   target_disable_feature(libobs "PulseAudio audio monitoring (Linux)")
 endif()
 
-if(TARGET gio::gio)
-  target_sources(libobs PRIVATE util/platform-nix-dbus.c util/platform-nix-portal.c)
-  target_link_libraries(libobs PRIVATE gio::gio)
-endif()
+# if(TARGET gio::gio)
+# target_sources(libobs PRIVATE util/platform-nix-dbus.c util/platform-nix-portal.c)
+# target_link_libraries(libobs PRIVATE gio::gio)
+# endif()
 
 if(ENABLE_WAYLAND)
   find_package(Wayland REQUIRED Client)
diff --git a/libobs/obsconfig.h.in b/libobs/obsconfig.h.in
index 2a7c87df8..bb7eccd5b 100644
--- a/libobs/obsconfig.h.in
+++ b/libobs/obsconfig.h.in
@@ -4,7 +4,7 @@
 #cmakedefine OBS_PLUGIN_PATH "@OBS_PLUGIN_PATH@"
 #cmakedefine OBS_PLUGIN_DESTINATION "@OBS_PLUGIN_DESTINATION@"
 
-#cmakedefine GIO_FOUND
+// #cmakedefine GIO_FOUND
 #cmakedefine PULSEAUDIO_FOUND
 #cmakedefine XCB_XINPUT_FOUND
 #cmakedefine ENABLE_WAYLAND

It will disable the normal dbus usage we control and let us know if the error is in our code or triggered by something else.

[lem-solutions]

The problem persists with that patch, I'll open an issue with qt. Thanks for your help :)