SIGSEGV in mono_type_get_name_full

[TimBurik]

Android framework version

net8.0-android, net9.0-android

Affected platform version

.NET 8.0.403, .NET 9.0.100

Description

Ever since switching from Xamarin.Android to .NET8, and recently to .NET9, we see a crash group which appears in all our releases in GooglePlay Console. Originally we were suspected that this crash group might be related to the "out-of-memory", but we don't see any other alloc-related issues originating from other places.
Interestingly, only a small percentage of reports GooglePlay Console considers to be a "user-perceived" once. We are suspecting that these crashes might be happening when one of our crash monitoring systems (AppCenter, Sentry) is collecting information.

We have a lot of similar crash reports: from different architectures (armeabi/arm64), with different levels of recursion, with different realloc implementations, but always from the same place of origin.
crash_symbolicated_example1.txt
crash_symbolicated_example2.txt
crash_symbolicated_example3.txt
crash_symbolicated_example4.txt

Steps to Reproduce

Unfortunately, we haven't found a reliable way to reproduce it manually, but it has a pretty big counter in production (around 0.5% affected sessions)

Did you find any workaround?

No workaround found yet

Relevant log output

*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 2783 >>> <app.bundle.id> <<<

backtrace:
  #00  pc 0x0000000000061a28  /apex/com.android.runtime/lib64/bionic/libc.so (je_arena_ralloc_no_move+88)
  #01  pc 0x0000000000061f2c  /apex/com.android.runtime/lib64/bionic/libc.so (je_arena_ralloc+160)
  #02  pc 0x0000000000059740  /apex/com.android.runtime/lib64/bionic/libc.so (je_realloc+236)
  #03  pc 0x0000000000053934  /apex/com.android.runtime/lib64/bionic/libc.so (realloc+88)
  #04  pc 0x00000000001ddba8  /data/app/~~vrG0vqSsCasZF9aCOcJVQg==/<app.bundle.id>-GHfs8XiUFY5dyQ87mHQkIA==/split_config.arm64_v8a.apk!libmonosgen-2.0.so (monoeg_realloc+108)
  #05  pc 0x00000000001e0720  /data/app/~~vrG0vqSsCasZF9aCOcJVQg==/<app.bundle.id>-GHfs8XiUFY5dyQ87mHQkIA==/split_config.arm64_v8a.apk!libmonosgen-2.0.so (monoeg_g_string_append+105)
  #06  pc 0x000000000020d820  /data/app/~~vrG0vqSsCasZF9aCOcJVQg==/<app.bundle.id>-GHfs8XiUFY5dyQ87mHQkIA==/split_config.arm64_v8a.apk!libmonosgen-2.0.so (mono_type_get_name_recurse+457)
  #07  pc 0x000000000020d740  /data/app/~~vrG0vqSsCasZF9aCOcJVQg==/<app.bundle.id>-GHfs8XiUFY5dyQ87mHQkIA==/split_config.arm64_v8a.apk!libmonosgen-2.0.so (mono_type_get_name_recurse+388)
  #08  pc 0x000000000020daf0  /data/app/~~vrG0vqSsCasZF9aCOcJVQg==/<app.bundle.id>-GHfs8XiUFY5dyQ87mHQkIA==/split_config.arm64_v8a.apk!libmonosgen-2.0.so (mono_type_get_name_recurse+494)
  #09  pc 0x000000000020daf0  /data/app/~~vrG0vqSsCasZF9aCOcJVQg==/<app.bundle.id>-GHfs8XiUFY5dyQ87mHQkIA==/split_config.arm64_v8a.apk!libmonosgen-2.0.so (mono_type_get_name_recurse+494)
  #10  pc 0x000000000020d514  /data/app/~~vrG0vqSsCasZF9aCOcJVQg==/<app.bundle.id>-GHfs8XiUFY5dyQ87mHQkIA==/split_config.arm64_v8a.apk!libmonosgen-2.0.so (mono_type_get_name_full+557)
  #11  pc 0x000000000023cd18  /data/app/~~vrG0vqSsCasZF9aCOcJVQg==/<app.bundle.id>-GHfs8XiUFY5dyQ87mHQkIA==/split_config.arm64_v8a.apk!libmonosgen-2.0.so (ves_icall_System_RuntimeType_getFullName_raw+5131)
  #12  pc 0x000000000000b2d0 

[jonpryor]

Unfortunately, in all of the attached crash_symbolicated*.txt files and in the Relevant log output section, the crash isn't within mono_type_get_name_full, but instead rooted by realloc(3):

  #00  pc 0x0000000000061a28  /apex/com.android.runtime/lib64/bionic/libc.so (je_arena_ralloc_no_move+88)
  #01  pc 0x0000000000061f2c  /apex/com.android.runtime/lib64/bionic/libc.so (je_arena_ralloc+160)
  #02  pc 0x0000000000059740  /apex/com.android.runtime/lib64/bionic/libc.so (je_realloc+236)
  #03  pc 0x0000000000053934  /apex/com.android.runtime/lib64/bionic/libc.so (realloc+88)

The caller in all cases is monoeg_realloc(), which is g_realloc(), which calls realloc(3).

I don't know how wee can protect against a crash within realloc(3).

My guess is that the RAM on these devices is going bad, or a cosmic ray is altering RAM contents. (See also, Effect of Cosmic Rays on Computer Memories.) The error described doesn't otherwise make any sense at all.

[TimBurik]

Hello @jonpryor
We were also initially thinking about memory corruption or issues with the realloc, but ultimately discarded this idea with the following reasoning:

1. The crash group which contains (mono_type_get_name_recurse+457) in stack trace has by far the biggest count in our latest release (262 events vs 164 for the top 2);
2. We don't see any crash groups, related to the memory allocation/reallocation/deallocation, but not related to the mono_type_get_name_full. If memory was corrupted - there would have been many reports from different places, it feels;
3. I've just went through the full list of crash reports for the last 60 days, and I've found one more very similar stack trace:

*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 18223 >>> <app-bundle-id> <<<

backtrace:
  #00  pc 0x00000000000fe6b4  /apex/com.android.runtime/lib64/bionic/libc.so (pthread_mutex_trylock+8)
  #01  pc 0x000000000005f0f8  /apex/com.android.runtime/lib64/bionic/libc.so (je_arena_dalloc_small+412)
  #02  pc 0x000000000005b77c  /apex/com.android.runtime/lib64/bionic/libc.so (arena_dalloc_no_tcache+172)
  #03  pc 0x0000000000057f20  /apex/com.android.runtime/lib64/bionic/libc.so (je_free+1532)
  #04  pc 0x000000000020d828  /data/app/~~RL6hdI-QsPXoCMvL9q104w==/<app-bundle-id>-Cffzh5p9axDpbK9h8aVXDQ==/split_config.arm64_v8a.apk!libmonosgen-2.0.so (mono_type_get_name_recurse+458)
  #05  pc 0x000000000020d620  /data/app/~~RL6hdI-QsPXoCMvL9q104w==/<app-bundle-id>-Cffzh5p9axDpbK9h8aVXDQ==/split_config.arm64_v8a.apk!libmonosgen-2.0.so (mono_type_get_name_recurse+445)
  #06  pc 0x000000000020db18  /data/app/~~RL6hdI-QsPXoCMvL9q104w==/<app-bundle-id>-Cffzh5p9axDpbK9h8aVXDQ==/split_config.arm64_v8a.apk!libmonosgen-2.0.so (mono_type_get_name_recurse+494)
  #07  pc 0x000000000020d514  /data/app/~~RL6hdI-QsPXoCMvL9q104w==/<app-bundle-id>-Cffzh5p9axDpbK9h8aVXDQ==/split_config.arm64_v8a.apk!libmonosgen-2.0.so (mono_type_get_name_full+557)
  #08  pc 0x000000000023cd18  /data/app/~~RL6hdI-QsPXoCMvL9q104w==/<app-bundle-id>-Cffzh5p9axDpbK9h8aVXDQ==/split_config.arm64_v8a.apk!libmonosgen-2.0.so (ves_icall_System_RuntimeType_getFullName_raw+5131)
  #09  pc 0x000000000000a218 

4. This means that the only two crash groups related to the memory allocation we had in last 60 days, both originating from two neighboring lines in source code:
   https://github.com/dotnet/runtime/blob/d2cdcdd69391bbfde1a8fb4aa275a6cc393ca65b/src/mono/mono/metadata/class.c#L457-L458