Tuning GC for specific byte size heap?

[HaroldCindy]

I'm looking to use Luau in a fairly resource-constrained embedded scenario, since the interpreter has nice performance characteristics. One problem I have is that I have very little memory available, and I'd like to make the GC get more aggressive as the total allocated bytes get closer to the max allowable.

I have a hard limit in my lua_Alloc hook that refuses to allocate if an alloc would cause the VM to go over the pre-defined memory limit, but it's not clear how to tune the GC to get more aggressive as the heap size approaches the limit.

I'm okay with messing with lgc.cpp and lmem.cpp internals, what would be the most reasonable way to achieve this? Have my alloc hook dynamically tune the GC goal size / step multiplier?

[vegorov-rbx]

You can use lua_gc to decrease goal % using LUA_GCSETGOAL and increase GC mark/sweep 'speed' using LUA_GCSETSTEPMUL.

Be careful of course to not choose values too extreme as it will increase the runtime cost.

[HaroldCindy]

Thanks! I'll do some profiling to see what values make the most sense.

I'm going to try pairing that with a forced "emergency" full GC in an interrupt handler if the heap ever goes above the slightly lower "soft" limit that I've defined to make sure the hard limit is never reached, and throw an error if the GC doesn't go below the soft limit after that. That should hopefully make allocation errors slightly more deterministic / easier to handle in Lua code (since I want to be able to do some manual clearing of caches and such on alloc errors in a pcall(), and that might itself require an alloc.)