velox_local_runner_test is flaky

[kagamiori]

Bug description

Example failure: https://github.com/facebookincubator/velox/actions/runs/12333843341/job/34423296887.

System information

CI

Relevant logs

2024-12-15T00:48:38.5214561Z 377/384 Test #382: velox_local_runner_test ..............................................Subprocess aborted***Exception:   9.40 sec
2024-12-15T00:48:38.5215885Z [==========] Running 3 tests from 1 test suite.
2024-12-15T00:48:38.5216613Z [----------] Global test environment set-up.
2024-12-15T00:48:38.5217136Z [----------] 3 tests from LocalRunnerTest
2024-12-15T00:48:38.5217560Z [ RUN      ] LocalRunnerTest.count
2024-12-15T00:48:38.5218254Z I1215 00:48:29.939764 94011 Compression.cpp:627] Initialized zstd compressor with compression level 7
2024-12-15T00:48:38.5218996Z [       OK ] LocalRunnerTest.count (2944 ms)
2024-12-15T00:48:38.5219425Z [ RUN      ] LocalRunnerTest.error
2024-12-15T00:48:38.5221297Z E20241215 00:48:33.879781 94011 Exceptions.h:66] Line: /home/runner/work/velox/velox/velox/velox/common/memory/SharedArbitrator.cpp:308, Function:shutdown, Expression: participants_.size() == 0 (1 vs. 0) Unexpected alive participants on destruction, Source: RUNTIME, ErrorCode: INVALID_STATE
2024-12-15T00:48:38.5223445Z terminate called after throwing an instance of 'facebook::velox::VeloxRuntimeError'
2024-12-15T00:48:38.5224125Z   what():  Exception: VeloxRuntimeError
2024-12-15T00:48:38.5224525Z Error Source: RUNTIME
2024-12-15T00:48:38.5224845Z Error Code: INVALID_STATE
2024-12-15T00:48:38.5225295Z Reason: (1 vs. 0) Unexpected alive participants on destruction
2024-12-15T00:48:38.5225803Z Retriable: False
2024-12-15T00:48:38.5226129Z Expression: participants_.size() == 0
2024-12-15T00:48:38.5226523Z Function: shutdown
2024-12-15T00:48:38.5227093Z File: /home/runner/work/velox/velox/velox/velox/common/memory/SharedArbitrator.cpp
2024-12-15T00:48:38.5227997Z Line: 308
2024-12-15T00:48:38.5228256Z Stack trace:
2024-12-15T00:48:38.5228591Z # 0  _ZN8facebook5velox7process10StackTraceC1Ei
2024-12-15T00:48:38.5229853Z # 1  _ZN8facebook5velox14VeloxException5State4makeIZNS1_C4EPKcmS5_St17basic_string_viewIcSt11char_traitsIcEES9_S9_S9_bNS1_4TypeES9_EUlRT_E_EESt10shared_ptrIKS2_ESA_SB_
2024-12-15T00:48:38.5231505Z # 2  _ZN8facebook5velox14VeloxExceptionC1EPKcmS3_St17basic_string_viewIcSt11char_traitsIcEES7_S7_S7_bNS1_4TypeES7_
2024-12-15T00:48:38.5232786Z # 3  _ZN8facebook5velox17VeloxRuntimeErrorC2EPKcmS3_St17basic_string_viewIcSt11char_traitsIcEES7_S7_S7_bS7_
2024-12-15T00:48:38.5234401Z # 4  _ZN8facebook5velox6detail14veloxCheckFailINS0_17VeloxRuntimeErrorERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEEvRKNS1_18VeloxCheckFailArgsET0_
2024-12-15T00:48:38.5236285Z # 5  _ZN8facebook5velox6memory16SharedArbitrator8shutdownEv
2024-12-15T00:48:38.5326710Z # 6  _ZN8facebook5velox6memory13MemoryManagerD1Ev
2024-12-15T00:48:38.5327638Z # 7  _ZN8facebook5velox6memory13MemoryManager18testingSetInstanceERKNS1_20MemoryManagerOptionsE
2024-12-15T00:48:38.5328698Z # 8  _ZN8facebook5velox4exec4test16OperatorTestBase11setupMemoryElllllll
2024-12-15T00:48:38.5329549Z # 9  _ZN8facebook5velox4exec4test16OperatorTestBase11resetMemoryEv
2024-12-15T00:48:38.5330286Z # 10 _ZN8facebook5velox4exec4test16OperatorTestBase8TearDownEv
2024-12-15T00:48:38.5331029Z # 11 _ZN8facebook5velox4exec4test21HiveConnectorTestBase8TearDownEv
2024-12-15T00:48:38.5332029Z # 12 _ZN7testing8internal38HandleSehExceptionsInMethodIfSupportedINS_4TestEvEET0_PT_MS4_FS3_vEPKc
2024-12-15T00:48:38.5333211Z # 13 _ZN7testing8internal35HandleExceptionsInMethodIfSupportedINS_4TestEvEET0_PT_MS4_FS3_vEPKc
2024-12-15T00:48:38.5334004Z # 14 _ZN7testing4Test3RunEv
2024-12-15T00:48:38.5334363Z # 15 _ZN7testing8TestInfo3RunEv
2024-12-15T00:48:38.5334746Z # 16 _ZN7testing9TestSuite3RunEv
2024-12-15T00:48:38.5335270Z # 17 _ZN7testing8internal12UnitTestImpl11RunAllTestsEv
2024-12-15T00:48:38.5336234Z # 18 _ZN7testing8internal38HandleSehExceptionsInMethodIfSupportedINS0_12UnitTestImplEbEET0_PT_MS4_FS3_vEPKc
2024-12-15T00:48:38.5337865Z # 19 _ZN7testing8internal35HandleExceptionsInMethodIfSupportedINS0_12UnitTestImplEbEET0_PT_MS4_FS3_vEPKc
2024-12-15T00:48:38.5338734Z # 20 _ZN7testing8UnitTest3RunEv
2024-12-15T00:48:38.5339087Z # 21 _Z13RUN_ALL_TESTSv
2024-12-15T00:48:38.5339387Z # 22 main
2024-12-15T00:48:38.5339643Z # 23 0x0000000000029d8f
2024-12-15T00:48:38.5339936Z # 24 __libc_start_main
2024-12-15T00:48:38.5340232Z # 25 _start
2024-12-15T00:48:38.5340387Z 
2024-12-15T00:48:38.5340654Z *** Aborted at 1734223714 (Unix time, try 'date -d @1734223714') ***
2024-12-15T00:48:38.5341818Z *** Signal 6 (SIGABRT) (0x3e900016f3b) received by PID 94011 (pthread TID 0x7f5dca29a340) (linux TID 94011) (maybe from PID 94011, UID 1001) (code: -6), stack trace: ***
2024-12-15T00:48:38.5343165Z     @ 00000000175ccb9e _ZN5folly10symbolizer12_GLOBAL__N_118innerSignalHandlerEiP9siginfo_tPv
2024-12-15T00:48:38.5344386Z                        /home/runner/work/velox/velox/velox/_build/debug/_deps/folly-src/folly/experimental/symbolizer/SignalHandler.cpp:453
2024-12-15T00:48:38.5345560Z     @ 00000000175ccc84 _ZN5folly10symbolizer12_GLOBAL__N_113signalHandlerEiP9siginfo_tPv
2024-12-15T00:48:38.5346725Z                        /home/runner/work/velox/velox/velox/_build/debug/_deps/folly-src/folly/experimental/symbolizer/SignalHandler.cpp:474
2024-12-15T00:48:38.5347817Z     @ 000000000004251f (unknown)
2024-12-15T00:48:38.5348177Z     @ 00000000000969fc pthread_kill
2024-12-15T00:48:38.5348535Z     @ 0000000000042475 raise
2024-12-15T00:48:38.5348861Z     @ 00000000000287f2 abort
2024-12-15T00:48:38.5349187Z     @ 00000000000a2b9d (unknown)
2024-12-15T00:48:38.5349534Z     @ 00000000000ae20b (unknown)
2024-12-15T00:48:38.5349876Z     @ 00000000000ad1e8 (unknown)
2024-12-15T00:48:38.5350238Z     @ 00000000000ad958 __gxx_personality_v0
2024-12-15T00:48:38.5350666Z     @ 000000000000afe8 __libunwind_Unwind_Resume
2024-12-15T00:48:38.5351307Z     @ 00000000173d1292 _ZN8facebook5velox6memory16SharedArbitrator8shutdownEv
2024-12-15T00:48:38.5352219Z                        /home/runner/work/velox/velox/velox/velox/common/memory/SharedArbitrator.cpp:308
2024-12-15T00:48:38.5353061Z     @ 0000000017384df9 _ZN8facebook5velox6memory13MemoryManagerD1Ev
2024-12-15T00:48:38.5353823Z                        /home/runner/work/velox/velox/velox/velox/common/memory/Memory.cpp:137
2024-12-15T00:48:38.5354932Z     @ 0000000017385495 _ZN8facebook5velox6memory13MemoryManager18testingSetInstanceERKNS1_20MemoryManagerOptionsE
2024-12-15T00:48:38.5356030Z                        /home/runner/work/velox/velox/velox/velox/common/memory/Memory.cpp:204
2024-12-15T00:48:38.5357195Z     @ 000000000c306de9 _ZN8facebook5velox4exec4test16OperatorTestBase11setupMemoryElllllll
2024-12-15T00:48:38.5358231Z                        /home/runner/work/velox/velox/velox/velox/exec/tests/utils/OperatorTestBase.cpp:113
2024-12-15T00:48:38.5359211Z     @ 000000000c307172 _ZN8facebook5velox4exec4test16OperatorTestBase11resetMemoryEv
2024-12-15T00:48:38.5360184Z                        /home/runner/work/velox/velox/velox/velox/exec/tests/utils/OperatorTestBase.cpp:120
2024-12-15T00:48:38.5361132Z     @ 000000000c307352 _ZN8facebook5velox4exec4test16OperatorTestBase8TearDownEv
2024-12-15T00:48:38.5362075Z                        /home/runner/work/velox/velox/velox/velox/exec/tests/utils/OperatorTestBase.cpp:160
2024-12-15T00:48:38.5363056Z     @ 000000000c2c60a9 _ZN8facebook5velox4exec4test21HiveConnectorTestBase8TearDownEv
2024-12-15T00:48:38.5364085Z                        /home/runner/work/velox/velox/velox/velox/exec/tests/utils/HiveConnectorTestBase.cpp:69
2024-12-15T00:48:38.5365333Z     @ 000000000c5ca136 _ZN7testing8internal38HandleSehExceptionsInMethodIfSupportedINS_4TestEvEET0_PT_MS4_FS3_vEPKc
2024-12-15T00:48:38.5366564Z                        /home/runner/work/velox/velox/velox/_build/debug/_deps/gtest-src/googletest/src/gtest.cc:2621
2024-12-15T00:48:38.5367633Z                        -> /home/runner/work/velox/velox/velox/_build/debug/_deps/gtest-src/googletest/src/gtest-all.cc
2024-12-15T00:48:38.5369025Z     @ 000000000c5c3302 _ZN7testing8internal35HandleExceptionsInMethodIfSupportedINS_4TestEvEET0_PT_MS4_FS3_vEPKc
2024-12-15T00:48:38.5370229Z                        /home/runner/work/velox/velox/velox/_build/debug/_deps/gtest-src/googletest/src/gtest.cc:2657
2024-12-15T00:48:38.5371283Z                        -> /home/runner/work/velox/velox/velox/_build/debug/_deps/gtest-src/googletest/src/gtest-all.cc
2024-12-15T00:48:38.5372038Z     @ 000000000c59ed61 _ZN7testing4Test3RunEv
2024-12-15T00:48:38.5372776Z                        /home/runner/work/velox/velox/velox/_build/debug/_deps/gtest-src/googletest/src/gtest.cc:2704
2024-12-15T00:48:38.5373830Z                        -> /home/runner/work/velox/velox/velox/_build/debug/_deps/gtest-src/googletest/src/gtest-all.cc
2024-12-15T00:48:38.5374599Z     @ 000000000c59f82a _ZN7testing8TestInfo3RunEv
2024-12-15T00:48:38.5375352Z                        /home/runner/work/velox/velox/velox/_build/debug/_deps/gtest-src/googletest/src/gtest.cc:2845
2024-12-15T00:48:38.5376417Z                        -> /home/runner/work/velox/velox/velox/_build/debug/_deps/gtest-src/googletest/src/gtest-all.cc
2024-12-15T00:48:38.5377182Z     @ 000000000c5a0177 _ZN7testing9TestSuite3RunEv
2024-12-15T00:48:38.5377937Z                        /home/runner/work/velox/velox/velox/_build/debug/_deps/gtest-src/googletest/src/gtest.cc:3004
2024-12-15T00:48:38.5378989Z                        -> /home/runner/work/velox/velox/velox/_build/debug/_deps/gtest-src/googletest/src/gtest-all.cc
2024-12-15T00:48:38.5379903Z     @ 000000000c5b05ef _ZN7testing8internal12UnitTestImpl11RunAllTestsEv
2024-12-15T00:48:38.5380821Z                        /home/runner/work/velox/velox/velox/_build/debug/_deps/gtest-src/googletest/src/gtest.cc:5890
2024-12-15T00:48:38.5381882Z                        -> /home/runner/work/velox/velox/velox/_build/debug/_deps/gtest-src/googletest/src/gtest-all.cc
2024-12-15T00:48:38.5383185Z     @ 000000000c5cb047 _ZN7testing8internal38HandleSehExceptionsInMethodIfSupportedINS0_12UnitTestImplEbEET0_PT_MS4_FS3_vEPKc
2024-12-15T00:48:38.5384495Z                        /home/runner/work/velox/velox/velox/_build/debug/_deps/gtest-src/googletest/src/gtest.cc:2621
2024-12-15T00:48:38.5385547Z                        -> /home/runner/work/velox/velox/velox/_build/debug/_deps/gtest-src/googletest/src/gtest-all.cc
2024-12-15T00:48:38.5386828Z     @ 000000000c5c43ba _ZN7testing8internal35HandleExceptionsInMethodIfSupportedINS0_12UnitTestImplEbEET0_PT_MS4_FS3_vEPKc
2024-12-15T00:48:38.5388278Z                        /home/runner/work/velox/velox/velox/_build/debug/_deps/gtest-src/googletest/src/gtest.cc:2657
2024-12-15T00:48:38.5389535Z                        -> /home/runner/work/velox/velox/velox/_build/debug/_deps/gtest-src/googletest/src/gtest-all.cc
2024-12-15T00:48:38.5390304Z     @ 000000000c5aebd6 _ZN7testing8UnitTest3RunEv
2024-12-15T00:48:38.5391059Z                        /home/runner/work/velox/velox/velox/_build/debug/_deps/gtest-src/googletest/src/gtest.cc:5455
2024-12-15T00:48:38.5392121Z                        -> /home/runner/work/velox/velox/velox/_build/debug/_deps/gtest-src/googletest/src/gtest-all.cc
2024-12-15T00:48:38.5392859Z     @ 000000000c2a515c _Z13RUN_ALL_TESTSv
2024-12-15T00:48:38.5393641Z                        /home/runner/work/velox/velox/velox/_build/debug/_deps/gtest-src/googletest/include/gtest/gtest.h:2314
2024-12-15T00:48:38.5394603Z                        -> /home/runner/work/velox/velox/velox/velox/runner/tests/Main.cpp
2024-12-15T00:48:38.5395168Z     @ 000000000c2a508d main
2024-12-15T00:48:38.5395683Z                        /home/runner/work/velox/velox/velox/velox/runner/tests/Main.cpp:28
2024-12-15T00:48:38.5396265Z     @ 0000000000029d8f (unknown)
2024-12-15T00:48:38.5396623Z     @ 0000000000029e3f __libc_start_main
2024-12-15T00:48:38.5397006Z     @ 000000000c283d04 _start
2024-12-15T00:48:38.5397435Z Fatal signal handler. ThreadDebugInfo object not found.