Fixed GIL release issue with Python System and Python TestFixture. #2618

AmalDevHaridevan · 2024-09-11T23:34:41Z

…gned-off-by: Amal Dev Haridevan [email protected]

🦟 Bug fix

Fixes #
Add explicit scoped acquire and release of GIL, so that Python Systems can be executed from a Python Server (run using TestFixture for example).

Summary

Python system plugin, which is attached to my model cannot be run using TestFixture in Python. This is due to GIL not being explicitly released in the PythonSystemLoader.
To overcome this issue, we need to do a scoped_acquire of GIL explicitly and then perform scoped_release after each of the system methods, namely, PreUpdate, Update, PostUpdate of the PythonSystem, so the GIL can be accessed by the TestFixture after.
As a safety mechanism, for future, I also added the scoped_acquire and release of GIL within the pybind code for TestFixture.

Checklist

Signed all commits for DCO
Added tests
Updated documentation (as needed)
Updated migration guide (as needed)
Consider updating Python bindings (if the library has them)
codecheck passed (See contributing)
All tests passed (See test coverage)
While waiting for a review on your PR, please help review another open pull request to support the maintainers

Note to maintainers: Remember to use Squash-Merge and edit the commit message to match the pull request summary while retaining Signed-off-by messages.

🔸🔸🔸🔸🔸🔸🔸🔸🔸🔸🔸🔸🔸🔸🔸🔸🔸🔸🔸🔸🔸🔸🔸🔸🔸🔸🔸🔸🔸🔸🔸🔸🔸🔸🔸🔸🔸

Note to maintainers: Remember to use Squash-Merge and edit the commit message to match the pull request summary while retaining Signed-off-by messages.

…gned-off-by: Amal Dev Haridevan [email protected] Signed-off-by: sdcnlab <[email protected]> Signed-off-by: Amal Dev Haridevan <[email protected]>

python/test/python_system_TEST.py

python/test/gravity.sdf

Co-authored-by: Alejandro Hernández Cordero <[email protected]> Signed-off-by: AmaldevHaridevan <[email protected]>

Signed-off-by: AmaldevHaridevan <[email protected]>

python/src/gz/sim/TestFixture.cc

azeey · 2024-10-14T22:06:38Z

@AmalDevHaridevan just wanted to give you a heads-up that I won't be able to review this for the next couple of weeks due to ROSCon.

AmalDevHaridevan · 2024-12-11T04:32:35Z

@AmalDevHaridevan just wanted to give you a heads-up that I won't be able to review this for the next couple of weeks due to ROSCon.

Hi Azeey, any updates on the pull request?

azeey

Thanks for your patience. I can see the issue now and I can reproduce it with

from gz_test_deps.sim import Server, ServerConfig
from gz_test_deps.common import set_verbosity

set_verbosity(4)
config  = ServerConfig()
config.set_sdf_file('gravity.sdf')
server = Server(config)
server.run(True, 1000, False)

So it has nothing to do with TestFixture, but with how Server is being loaded from python. As mentioned in

gz-sim/src/SimulationRunner.cc

Lines 69 to 77 in c72be04

    
           // Helper struct to maybe do a scoped release of the Python GIL. There are a 
        
           // number of ways where releasing and acquiring the GIL is not necessary: 
        
           // 1. Gazebo is built without Pybind11 
        
           // 2. The python interpreter is not initialized. This could happen in tests that 
        
           //    create a SimulationRunner without sim::Server where the interpreter is 
        
           //    intialized. 
        
           // 3. sim::Server was instantiated by a Python module. In this case, there's a 
        
           //    chance that this would be called with the GIL already released. 
        
           struct MaybeGilScopedRelease

When Server is instantiated from a python interpreter, pybind11 automatically releases the GIL before calling the C++ code (Server()). When PythonSystemLoader then tries to load the python system, it fails since any access to Python state requires locking the GIL. The solution you proposed works pretty well I think. My only comment is that we limit the changes to PythonSystemLoader and acquire the GIL in every System interface call.

azeey · 2025-01-24T20:41:43Z

src/systems/python_system_loader/PythonSystemLoader.cc

  CallPythonMethod(this->preUpdateMethod, _info, &_ecm);
+  py::gil_scoped_release gilr;


This is not necessary since py::gil_scoped_acquire will release the gil when going out of scope.

azeey · 2025-01-24T20:43:55Z

python/test/python_system_TEST.py

Would it be possible to use the TestModelSystem plugin that's already in python/test/plugin?

Can you remove this file then?

azeey · 2025-01-24T20:45:03Z

src/systems/python_system_loader/PythonSystemLoader.cc

I think need more py::gil_scoped_acquire gil; calls in PythonSystemLoader::Configure, PythonSystemLoader::Reset(), and PythonSystemLoader::~PythonSystemLoader(). Basically, all System interfaces since they will be accessing python state.

azeey · 2025-01-24T20:46:43Z

python/src/gz/sim/TestFixture.cc

@@ -56,7 +56,14 @@ defineSimTestFixture(pybind11::object module)
      [](TestFixture* self, std::function<void(
          const UpdateInfo &, EntityComponentManager &)> _cb)
      {
+        // Add explicit scoped acquire and release of GIL, so that Python Systems


If we apply the changes in PythonSystemLoader, we shouldn't need any change in this file. Can you revert?

Hi azeey,
I addressed all your comments on the latest commit.
Hope this can be merged now.

…sed the TestModelSystem plugin for testing Signed-off-by: Amal Dev Haridevan <[email protected]>

Signed-off-by: Amal Dev Haridevan <[email protected]>

…ridevan/gz-sim into Amal-python-system-GIL

azeey

Thanks for the contribution and for your patience 🙏🏾

All comments have been addressed

AmalDevHaridevan requested a review from mjcarroll as a code owner September 11, 2024 23:34

github-actions bot added the 🏛️ ionic Gazebo Ionic label Sep 11, 2024

AmalDevHaridevan force-pushed the Amal-python-system-GIL branch from d11629b to c436b61 Compare September 11, 2024 23:37

Fixed GIL release issue with Python System and Python TestFixture. Si…

29598fd

…gned-off-by: Amal Dev Haridevan [email protected] Signed-off-by: sdcnlab <[email protected]> Signed-off-by: Amal Dev Haridevan <[email protected]>

AmalDevHaridevan force-pushed the Amal-python-system-GIL branch from c436b61 to 29598fd Compare September 11, 2024 23:40

ahcorde previously requested changes Sep 12, 2024

View reviewed changes

python/test/python_system_TEST.py Outdated Show resolved Hide resolved

python/test/python_system_TEST.py Outdated Show resolved Hide resolved

python/test/gravity.sdf Show resolved Hide resolved

AmalDevHaridevan and others added 4 commits September 12, 2024 10:28

Update python/test/python_system_TEST.py

334b432

Co-authored-by: Alejandro Hernández Cordero <[email protected]> Signed-off-by: AmaldevHaridevan <[email protected]>

Update python_system_TEST.py

8aff92f

Signed-off-by: AmaldevHaridevan <[email protected]>

Update gravity.sdf

9df6542

Signed-off-by: AmaldevHaridevan <[email protected]>

Merge branch 'gz-sim9' into Amal-python-system-GIL

ccf90c5

azeey reviewed Sep 13, 2024

View reviewed changes

python/src/gz/sim/TestFixture.cc Outdated Show resolved Hide resolved

python/src/gz/sim/TestFixture.cc Outdated Show resolved Hide resolved

AmalDevHaridevan added 2 commits September 13, 2024 21:52

Merge branch 'gz-sim9' into Amal-python-system-GIL

1d01094

Merge branch 'gz-sim9' into Amal-python-system-GIL

26f69d8

AmalDevHaridevan requested review from ahcorde and azeey October 10, 2024 16:03

Merge branch 'gz-sim9' into Amal-python-system-GIL

b5f7862

azeey requested changes Jan 24, 2025

View reviewed changes

AmalDevHaridevan and others added 4 commits February 6, 2025 20:23

Added gil acquire to Configure, Destructor, and Reset methods. Also u…

33ca702

…sed the TestModelSystem plugin for testing Signed-off-by: Amal Dev Haridevan <[email protected]>

Merge branch 'gz-sim9' into Amal-python-system-GIL

51153a0

Removed python_system_TEST.py

d5b86ae

Signed-off-by: Amal Dev Haridevan <[email protected]>

Merge branch 'Amal-python-system-GIL' of https://github.com/AmalDevHa…

46ac9be

…ridevan/gz-sim into Amal-python-system-GIL

azeey approved these changes Feb 21, 2025

View reviewed changes

Merge branch 'gz-sim9' into Amal-python-system-GIL

8308a35

azeey merged commit 4df5170 into gazebosim:gz-sim9 Feb 22, 2025
11 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fixed GIL release issue with Python System and Python TestFixture. #2618

Fixed GIL release issue with Python System and Python TestFixture. #2618

AmalDevHaridevan commented Sep 11, 2024

azeey commented Oct 14, 2024

AmalDevHaridevan commented Dec 11, 2024

azeey left a comment

azeey Jan 24, 2025

azeey Jan 24, 2025

azeey Feb 7, 2025

AmalDevHaridevan Feb 8, 2025

azeey Jan 24, 2025

azeey Jan 24, 2025

AmalDevHaridevan Feb 7, 2025

azeey left a comment

	// Helper struct to maybe do a scoped release of the Python GIL. There are a
	// number of ways where releasing and acquiring the GIL is not necessary:
	// 1. Gazebo is built without Pybind11
	// 2. The python interpreter is not initialized. This could happen in tests that
	// create a SimulationRunner without sim::Server where the interpreter is
	// intialized.
	// 3. sim::Server was instantiated by a Python module. In this case, there's a
	// chance that this would be called with the GIL already released.
	struct MaybeGilScopedRelease

		CallPythonMethod(this->preUpdateMethod, _info, &_ecm);
		py::gil_scoped_release gilr;

Fixed GIL release issue with Python System and Python TestFixture. #2618

Fixed GIL release issue with Python System and Python TestFixture. #2618

Conversation

AmalDevHaridevan commented Sep 11, 2024

🦟 Bug fix

Summary

Checklist

azeey commented Oct 14, 2024

AmalDevHaridevan commented Dec 11, 2024

azeey left a comment

Choose a reason for hiding this comment

azeey Jan 24, 2025

Choose a reason for hiding this comment

azeey Jan 24, 2025

Choose a reason for hiding this comment

azeey Feb 7, 2025

Choose a reason for hiding this comment

AmalDevHaridevan Feb 8, 2025

Choose a reason for hiding this comment

azeey Jan 24, 2025

Choose a reason for hiding this comment

azeey Jan 24, 2025

Choose a reason for hiding this comment

AmalDevHaridevan Feb 7, 2025

Choose a reason for hiding this comment

azeey left a comment

Choose a reason for hiding this comment