GH-129817: thread safety for tp_flags #130983
Draft
+166
−41
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Note: this PR needs a bit more polish before it becomes non-draft. I added Sam and Matt to the reviewers in case they have some initial feedback on this approach.
Use a separate structure member in
PyHeapTypeObject
,ht_flags
, to store a copy of the type flags. That allows safe toggling of some of the flags after the type has been initialized and potentially exposed to other threads.We would prefer to not use an atomic load whenever
tp_flags
is read. That causes a lot of code churn (see gh-130892) and potentially some performance hit on platforms with weak memory ordering. Instead, we would like to use a normal load and only set the flags before the type has been exposed to other threads. That's mostly the case except for the flags listed below.The following type flags cause issues in that they may be toggled after the type is initially created and exposed:
Py_TPFLAGS_SEQUENCE
andPy_TPFLAGS_MAPPING
. Toggled by assigning special methods to the type object.Py_TPFLAGS_IS_ABSTRACT
. Toggled by assigning a non-empty set to__abstractmethods__
Py_TPFLAGS_HAVE_VECTORCALL
. Toggled off (no way to turn back on) by assigning__call__
to the type.This PR does the following (only to the free-threaded build, the default build continues to work the same):
PyHeapTypeObject
,ht_flags
. This member only exists in the free-threaded build.tp_flags
value intoht_flags
whentype_ready()
completes.ht_flags
. Use atomic operations to read and write it._PyType_HasFeatureSafe()
to test the above flags. That function uses an atomic load.This approach causes a bit of extra complication since we have to check
Py_TPFLAGS_HEAPTYPE
first to know where to look at the flags. For non-heap types, they are intp_flags
always. This could be avoided by adding an additional member toPyTypeObject
. However, I think thePy_TPFLAGS_HEAPTYPE
check should be cheap enough and putting it inPyHeapTypeObject
ensures that extension types don't get confused by it.Note that since all non-heap types are immutable, it is not possible to toggle type flags for them (at least, CPython itself doesn't do so). Also note that this change can break extensions that manipulate
tp_flags
directly or directly tests them, rather than using functions.tp_flags
with subinterpreters and static types #129817