import of paho.mqtt.client not possible when ONLY importing in subthread

[KonstantinWaser]

Bug Description

When running a python script and trying to import paho.mqtt.client in a subthread, without first importing the library in the main thread, an exception occurs. If the library is imported before in the main thread, no problems occurr.

Reproduction

This simple code snippet:

import threading
def mqtt_thread():
    print("Starting MQTT")
    import paho.mqtt.client as mqtt
    print("MQTT imported in thread")

t = threading.Thread(target=mqtt_thread)
t.start()

executes up to the first print statement, I see the "Starting MQTT" message, then the following exception occurrs:
RuntimeError: can't register atexit after shutdown

When doing the same without the subthread, there are no problems at all. Also, when first importing the package in the main thread, like here:

import threading
import paho.mqtt.client as mqtt
def mqtt_thread():
    print("Starting MQTT")
    **import paho.mqtt.client as mqtt**
    print("MQTT imported in thread")

t = threading.Thread(target=mqtt_thread)
t.start()

also works just fine.

Environment

• Python version: tested on 3.11.3 and 3.11.9
• Library version: 2.1.0
• Operating system: Windows 11 (tested on different versions)

Whole Stacktrace:

Starting MQTT
Exception in thread Thread-1 (mqtt_thread):
Traceback (most recent call last):
File "C:\Python\311_x64\Lib\threading.py", line 1045, in _bootstrap_inner
self.run()
File "C:\Python\311_x64\Lib\threading.py", line 982, in run
self._target(*self._args, **self._kwargs)
File "c:\Users\KonstantinWaser\Desktop\pahotest.py", line 18, in mqtt_thread
import paho.mqtt.client as mqtt
File "C:\Python\311_x64\Lib\site-packages\paho\mqtt\client.py", line 116, in
import dns.resolver
File "C:\Python\311_x64\Lib\site-packages\dns\resolver.py", line 30, in
import dns._ddr
File "C:\Python\311_x64\Lib\site-packages\dns_ddr.py", line 12, in
import dns.nameserver
File "C:\Python\311_x64\Lib\site-packages\dns\nameserver.py", line 5, in
import dns.asyncquery
File "C:\Python\311_x64\Lib\site-packages\dns\asyncquery.py", line 38, in
from dns.query import (
File "C:\Python\311_x64\Lib\site-packages\dns\query.py", line 64, in
import httpcore.backends.sync
File "C:\Python\311_x64\Lib\site-packages\httpcore_init.py", line 1, in
from ._api import request, stream
File "C:\Python\311_x64\Lib\site-packages\httpcore_api.py", line 5, in
from .sync.connection_pool import ConnectionPool
File "C:\Python\311_x64\Lib\site-packages\httpcore_sync_init.py", line 1, in
from .connection import HTTPConnection
File "C:\Python\311_x64\Lib\site-packages\httpcore_sync\connection.py", line 12, in
from ..synchronization import Lock
File "C:\Python\311_x64\Lib\site-packages\httpcore_synchronization.py", line 16, in
import anyio
File "C:\Python\311_x64\Lib\site-packages\anyio_init.py", line 21, in
from ._core.fileio import AsyncFile as AsyncFile
File "C:\Python\311_x64\Lib\site-packages\anyio_core_fileio.py", line 21, in
from .. import to_thread
File "C:\Python\311_x64\Lib\site-packages\anyio\to_thread.py", line 9, in
from .abc import CapacityLimiter
File "C:\Python\311_x64\Lib\site-packages\anyio\abc_init.py", line 50, in
from ..from_thread import BlockingPortal as BlockingPortal
File "C:\Python\311_x64\Lib\site-packages\anyio\from_thread.py", line 6, in
from concurrent.futures import FIRST_COMPLETED, Future, ThreadPoolExecutor, wait
File "", line 1229, in handle_fromlist
File "C:\Python\311_x64\Lib\concurrent\futures_init.py", line 49, in getattr
from .thread import ThreadPoolExecutor as te
File "C:\Python\311_x64\Lib\concurrent\futures\thread.py", line 37, in
threading._register_atexit(_python_exit)
File "C:\Python\311_x64\Lib\threading.py", line 1527, in _register_atexit
raise RuntimeError("can't register atexit after shutdown")
RuntimeError: can't register atexit after shutdown

[rubenbaer]

Can you provide all package versions using pip freeze? This might not be a paho mqtt issue.

[KonstantinWaser]

absl-py==2.1.0
ace_tools==0.0
aiofiles==24.1.0
aiohttp==3.9.5
aiosignal==1.3.1
alembic==1.13.2
altgraph==0.17.4
annotated-types==0.6.0
anyio==4.3.0
appdirs==1.4.4
Appium-Python-Client==0.24
argon2-cffi==23.1.0
argon2-cffi-bindings==21.2.0
arrow==1.3.0
asttokens==2.4.1
astunparse==1.6.3
async-lru==2.0.4
attrs==23.2.0
azure-core==1.32.0
azure-identity==1.19.0
azure-storage-blob==12.24.1
Babel==2.14.0
bcrypt==4.1.2
beautifulsoup4==4.12.3
bidict==0.23.1
bleach==6.1.0
blinker==1.7.0
blis==0.7.11
bokeh==3.6.2
Brotli==1.1.0
bs4==0.0.2
cached-property==1.5.2
cachetools==5.3.3
catalogue==2.0.10
category-encoders==2.6.3
certifi==2024.2.2
cffi==1.16.0
charset-normalizer==3.3.2
chilkat==10.1.2
ci-info==0.3.0
click==8.1.7
cloudpathlib==0.18.1
cloudpickle==3.0.0
clr-loader==0.2.6
colorama==0.4.6
comm==0.2.2
confection==0.1.5
ConfigArgParse==1.7
configobj==5.0.9
configparser==7.1.0
construct==2.10.70
contourpy==1.2.1
cornac==1.18.0
cramjam==2.9.1
crate==0.35.2
cryptography==42.0.5
CTkMessagebox==2.5
CTkToolTip==0.8
customtkinter==5.2.2
cycler==0.12.1
cymem==2.0.8
darkdetect==0.8.0
debugpy==1.8.1
decorator==5.1.1
defusedxml==0.7.1
dill==0.3.8
dnspython==2.6.1
docutils==0.19
easyocr==1.7.1
EasyProcess==1.1
ebrec==0.0.1
entrypoint2==1.1
et-xmlfile==1.1.0
etelemetry==0.3.1
exchangelib==5.4.3
executing==2.0.1
fastai==2.7.15
fastapi==0.109.2
fastcore==1.5.45
fastdownload==0.0.7
fastjsonschema==2.19.1
fastparquet==2024.11.0
fastprogress==1.0.3
filelock==3.13.4
Flask==3.0.2
Flask-Cors==4.0.1
Flask-Login==0.6.3
Flask-Migrate==4.0.7
Flask-SQLAlchemy==3.1.1
Flask-WTF==1.2.1
flatbuffers==24.3.25
fonttools==4.51.0
fqdn==1.5.1
frozenlist==1.4.1
fsspec==2024.3.1
future==1.0.0
gast==0.5.4
gensim==4.3.3
geojson==3.1.0
gevent==24.2.1
geventhttpclient==2.2.1
google-auth==2.30.0
google-auth-oauthlib==1.2.0
google-pasta==0.2.0
greenlet==3.0.3
grpcio==1.62.2
h11==0.14.0
h2==4.1.0
h5py==3.11.0
hdbcli==2.19.21
hpack==4.0.0
html5lib==1.1
htmlmin==0.1.12
htmx==0.0.0
httpcore==1.0.5
httplib2==0.22.0
httptools==0.6.1
httpx==0.27.0
huggingface-hub==0.22.2
hyperframe==6.0.1
hyperopt==0.2.7
hypothesis==6.100.4
idna==3.6
ifaddr==0.2.0
ImageHash==4.3.1
imageio==2.34.2
imbalanced-learn==0.13.0
importlib-metadata==7.0.1
iniconfig==2.0.0
intel-openmp==2021.4.0
ipykernel==6.29.4
ipython==8.24.0
ipywidgets==8.1.5
IronPdf==2024.9.1.3
isodate==0.6.1
isoduration==20.11.0
itsdangerous==2.1.2
jaraco.classes==3.3.1
jedi==0.19.1
Jinja2==3.1.4
joblib==1.4.2
json5==0.9.25
jsonpointer==2.4
jsonschema==4.22.0
jsonschema-specifications==2023.12.1
jupyter-events==0.10.0
jupyter-lsp==2.2.5
jupyter_client==8.6.1
jupyter_core==5.7.2
jupyter_server==2.14.0
jupyter_server_terminals==0.5.3
jupyterlab==4.1.8
jupyterlab_pygments==0.3.0
jupyterlab_server==2.27.1
jupyterlab_widgets==3.0.13
kaitaistruct==0.10
keras==2.15.0
keyring==24.3.0
kiwisolver==1.4.5
langcodes==3.4.0
language_data==1.2.0
lazy_loader==0.4
libclang==18.1.1
lightfm==1.17
lightgbm==4.3.0
llvmlite==0.42.0
locust==2.26.0
looseversion==1.3.0
lxml==5.1.0
Mako==1.3.5
marisa-trie==1.2.0
Markdown==3.6
markdown-it-py==3.0.0
markdown2==2.4.13
MarkupSafe==2.1.5
matplotlib==3.8.4
matplotlib-inline==0.1.7
mdurl==0.1.2
mechanize==0.4.10
memory-profiler==0.61.0
missingno==0.5.2
mistune==3.0.2
mkl==2021.4.0
ml-dtypes==0.3.2
modbus-tk==1.1.3
more-itertools==10.2.0
mplcursors==0.6
mpmath==1.3.0
msal==1.31.1
msal-extensions==1.2.0
msgpack==1.0.8
mss==9.0.1
multidict==6.0.5
multimethod==1.10
murmurhash==1.0.10
mypy==1.9.0
mypy-extensions==1.0.0
mysql-connector-python==8.3.0
mystring==0.2.252
namex==0.0.8
nbclient==0.10.0
nbconvert==7.16.4
nbformat==5.10.4
nest-asyncio==1.6.0
networkx==3.2.1
nibabel==5.2.1
nicegui==1.4.28
ninja==1.11.1.1
nipype==1.8.6
nltk==3.8.1
node2vec==0.5.0
notebook==7.1.3
notebook_shim==0.2.4
numba==0.59.1
numpy==1.26.4
nvidia-ml-py==12.555.43
oauthlib==3.2.2
opencv-python-headless==4.10.0.84
openpyxl==3.1.2
opt-einsum==3.3.0
optree==0.11.0
oracledb==2.5.0
orjson==3.10.5
outcome==1.3.0.post0
overrides==7.7.0
packaging==23.2
pafy==0.5.5
paho-mqtt==2.1.0
pandas==2.2.1
pandas-profiling==3.2.0
pandas-stubs==2.2.0.240218
pandas_ui==0.1
pandasql==0.7.3
pandera==0.18.3
pandocfilters==1.5.1
paramiko==3.4.0
parso==0.8.4
pathlib==1.0.1
patsy==0.5.6
pdf2image==1.17.0
pdfminer.six==20231228
pdfplumber==0.11.2
pefile==2023.2.7
phik==0.12.4
pillow==10.4.0
platformdirs==4.2.1
plotly==5.22.0
pluggy==1.4.0
polars==0.20.21
portalocker==2.10.1
powerlaw==1.5
preshed==3.0.9
prometheus_client==0.20.0
prompt-toolkit==3.0.43
protobuf==4.25.3
prov==2.0.1
pscript==0.7.7
psutil==5.9.8
pure-eval==0.2.2
py-machineid==0.5.1
py4j==0.10.9.7
pyarmor==8.5.12
pyarmor.cli.core==6.5.3
pyarrow==19.0.0
pyasn1==0.5.1
pyasn1_modules==0.4.0
pyclipper==1.3.0.post5
pycountry==24.6.1
pycparser==2.21
pycryptodome==3.20.0
pycryptodomex==3.20.0
pydantic==1.10.17
pydantic_core==2.18.2
pydivert==2.1.0
pydot==3.0.2
pygame==2.6.1
Pygments==2.17.2
pyinstaller==6.4.0
pyinstaller-hooks-contrib==2024.1
PyJWT==2.8.0
pykeepass==4.0.6
pymongo==4.6.2
PyMuPDF==1.24.11
PyMuPDFb==1.24.6
PyNaCl==1.5.0
pyodbc==5.1.0
pyOpenSSL==24.0.0
pyparsing==3.1.1
PyPDF2==3.0.1
pypdfium2==4.30.0
pyperclip==1.8.2
pyplotlib==0.0.56
pyrfc==3.3.1
pyscreenshot==3.1
pyserial==3.5
PySocks==1.7.1
pyspark==3.5.1
pyspnego==0.10.2
pytesseract==0.3.10
pytest==8.0.2
python-bidi==0.4.2
python-dateutil==2.8.2
python-dotenv==1.0.1
python-engineio==4.9.1
python-json-logger==2.0.7
python-multipart==0.0.9
python-socketio==5.11.3
python-vlc==3.0.21203
pythonnet==3.0.4
pytz==2024.1
PyWavelets==1.8.0
pywin32==306
pywin32-ctypes==0.2.2
pywinpty==2.0.13
pyxnat==1.6.2
PyYAML==6.0.1
pyzmq==26.0.3
qgrid==1.3.1
rdflib==6.3.2
recommenders==1.2.0
referencing==0.35.1
regex==2024.4.16
requests==2.32.3
requests-mock==1.12.1
requests-ntlm==1.2.0
requests-oauthlib==2.0.0
retrying==1.3.4
rfc3339-validator==0.1.4
rfc3986-validator==0.1.1
rich==13.7.1
roundrobin==0.0.4
rpds-py==0.18.0
rsa==4.9
safetensors==0.4.3
schedule==1.2.1
scikit-image==0.24.0
scikit-learn==1.4.2
scikit-surprise==1.1.3
scipy==1.12.0
seaborn==0.13.2
selenium==3.5.0
selenium-wire==5.1.0
Send2Trash==1.8.3
shapely==2.0.4
shellingham==1.5.4
siemens-pac==0.20
simple-websocket==1.0.0
simplejson==3.19.3
six==1.16.0
sklearn-compat==0.1.3
smart-open==7.0.4
sniffio==1.3.1
sortedcontainers==2.4.0
soupsieve==2.5
spacy==3.7.5
spacy-legacy==3.0.12
spacy-loggers==1.0.5
SQLAlchemy==2.0.27
sqlalchemy-access==2.0.2
sqlalchemy-hana==1.3.0
srsly==2.4.8
sspilib==0.1.0
stack-data==0.6.3
starlette==0.36.3
statsmodels==0.14.2
sympy==1.12.1
tangled-up-in-unicode==0.2.0
tbb==2021.12.0
tenacity==8.3.0
tensorboard==2.15.2
tensorboard-data-server==0.7.2
tensorflow==2.15.1
tensorflow-estimator==2.15.0
tensorflow-intel==2.15.1
tensorflow-io-gcs-filesystem==0.31.0
termcolor==2.4.0
terminado==0.18.1
tf-slim==1.1.0
thinc==8.2.4
threadpoolctl==3.5.0
tifffile==2024.7.2
tinycss2==1.3.0
tokenizers==0.19.1
torch==2.3.1
torchvision==0.18.1
tornado==6.4
tqdm==4.66.2
traitlets==5.14.3
traits==6.3.2
transformers==4.40.0
trio==0.24.0
trio-websocket==0.11.1
typeguard==4.2.1
typer==0.12.3
types-docutils==0.20.0.20240227
types-Pillow==10.2.0.20240213
types-Pygments==2.17.0.20240106
types-python-dateutil==2.9.0.20240316
types-pytz==2024.1.0.20240203
types-setuptools==69.1.0.20240223
typing-inspect==0.9.0
typing_extensions==4.10.0
tzdata==2024.1
tzlocal==5.2
undetected-chromedriver==3.5.5
uri-template==1.3.0
urllib3==2.1.0
usersettings==1.1.5
uvicorn==0.30.1
vboxapi==1.0
vbuild==0.8.2
verlib2==0.2.0
visions==0.7.4
wasabi==1.1.3
watchfiles==0.22.0
wcwidth==0.2.13
weasel==0.4.1
webcolors==1.13
webencodings==0.5.1
websocket-client==1.8.0
websockets==12.0
Werkzeug==3.0.1
widgetsnbextension==4.0.13
winregistry==1.1.1
wrapt==1.14.1
wsproto==1.2.0
WTForms==3.1.2
xlrd==2.0.1
xyzservices==2025.1.0
yarl==1.9.4
youtube-dl==2021.12.17
zipp==3.17.0
zope.event==5.0
zope.interface==6.3
zstandard==0.22.0

[rubenbaer]

Thanks.

I just cooked up a more basic version. Note that this must run as a script and not in a REPL.

import threading

def run():
    import concurrent.futures.thread

threading.Thread(target=run).start()

There are no dependencies needed and thus it looks like a python issue. There might be other issues out there related to this.

To trigger this as written in the issue, requires some packages like dns.resolver, anyio and and others.

[PierreF]

The exception give you the hints of the problem: RuntimeError: can't register atexit after shutdown

You stopped the Python main thread (which I should be "after shutdown")

Slightly extended version of last sample

import time
import threading

def run():
    time.sleep(1) # simulate some expensive computation
    print("I'm called after main thread shutdown")
    import concurrent.futures.thread

threading.Thread(target=run).start()
print("main thread will shutdown now")

You should wait for the thread:

import threading

def run():
    import concurrent.futures.thread

t = threading.Thread(target=run)
t.start()
t.join()