Attempting to dill a function defined in a doctest run with pytest causes a TypeError: cannot pickle 'EncodedFile' object.

[qthequartermasterman]

This is cross posted from pytest-dev/pytest#12448, because the pytest devs suggest that this is a problem with dill.

Description of the Problem

When I attempt to dill a function defined inside of a doctest, pytest raises an error, complaining that TypeError: cannot pickle 'EncodedFile' object. Best I can tell, this is the EncodedFile object in pytest.

This error only occurs when I run those doctests via pytest. They run just fine via vanilla doctest. The same code also works just fine in a script and as a regular pytest test function.

I fear that this is related to pytest-dev/pytest#10845, which doesn't seem to have a solution.

Minimal Reproducible Example (Serialization)

class MyClass:
    """Running a file containing this class with `python -m pytest this_file.py --doctest-modules` will fail with `TypeError: cannot pickle 'EncodedFile' object`.

    Examples:
        >>> def template_function():
        ...     return "Hello, World!"

        >>> import dill
        >>> dill.dumps(template_function)
    """

Using pytest to run those doctests will cause the TypeError: cannot pickle 'EncodedFile' object.

Using vanilla doctest works fine. (Ignore the fact that the test failed. The expected output did not match. Note that it did successfully dump the function using dill.)

Failed example:
dill.dumps(template_function)
Expected nothing
Got:
b'\x80\x04\x95L\x03\x00\x00\x00\x00\x00\x00\x8c\ndill.dill\x94\x8c\x10_create_function\x94\x93\x94(h\x00\x8c\x0c_create_code\x94\x93\x94(C\x02\x02\x01\x94K\x00K\x00K\x00K\x00K\x01K\x03C\x06\x97\x00d\x01S\x00\x94N\x8c\rHello, World!\x94\x86\x94))\x8c <doctest test_client.MyClass[0]>\x94\x8c\x11template_function\x94h\nK\x01C\x07\x80\x00\xd8\x0b\x1a\x88?\x94C\x00\x94))t\x94R\x94}\x94\x8c\x08__name_\x94\x8c\x0btest_client\x94sh\nNNt\x94R\x94}\x94}\x94\x8c\x0f__annotations__\x94}\x94s\x86\x94bh\x0f(h\x10h\x11\x8c\x07__doc__\x94N\x8c\x0b__package__\x94\x8c\x00\x94\x8c\n__loader__\x94\x8c\x1a_frozen_importlib_external\x94\x8c\x10SourceFileLoader\x94\x93\x94)\x81\x94}\x94(\x8c\x04name\x94h\x11\x8c\x04path\x94\x8c@/Users/username/test_client.py\x94ub\x8c\x08__spec__\x94\x8c\x11_frozen_importlib\x94\x8c\nModuleSpec\x94\x93\x94)\x81\x94}\x94(h"h\x11\x8c\x06loader\x94h \x8c\x06origin\x94h$\x8c\x0cloader_state\x94N\x8c\x1asubmodule_search_locations\x94N\x8c\x19_uninitialized_submodules\x94]\x94\x8c\r_set_fileattr\x94\x88\x8c\x07_cached\x94\x8cY/Users/username/pycache/test_client.cpython-311.pyc\x94\x8c\r_initializing\x94\x89ub\x8c\x08__file__\x94h$\x8c\n__cached__\x94h3\x8c\x0c__builtins__\x94cbuiltins\n__dict__\n\x8c\x07MyClass\x94h\x11h8\x93\x94h\nh\x13\x8c\x04dill\x94h\x00\x8c\x0e_import_module\x94\x93\x94h:\x85\x94R\x94u0.'

---

1 items had failures:
1 of 3 in test_client.MyClass

Using a regular pytest function to perform the test also works fine. This is only when attempting to dill a function defined in a doctest run with pytest.

Minimal Reproducible Example (Deserialization after disabling pytest's logging capture)

If I extend the example to include dill.loads, and then disable capture with pytest -s, I get a recursion error. The serialization is successful (because disabling capture means that EncodedFile isn't injected), but dill is trying to load something recursively, I think? Again, this code works as expected with vanilla doctest and in a script.

class MyClass:
    """Running a file containing this class with `python -m pytest  -s this_file.py --doctest-modules` will fail with `RecursionError('maximum recursion depth exceeded').

    Examples:
        >>> def template_function():
        ...     return "Hello, World!"

        >>> import dill
        >>> string = dill.dumps(template_function)
        >>> dill.loads(string)()
        'Hello, World!'
    """

010         >>> string = dill.dumps(template_function)
011         >>> dill.loads(string)()
UNEXPECTED EXCEPTION: RecursionError('maximum recursion depth exceeded')
Traceback (most recent call last):
  File "/opt/homebrew/Caskroom/miniforge/base/envs/my-env/lib/python3.11/doctest.py", line 1350, in __run
    exec(compile(example.source, filename, "single",
  File "<doctest test_client.MyClass[3]>", line 1, in <module>
  File "/opt/homebrew/Caskroom/miniforge/base/envs/my-env/lib/python3.11/site-packages/dill/_dill.py", line 286, in loads
    return load(file, ignore, **kwds)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniforge/base/envs/my-env/lib/python3.11/site-packages/dill/_dill.py", line 272, in load
    return Unpickler(file, ignore=ignore, **kwds).load()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniforge/base/envs/my-env/lib/python3.11/site-packages/dill/_dill.py", line 419, in load
    obj = StockUnpickler.load(self)
          ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniforge/base/envs/my-env/lib/python3.11/site-packages/pluggy/_manager.py", line 74, in __getattr__
    return getattr(self._dist, attr, default)
                   ^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniforge/base/envs/my-env/lib/python3.11/site-packages/pluggy/_manager.py", line 74, in __getattr__
    return getattr(self._dist, attr, default)
                   ^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniforge/base/envs/my-env/lib/python3.11/site-packages/pluggy/_manager.py", line 74, in __getattr__
    return getattr(self._dist, attr, default)
                   ^^^^^^^^^^
  [Previous line repeated 955 more times]
RecursionError: maximum recursion depth exceeded

Environment

This was run on Pytest 8.2.2, Python 3.11 on an M1 Macbook Air (macOS Sonoma 14.4.1). The same errors were also seen on Python 3.9 on the same Macbook and on Python 3.11 on an amd64 Ubuntu Machine (Ubuntu 22.04 LTS). All experiments were run with dill==0.3.8.

Pip list

Package    Version
---------- -------
coverage   7.5.3
dill       0.3.8
iniconfig  2.0.0
packaging  24.1
pip        24.0
pluggy     1.5.0
pytest     8.2.2
pytest-cov 5.0.0
setuptools 70.0.0
uv         0.2.10
wheel      0.43.0