A lot of leaks in the test suite on the JIT build

[Eclips4]

Bug report

Bug description:

List of failed tests(I'm used ./python.exe -m test -R 3:3 to run the test suite):

319 tests failed:
    test.test_asyncio.test_base_events
    test.test_asyncio.test_buffered_proto
    test.test_asyncio.test_eager_task_factory
    test.test_asyncio.test_events test.test_asyncio.test_futures
    test.test_asyncio.test_futures2 test.test_asyncio.test_locks
    test.test_asyncio.test_pep492
    test.test_asyncio.test_proactor_events
    test.test_asyncio.test_queues test.test_asyncio.test_runners
    test.test_asyncio.test_selector_events
    test.test_asyncio.test_sendfile test.test_asyncio.test_server
    test.test_asyncio.test_sock_lowlevel test.test_asyncio.test_ssl
    test.test_asyncio.test_sslproto test.test_asyncio.test_staggered
    test.test_asyncio.test_streams test.test_asyncio.test_subprocess
    test.test_asyncio.test_taskgroups test.test_asyncio.test_tasks
    test.test_asyncio.test_threads test.test_asyncio.test_timeouts
    test.test_asyncio.test_unix_events test.test_asyncio.test_waitfor
    test.test_concurrent_futures.test_as_completed
    test.test_concurrent_futures.test_deadlock
    test.test_concurrent_futures.test_future
    test.test_concurrent_futures.test_process_pool
    test.test_concurrent_futures.test_thread_pool
    test.test_concurrent_futures.test_wait
    test.test_inspect.test_inspect
    test.test_multiprocessing_forkserver.test_manager
    test.test_multiprocessing_forkserver.test_misc
    test.test_multiprocessing_forkserver.test_processes
    test.test_multiprocessing_forkserver.test_threads
    test.test_multiprocessing_spawn.test_manager
    test.test_multiprocessing_spawn.test_misc
    test.test_multiprocessing_spawn.test_processes
    test.test_multiprocessing_spawn.test_threads
    test.test_pydoc.test_pydoc test___all__ test__interpchannels
    test__interpreters test__opcode test__osx_support test_abc
    test_argparse test_array test_asdl_parser test_ast test_asyncgen
    test_audit test_base64 test_baseexception test_bdb test_bigmem
    test_binascii test_binop test_bisect test_buffer test_bufio
    test_builtin test_bytes test_bz2 test_c_locale_coercion
    test_calendar test_call test_capi test_class test_clinic
    test_cmath test_cmd test_cmd_line test_cmd_line_script test_code
    test_code_module test_codeccallbacks test_codecencodings_cn
    test_codecencodings_hk test_codecencodings_iso2022
    test_codecencodings_jp test_codecencodings_kr
    test_codecencodings_tw test_codecs test_codeop test_collections
    test_colorsys test_compare test_compile test_compileall
    test_compiler_codegen test_complex test_configparser test_context
    test_contextlib test_contextlib_async test_coroutines test_csv
    test_ctypes test_dataclasses test_datetime test_dbm test_dbm_dumb
    test_dbm_sqlite3 test_decimal test_deque test_descr test_descrtut
    test_dict test_difflib test_dis test_doctest test_docxmlrpc
    test_email test_embed test_ensurepip test_enum test_errno
    test_except_star test_exception_group test_exceptions test_extcall
    test_faulthandler test_filecmp test_float test_fnmatch test_format
    test_fractions test_frame test_free_threading test_fstring
    test_ftplib test_functools test_gc test_generated_cases
    test_generators test_genericalias test_genexps test_getopt
    test_getpass test_getpath test_gettext test_glob test_grammar
    test_graphlib test_grp test_gzip test_hashlib test_heapq test_hmac
    test_html test_htmlparser test_http_cookiejar test_http_cookies
    test_httplib test_httpservers test_idle test_imaplib test_import
    test_int test_interpreters test_io test_ipaddress test_iterlen
    test_itertools test_json test_list test_listcomps test_locale
    test_logging test_long test_lzma test_mailbox test_marshal
    test_math test_memoryio test_metaclass test_mimetypes test_minidom
    test_mmap test_module test_modulefinder test_monitoring
    test_multiprocessing_main_handling test_named_expressions
    test_netrc test_ntpath test_numeric_tower test_opcache
    test_operator test_optparse test_ordered_dict test_os test_pathlib
    test_patma test_pdb test_peepholer test_pep646_syntax test_pkg
    test_pkgutil test_platform test_plistlib test_poll test_poplib
    test_positional_only_arg test_posix test_posixpath test_pow
    test_pprint test_property test_pstats test_pty test_pulldom
    test_pwd test_py_compile test_pyclbr test_pyexpat test_queue
    test_quopri test_random test_range test_re test_regrtest
    test_reprlib test_richcmp test_rlcompleter test_robotparser
    test_runpy test_sax test_script_helper test_selectors test_set
    test_setcomps test_shelve test_shlex test_shutil test_signal
    test_site test_slice test_smtplib test_socket test_sort
    test_sqlite3 test_ssl test_stable_abi_ctypes test_stat
    test_statistics test_str test_strftime test_string
    test_string_literals test_strptime test_strtod test_struct
    test_structseq test_subprocess test_super test_support
    test_symtable test_syntax test_sys test_sys_settrace test_tabnanny
    test_tarfile test_tcl test_tempfile test_textwrap test_thread
    test_threadedtempfile test_threading test_threading_local
    test_time test_tokenize test_tools test_traceback test_tracemalloc
    test_tuple test_type_aliases test_type_annotations test_type_cache
    test_type_comments test_type_params test_types test_typing
    test_ucn test_unicode_file_functions test_unicodedata
    test_unittest test_univnewlines test_unpack test_unpack_ex
    test_unparse test_urllib test_urllib2 test_urllib2_localnet
    test_urlparse test_userdict test_userlist test_userstring
    test_uuid test_venv test_warnings test_wave test_weakref
    test_weakset test_webbrowser test_wsgiref test_xml_etree
    test_xml_etree_c test_xmlrpc test_yield_from test_zipapp
    test_zipfile test_zipimport test_zipimport_support test_zlib
    test_zoneinfo

CPython versions tested on:

CPython main branch

Operating systems tested on:

macOS

Linked PRs

• gh-120501: Fix reference leak in JIT build #120649

[Eclips4]

Oh, I'm sorry, this also happens on non-jit build (just a --with-pydebug option).

[Eclips4]

Uhm, sorry again 😄 It's actually related to JIT build. I cannot reproduce it in pure debug build anymore. But on the JIT build, I can.

[JeffersGlass]

I'm not able to recreate this on Linux build on main - maybe Mac specific? (Well, a few skipped tests from missing extras, but not 300+ failures):

./make distclean && ./configure --enable-experimental-jit --with-pydebug && make -j8
./python -VV # Python 3.14.0a0 (heads/main:6af190f8d0, Jun 14 2024, 08:56:48) [GCC 11.4.0]
./python -m test -R 3:3 -j8

== Tests result: SUCCESS ==

12 tests skipped:
    test.test_asyncio.test_windows_events
    test.test_asyncio.test_windows_utils test_android test_devpoll
    test_kqueue test_launcher test_msvcrt test_startfile test_winapi
    test_winconsoleio test_winreg test_wmi

11 tests skipped (resource denied):
    test_curses test_peg_generator test_pyrepl test_smtpnet
    test_socketserver test_tkinter test_ttk test_urllib2net
    test_urllibnet test_winsound test_zipfile64

455 tests OK.

[Eclips4]

I'm not able to recreate this on Linux build on main (well, a few skipped tests from missing extras, but not 300+ failures):

./make distclean && ./configure --enable-experimental-jit --with-pydebug && make -j8
./python -VV # Python 3.14.0a0 (heads/main:6af190f8d0, Jun 14 2024, 08:56:48) [GCC 11.4.0]
./python -m test -R 3:3 -j8

== Tests result: SUCCESS ==

12 tests skipped:
    test.test_asyncio.test_windows_events
    test.test_asyncio.test_windows_utils test_android test_devpoll
    test_kqueue test_launcher test_msvcrt test_startfile test_winapi
    test_winconsoleio test_winreg test_wmi

11 tests skipped (resource denied):
    test_curses test_peg_generator test_pyrepl test_smtpnet
    test_socketserver test_tkinter test_ttk test_urllib2net
    test_urllibnet test_winsound test_zipfile64

455 tests OK.

Interesting. I can reproduce this on both of my WSL and macOS setups.
I'm using the command below to check it for leaks.

git clean -fdx && ./configure --with-pydebug --enable-experimental-jit && make -j && ./python.exe -m test -R 3:3 test___all__

[Eclips4]

Additionally, I can reproduce it on my Windows setup, using this command:

 pcbuild/build.bat -c Debug --experimental-jit ; ./python -m test -R 3:3 test___all__
 ...many lines of build logs
 Running Debug|x64 interpreter...
Using random seed: 2230268839
0:00:00 Run 1 test sequentially in a single process
0:00:00 [1/1] test___all__
beginning 6 repetitions. Showing number of leaks (. for 0 or less, X for 10 or more)
123:456
XXX XXX
test___all__ leaked [11, 11, 11] references, sum=33
test___all__ leaked [11, 11, 11] memory blocks, sum=33
test___all__ failed (reference leak)

== Tests result: FAILURE ==

1 test failed:
    test___all__

Total duration: 19.5 sec
Total tests: run=1
Total test files: run=1/1 failed=1
Result: FAILURE

[JeffersGlass]

Agreed, I'm running the full windows tests on a native build now and seeing tons of leaking tests. I'll be curious if the failure list is the same as on your original Mac build or different.

[JeffersGlass]

Just ran the Linux tests again sequentially (i.e. without -j8) to see if that made a difference. Again all tests pass on Linux (or skip).

[Eclips4]

Bisected to f6fab21
cc @markshannon

[JeffersGlass]

On Windows build, 314 tests failed:

./PCBuild/amd64/python_d.exe -VV #
./PCBuild/amg64/python_d.exe -m test -R 3:3 -j8

Total test files: run=468/478 failed=314 skipped=41 resource_denied=10         

[picnixz]

I would like to test on my machine but it incorrectly assumes that I need clang-18 instead of gcc. I'm using OpenSUSE 15.5 and my clang is 15.0.7. How can I actually force to use the gcc for JIT? (and is there any support for it actually?)

[Eclips4]

I would like to test on my machine but it incorrectly assumes that I need clang-18 instead of gcc. I'm using OpenSUSE 15.5 and my clang is 15.0.7. How can I actually force to use the gcc for JIT? (and is there any support for it actually?)

I think there's no support for gcc since our JIT is based on LLVM

[picnixz]

Oh, I thought that the JIT support explained in PEP 744 was for all Tier 1 platforms (especially "x86_64-unknown-linux-gnu/gcc") but I still need LLVM and Clang. Ok, I'll install local versions for that.

[Eclips4]

@picnixz Actually we have a README for JIT, which is located at Tools/jit/README.md 🙂

[picnixz]

(Yes, I've seen the README after actually. I first looked at the PEP to see whether my platform was supported or not and was surprised)

[sobolevn]

Is this rather new?

 ======================================================================
FAIL: test_strftime_y2k (test.datetimetester.TestSubclassDateTime_Fast.test_strftime_y2k) (year=999, specifier='F')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/cpython/cpython/Lib/test/datetimetester.py", line 1728, in test_strftime_y2k
    self.assertEqual(d.strftime(f"%{specifier}"), expected)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: '999-01-01' != '0999-01-01'
- 999-01-01
+ 0999-01-01
? +


======================================================================
FAIL: test_strftime_y2k (test.datetimetester.TestSubclassDateTime_Fast.test_strftime_y2k) (year=999, specifier='C')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/cpython/cpython/Lib/test/datetimetester.py", line 1728, in test_strftime_y2k
    self.assertEqual(d.strftime(f"%{specifier}"), expected)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: '9' != '09'
- 9
+ 09
? +


----------------------------------------------------------------------
Ran 8 tests in 0.567s

FAILED (failures=48)
test test_datetime failed
1 test failed again:
    test_datetime

== Tests result: FAILURE then FAILURE ==

38 tests skipped:
    test.test_asyncio.test_windows_events
    test.test_asyncio.test_windows_utils test.test_gdb.test_backtrace
    test.test_gdb.test_cfunction test.test_gdb.test_cfunction_full
    test.test_gdb.test_misc test.test_gdb.test_pretty_print
    test_android test_bz2 test_ctypes test_dbm_gnu test_dbm_ndbm
    test_dbm_sqlite3 test_devpoll test_free_threading test_gzip
    test_idle test_ioctl test_kqueue test_launcher test_lzma
    test_msvcrt test_readline test_smtpnet test_sqlite3 test_ssl
    test_stable_abi_ctypes test_startfile test_tcl test_tkinter
    test_ttk test_ttk_textonly test_turtle test_winapi
    test_winconsoleio test_winreg test_wmi test_zlib

8 tests skipped (resource denied):
    test_curses test_peg_generator test_pyrepl test_socketserver
    test_urllib2net test_urllibnet test_winsound test_zipfile64

5 tests run no tests:
    test.test_multiprocessing_fork.test_manager
    test.test_multiprocessing_fork.test_misc
    test.test_multiprocessing_fork.test_processes
    test.test_multiprocessing_fork.test_threads
    test_external_inspection

1 re-run test:
    test_datetime

1 test failed:
    test_datetime

Log: https://github.com/sobolevn/cpython/actions/runs/10603373267/job/29387830804

[picnixz]

I think it was #120713.