Python 3.12: TypeError: cannot create weak reference to 'typing.TypeVar' object

[musicinmybrain]

python3.12 -m venv _e
. _e/bin/activate
pip install -e .
pip install -rdev-requirements.txt
python -m pytest

Sample test failure:

_______________________________________ CloudPickleTest.test_generic_subclass ________________________________________

self = <tests.cloudpickle_test.CloudPickleTest testMethod=test_generic_subclass>

    def test_generic_subclass(self):
        T = typing.TypeVar('T')

        class Base(typing.Generic[T]):
            pass

        class DerivedAny(Base):
            pass

        class LeafAny(DerivedAny):
            pass

        class DerivedInt(Base[int]):
            pass

        class LeafInt(DerivedInt):
            pass

        class DerivedT(Base[T]):
            pass

        class LeafT(DerivedT[T]):
            pass

        klasses = [
            Base, DerivedAny, LeafAny, DerivedInt, LeafInt, DerivedT, LeafT
        ]
        for klass in klasses:
>           assert pickle_depickle(klass, protocol=self.protocol) is klass

tests/cloudpickle_test.py:2423:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/cloudpickle_test.py:81: in pickle_depickle
    return pickle.loads(cloudpickle.dumps(obj, protocol=protocol))
cloudpickle/cloudpickle_fast.py:73: in dumps
    cp.dump(obj)
cloudpickle/cloudpickle_fast.py:632: in dump
    return Pickler.dump(self, obj)
cloudpickle/cloudpickle.py:909: in _typevar_reduce
    return (_make_typevar, _decompose_typevar(obj))
cloudpickle/cloudpickle.py:899: in _decompose_typevar
    _get_or_create_tracker_id(obj),
cloudpickle/cloudpickle.py:107: in _get_or_create_tracker_id
    class_tracker_id = _DYNAMIC_CLASS_TRACKER_BY_CLASS.get(class_def)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <WeakKeyDictionary at 0x7f089efba2a0>, key = ~T, default = None

    def get(self, key, default=None):
>       return self.data.get(ref(key),default)
E       TypeError: cannot create weak reference to 'typing.TypeVar' object

/usr/lib64/python3.12/weakref.py:452: TypeError

List of test failures:

============================================== short test summary info ===============================================
FAILED tests/cloudpickle_test.py::CloudPickleTest::test_generic_subclass - TypeError: cannot create weak reference to 'typing.TypeVar' object
FAILED tests/cloudpickle_test.py::CloudPickleTest::test_generic_type - TypeError: cannot create weak reference to 'typing.TypeVar' object
FAILED tests/cloudpickle_test.py::CloudPickleTest::test_locally_defined_class_with_type_hints - TypeError: cannot create weak reference to 'typing.TypeVar' object
FAILED tests/cloudpickle_test.py::CloudPickleTest::test_module_importability - ModuleNotFoundError: No module named 'distutils'
FAILED tests/cloudpickle_test.py::CloudPickleTest::test_pickle_constructs_from_module_registered_for_pickling_by_value - TypeError: cannot create weak reference to 'typing.TypeVar' object
FAILED tests/cloudpickle_test.py::CloudPickleTest::test_pickle_dynamic_typevar - TypeError: cannot create weak reference to 'typing.TypeVar' object
FAILED tests/cloudpickle_test.py::CloudPickleTest::test_pickle_dynamic_typevar_memoization - TypeError: cannot create weak reference to 'typing.TypeVar' object
FAILED tests/cloudpickle_test.py::CloudPickleTest::test_pickle_dynamic_typevar_tracking - TypeError: cannot create weak reference to 'typing.TypeVar' object
FAILED tests/cloudpickle_test.py::Protocol2CloudPickleTest::test_generic_subclass - TypeError: cannot create weak reference to 'typing.TypeVar' object
FAILED tests/cloudpickle_test.py::Protocol2CloudPickleTest::test_generic_type - TypeError: cannot create weak reference to 'typing.TypeVar' object
FAILED tests/cloudpickle_test.py::Protocol2CloudPickleTest::test_locally_defined_class_with_type_hints - TypeError: cannot create weak reference to 'typing.TypeVar' object
FAILED tests/cloudpickle_test.py::Protocol2CloudPickleTest::test_module_importability - ModuleNotFoundError: No module named 'distutils'
FAILED tests/cloudpickle_test.py::Protocol2CloudPickleTest::test_pickle_constructs_from_module_registered_for_pickling_by_value - TypeError: cannot create weak reference to 'typing.TypeVar' object
FAILED tests/cloudpickle_test.py::Protocol2CloudPickleTest::test_pickle_dynamic_typevar - TypeError: cannot create weak reference to 'typing.TypeVar' object
FAILED tests/cloudpickle_test.py::Protocol2CloudPickleTest::test_pickle_dynamic_typevar_memoization - TypeError: cannot create weak reference to 'typing.TypeVar' object
FAILED tests/cloudpickle_test.py::Protocol2CloudPickleTest::test_pickle_dynamic_typevar_tracking - TypeError: cannot create weak reference to 'typing.TypeVar' object
============================== 16 failed, 227 passed, 17 skipped, 12 warnings in 8.26s ===============================

Fedora Linux downstream issue

The No module named 'distutils' failures are a separate issue; distutils is removed from the standard library in Python 3.12, and it is necessary to either port away from it or introduce an explicit dependency on setuptools, which still provides it.

[ogrisel]

@musicinmybrain would you be interested in building cpython from the python/cpython#106403 branch by @encukou to check that this is enough for cloudpickle?

[musicinmybrain]

$ ./python -VV
Python 3.13.0a0 (remotes/JelleZijlstra/weakref:df59646acf, Jul  5 2023, 12:13:52) [GCC 13.1.1 20230614 (Red Hat 13.1.1-4)]

./python -m venv _e
. _e/bin/activate
cd ~/src/cloudpickle
pip install -e .
pip install -rdev-requirements.txt
python -m pytest

test.log

That fixed the issues reported here, but there were others, and there was an abort related to a weak reference. Next I’ll try cherry-picking JelleZijlstra/cpython@df59646 onto v3.12.0b3, since I suspect some of the new problems could related to other Python 3.13 work.

[musicinmybrain]

Next I’ll try cherry-picking JelleZijlstra/cpython@df59646 onto v3.12.0b3, since I suspect some of the new problems could related to other Python 3.13 work.

$ ./python -VV
Python 3.12.0b3 (heads/weakref-v3.12:e1f44c5319, Jul  5 2023, 12:25:17) [GCC 13.1.1 20230614 (Red Hat 13.1.1-4)]

Nope, it fails similarly to Python 3.13: test-py312.log

Four tests fail,

=========================== short test summary info ============================
FAILED tests/cloudpickle_test.py::CloudPickleTest::test_pickle_constructs_from_module_registered_for_pickling_by_value
FAILED tests/cloudpickle_test.py::CloudPickleTest::test_pickle_dynamic_typevar_tracking
FAILED tests/cloudpickle_test.py::Protocol2CloudPickleTest::test_pickle_constructs_from_module_registered_for_pickling_by_value
FAILED tests/cloudpickle_test.py::Protocol2CloudPickleTest::test_pickle_dynamic_typevar_tracking
============ 4 failed, 239 passed, 17 skipped, 5 warnings in 19.67s ============

and a traceback for a weakref-related segfault is printed during tests/cloudpickle_test.py::Protocol2CloudPickleTest::test_pickle_constructs_from _module_registered_for_pickling_by_value.

If I skip these four tests (python -m pytest -k 'not (CloudPickleTest and test_pickle_dynamic_typevar_tracking) and not (CloudPickleTest and test_pickle_constructs_from_module_registered_for_pickling_by_value) and not (Protocol2CloudPickleTest and test_pickle_constructs_from_module_registered_for_pickling_by_value) and not (Protocol2CloudPickleTest and test_pickle_dynamic_typevar_tracking)') the tests seem to pass, and the traceback disappears, but there a segfault is still reported at the end:

============================= test session starts ==============================
platform linux -- Python 3.12.0b3, pytest-7.4.0, pluggy-1.2.0
rootdir: /home/ben/src/cloudpickle
configfile: tox.ini
plugins: cov-4.1.0
collected 260 items / 4 deselected / 256 selected                              

tests/cloudpickle_file_test.py .......
tests/cloudpickle_test.py .................s...................s.........ss...........................................s..........................................s...................s.........ss...........................................s...............................
tests/test_backward_compat.py sssssss

=============================== warnings summary ===============================
tests/cloudpickle_test.py::CloudPickleTest::test_itertools_count
tests/cloudpickle_test.py::Protocol2CloudPickleTest::test_itertools_count
  /home/ben/src/cloudpickle/cloudpickle/cloudpickle_fast.py:632: DeprecationWarning: Pickle, copy, and deepcopy support will be removed from itertools in Python 3.14.
    return Pickler.dump(self, obj)

tests/cloudpickle_test.py::CloudPickleTest::test_tornado_coroutine
  /home/ben/src/cpython/_e/lib/python3.12/site-packages/tornado/ioloop.py:262: DeprecationWarning: There is no current event loop
    loop = asyncio.get_event_loop()

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
========== 239 passed, 17 skipped, 4 deselected, 3 warnings in 15.67s ==========
Segmentation fault (core dumped)

[ogrisel]

Pseudo-random ping to fellow cloudpickle contributors/maintainers: @mrocklin @jrbourbeau @jakirkham @robertnishihara @ueshin @HyukjinKwon @JoshRosen.

This issue with Python 3.12 will likely be disruptive for your downstream projects / businesses if left unattended. But I am personally running a bit out of spare cycles to investigate and report to CPython developers what's the root cause of the regression.

It would be great if some of you could have a look :)

[ogrisel]

I tried to run the cloudpickle tests with a fresh new conda env running Python 3.12.0 from conda-forge on my macOS laptop and I can not reproduce the segfault when running the tests.

Furthermore #514 has an updated CI that runs Python 3.12 on linux / windows / macos and all tests are green.

I think we can close this issue. Thanks @musicinmybrain for your work with upstream.