Official support for python 3.12 and code clean-up after EOLed Python 3.6 and 3.7. #517

ogrisel · 2023-10-09T16:53:55Z

This is a takeover of #514 by @hugovk to be able to:

run the CI with the result of the merge with master,
update the changelog,
do more dead-code clean-up after the update of the minimum Python version requirement.

codecov · 2023-10-09T16:55:41Z

Codecov Report

Attention: 50 lines in your changes are missing coverage. Please review.

Comparison is base (45dcb75) 83.44% compared to head (dc794d1) 88.90%.

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #517      +/-   ##
==========================================
+ Coverage   83.44%   88.90%   +5.46%     
==========================================
  Files           4        3       -1     
  Lines         725      550     -175     
  Branches      157      114      -43     
==========================================
- Hits          605      489     -116     
+ Misses         95       48      -47     
+ Partials       25       13      -12

Files	Coverage Δ
cloudpickle/__init__.py	`100.00% <100.00%> (ø)`
cloudpickle/cloudpickle_fast.py	`100.00% <100.00%> (+8.38%)`	⬆️
cloudpickle/cloudpickle.py	`88.80% <84.32%> (+12.41%)`	⬆️

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

…thon 3.8+

… observe on my dev machine

…files

…kle_version.txt file

ogrisel · 2023-10-10T16:10:06Z

@HyukjinKwon I cleaned up a lot of dead code. As a result, the code base is much simpler and tests/test_backward_compat.py still pass with Python 3.8 and I added new compat pickle files for Python 3.9 and cloudpickle 1.4.1.

Please feel free to have a look at this PR to let me know if you see potential problems.

I still haven't tested with the downstream tests since their config need an update. I think we should do that before merging this PR. I will try to work a distinct PR against master to do that.

I would also like to blackify the code base before the release, but again in a dedicated PR to be able to remove the blackification commit from the git blame history.

cloudpickle/cloudpickle_fast.py

HyukjinKwon · 2023-10-12T00:09:10Z

(I will likely be able to test this w/ PySpark few weeks later. would be great if we don't block this by me, and merge this first - I will report/fix separately if there's any issue)

ogrisel · 2023-10-12T13:44:28Z

Thank you very much @HyukjinKwon. BTW, if you know a way to run a fraction of the tests of PySpark with a patched nightly build with the dev version of cloudpickle as we do for Ray in #519 that would help automate this process.

codecov · 2023-10-12T14:00:11Z

Codecov Report

Attention: 11 lines in your changes are missing coverage. Please review.

Comparison is base (45dcb75) 83.44% compared to head (c5c0b44) 96.07%.

Additional details and impacted files

@@             Coverage Diff             @@
##           master     #517       +/-   ##
===========================================
+ Coverage   83.44%   96.07%   +12.62%     
===========================================
  Files           4        3        -1     
  Lines         725      560      -165     
  Branches      157      116       -41     
===========================================
- Hits          605      538       -67     
+ Misses         95       11       -84     
+ Partials       25       11       -14

Files	Coverage Δ
cloudpickle/__init__.py	`100.00% <100.00%> (ø)`
cloudpickle/cloudpickle_fast.py	`100.00% <100.00%> (+8.38%)`	⬆️
cloudpickle/cloudpickle.py	`96.02% <96.56%> (+19.63%)`	⬆️

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

… file

ogrisel · 2023-10-12T15:00:49Z

Yay, downstream CI is green!

…n 3.12

…h is also part of the public API

…e_testpkg

ogrisel · 2023-10-12T15:51:30Z

@pierreglaser I think this should be ready for review if your have the time.

pierreglaser

Looking great, thank you @ogrisel! Just one minor comment.

cloudpickle/cloudpickle.py

tests/cloudpickle_test.py

dev-requirements.txt

ogrisel · 2023-10-13T09:28:41Z

Let me merge and follow-up with a PR dedicated to using black to enforce a consistent code style automatically.

### What changes were proposed in this pull request? This PR proposes to upgrade Cloudpickle from 2.2.1 to 3.0.0. ### Why are the changes needed? It includes official support of Python 3.12 (cloudpipe/cloudpickle#517) ### Does this PR introduce _any_ user-facing change? It includes official support of Python 3.12 (cloudpipe/cloudpickle#517) ### How was this patch tested? Relies on cloudpickle's unittests. Existing test cases should pass too. Closes #43989 from HyukjinKwon/SPARK-46080. Authored-by: Hyukjin Kwon <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

HyukjinKwon · 2023-11-24T02:07:43Z

Apologies for being late. I tested all, and at least all CI seems working fine with PySpark. We upgraded cloudpickle to 3.0.0 for Apache Spark 4.0.0.

hugovk and others added 6 commits September 13, 2023 22:55

pytest: drop the dot

da5e30a

Add support for Python 3.12

eb2231f

Drop support for EOL Python 3.6 and 3.7

fa838d1

Upgrade Python syntax with pyupgrade --py38-plus

99ed86f

Merge branch 'master' into official-support-for-python-3.12

8ce7386

Add changelog entry

b40f270

ogrisel added 6 commits October 10, 2023 10:12

Remove dead code now that Python 3.8+ is required

48d9d0a

Remove more useless code and test harness now that we only support Py…

e3d7aca

…thon 3.8+

Adapt test_interactive_remote_function_calls_no_memory_leak to what I…

0eb8805

… observe on my dev machine

Remove cell_set now that it's a trivial operation

e58fb34

Fix code style in test while and improve test coverage

556a3ca

Update non-regression test for modules without a __package__ attribute

e82ee46

ogrisel changed the title ~~Official support for python 3.12~~ Official support for python 3.12 and code clean-up after EOLed Python 3.6 and 3.7. Oct 10, 2023

ogrisel added 6 commits October 10, 2023 11:32

Reformat inline comment

d4688b0

More dead code cleanup

4668392

Fix linting errors

adbe0d3

Store the cloudpickle version when generating backward compat pickle …

796161e

…files

Forgot to include the generated tests/old_pickles/cpython_38/cloudpic…

8549e57

…kle_version.txt file

Generate test compat pickle files for Python 3.9 and cloudpickle 1.4.1

6b23265

ogrisel mentioned this pull request Oct 10, 2023

Add support for Python 3.12 and drop EOL 3.6 and 3.7 #514

Closed

ogrisel added 8 commits October 10, 2023 14:54

More code style cleanups

7c12a86

More dead code cleanup

39ae541

Given the amount of code change, let's make this a major release

319cfb8

Use ruff instead of flake8_diff.sh

b69d56c

Dead code in _code_reduce

e0dff44

Remove the cloudpickle.compat submodule

b508da4

More dead code cleanup

48df2f2

Simplify test to remove Python 3.5 compat boilerplate

ceb43da

ogrisel commented Oct 11, 2023

View reviewed changes

cloudpickle/cloudpickle_fast.py Show resolved Hide resolved

ogrisel added 2 commits October 11, 2023 16:35

Link to tests/test_backward_compat.py in compat module.

ab85b0f

More consistent style for docstrings.

9e43da6

ogrisel mentioned this pull request Oct 12, 2023

Fix downstream CI #519

Merged

Merge branch 'master' into official-support-for-python-3.12

4512710

ogrisel added the ci downstream Signal the CI to run the test suite of all registered cloudpickle downstream projects. label Oct 12, 2023

Trigger CI

dc794d1

ogrisel added 2 commits October 12, 2023 16:05

Update the Ray CI config for the removal of the cloudpickle/compat.py…

2a60637

… file

Also skip test_client_worker in dask distributed's test suite

8294be0

ogrisel added 3 commits October 12, 2023 17:20

Update tornado test to avoid raising a deprecation warning with Pytho…

e5b6162

…n 3.12

Define an official public namedpace in the top level package

8708303

Forgot about register_pickle_by_value/unregister_pickle_by_value whic…

c9f4b6e

…h is also part of the public API

This was referenced Oct 12, 2023

Alias CloudPickler in the cloudpickle namespace as Pickler #235

Closed

Prepare transition to let cloudpickle.Pickler point to CloudPickler #372

Closed

Make it possible to run the tests without installing tests/cloudpickl…

be52c1a

…e_testpkg

pierreglaser approved these changes Oct 12, 2023

View reviewed changes

cloudpickle/cloudpickle.py Show resolved Hide resolved

tests/cloudpickle_test.py Show resolved Hide resolved

dev-requirements.txt Show resolved Hide resolved

Deprecate cloudpickle.is_tornado_coroutine

c5c0b44

ogrisel merged commit c6f8cd4 into cloudpipe:master Oct 13, 2023
20 of 21 checks passed

ogrisel deleted the official-support-for-python-3.12 branch October 13, 2023 09:28

This was referenced Oct 13, 2023

Remove and deprecate unused functions #444

Closed

Instance-level dispatch table #395

Open

HyukjinKwon mentioned this pull request Nov 24, 2023

[SPARK-46080][PYTHON] Upgrade Cloudpickle to 3.0.0 apache/spark#43989

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Official support for python 3.12 and code clean-up after EOLed Python 3.6 and 3.7. #517

Official support for python 3.12 and code clean-up after EOLed Python 3.6 and 3.7. #517

ogrisel commented Oct 9, 2023 •

edited

Loading

codecov bot commented Oct 9, 2023 •

edited

Loading

ogrisel commented Oct 10, 2023

HyukjinKwon commented Oct 12, 2023 •

edited

Loading

ogrisel commented Oct 12, 2023 •

edited

Loading

codecov bot commented Oct 12, 2023 •

edited

Loading

ogrisel commented Oct 12, 2023

ogrisel commented Oct 12, 2023

pierreglaser left a comment

ogrisel commented Oct 13, 2023

HyukjinKwon commented Nov 24, 2023

Official support for python 3.12 and code clean-up after EOLed Python 3.6 and 3.7. #517

Official support for python 3.12 and code clean-up after EOLed Python 3.6 and 3.7. #517

Conversation

ogrisel commented Oct 9, 2023 • edited Loading

codecov bot commented Oct 9, 2023 • edited Loading

Codecov Report

ogrisel commented Oct 10, 2023

HyukjinKwon commented Oct 12, 2023 • edited Loading

ogrisel commented Oct 12, 2023 • edited Loading

codecov bot commented Oct 12, 2023 • edited Loading

Codecov Report

ogrisel commented Oct 12, 2023

ogrisel commented Oct 12, 2023

pierreglaser left a comment

Choose a reason for hiding this comment

ogrisel commented Oct 13, 2023

HyukjinKwon commented Nov 24, 2023

ogrisel commented Oct 9, 2023 •

edited

Loading

codecov bot commented Oct 9, 2023 •

edited

Loading

HyukjinKwon commented Oct 12, 2023 •

edited

Loading

ogrisel commented Oct 12, 2023 •

edited

Loading

codecov bot commented Oct 12, 2023 •

edited

Loading