Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

handle dataclass field type sentinels #513

Merged
merged 5 commits into from
Oct 9, 2023

Conversation

rmorshea
Copy link
Contributor

Closes: #386

In short, the dataclasses module relies on the identity of certain sentinel values being preserved. To preserve their identity, this PR relies on a minimal set of internal dataclasses APIs to reduce those sentinels to their _FIELD_BASE.name. Then, when reconstituting them, the names key into a _DATACLASSE_FIELD_TYPE_SENTINELS mapping that contains the expected sentinel instances.

The implementation accesses the following private APIs:

  • dataclasses._FIELD_BASE
  • dataclasses._FIELD
  • dataclasses._FIELD_CLASSVAR
  • dataclasses._FIELD_INITVAR

Copy link
Member

@jrbourbeau jrbourbeau left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We started running into this over in Dask when sending dataclasses around. @ogrisel @pierreglaser I wonder if this is something you have bandwidth to take a look at? I'm not super familiar with the dataclass logic here, but can confirm it resolves the serialization issue I've seen recently.

@ogrisel
Copy link
Contributor

ogrisel commented Oct 9, 2023

@jrbourbeau do you have a minimal reproducer that involves dask? That would help me review this PR.

EDIT: hopefully the new test in e15fb62 covers both the original bug report in #386 and your use case with dask.

@codecov
Copy link

codecov bot commented Oct 9, 2023

Codecov Report

All modified lines are covered by tests ✅

Comparison is base (0f330b6) 83.28% compared to head (9892fba) 83.44%.

❗ Current head 9892fba differs from pull request most recent head 550f7df. Consider uploading reports for the commit 550f7df to get more accurate results

Additional details and impacted files
@@            Coverage Diff             @@
##           master     #513      +/-   ##
==========================================
+ Coverage   83.28%   83.44%   +0.16%     
==========================================
  Files           4        4              
  Lines         718      725       +7     
  Branches      157      157              
==========================================
+ Hits          598      605       +7     
  Misses         95       95              
  Partials       25       25              
Files Coverage Δ
cloudpickle/cloudpickle_fast.py 91.61% <100.00%> (+0.17%) ⬆️

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

tests/cloudpickle_test.py Outdated Show resolved Hide resolved
@ogrisel ogrisel added the ci downstream Signal the CI to run the test suite of all registered cloudpickle downstream projects. label Oct 9, 2023
Copy link
Contributor

@ogrisel ogrisel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The dowstream CI config is broken (for unrelated causes) for joblib, dask and ray and would require some maintenance effort.

The PR itself looks good to me: the changes are not invasive and I checked that it actually fixes problem from the original report.

@ogrisel ogrisel merged commit ab32ba5 into cloudpipe:master Oct 9, 2023
14 of 17 checks passed
@ogrisel
Copy link
Contributor

ogrisel commented Oct 9, 2023

Thank you very much @rmorshea!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ci downstream Signal the CI to run the test suite of all registered cloudpickle downstream projects.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Cloudpickle breaks local dataclass
3 participants