unidist 0.7.1

This release contains an important bugfix for MPI backend using shared
memory.

Key Features and Updates

• Stability and Bugfixes
  • FIX-#425: Fix type of shared MPI.buffer (#426)

Contributors

@Retribution98

unidist 0.7.0

This release makes wheels for PyPI valid, as well as
upgrade minimal supported python version to 3.9.

Key Features and Updates

• Stability and Bugfixes
  • FIX-#416: Modify setup.py to enable python and platforms specific
    builds (#417)
  • FIX-#416: Add missing files into sdist (#418)
  • FIX-#416: Up minimal python version to 3.9 (#420)
  • FIX-#416: Build cython extension with c++11 support (#421)
  • FIX-#416: Upgrade versioneer to get correct version on Windows
    (#422)
  • FIX-#416: Build bdist and sdist in CI and upload to PyPI (#423)

Contributors

@YarShev

unidist 0.6.0

This release contains a few important improvements for MPI backend.

Key Features and Updates

• Stability and Bugfixes
  • FIX-#402: Fix MPICH version check (#403)
  • FIX-#404: Add psutil as a required dependency for unidist[mpi] (#406)
  • FIX-#407: Make data put into MPI object store immutable (#409)
• Performance enhancements
  • PERF-#360: Re - Implement asyncronous get request with a fix (#405)
• Refactor Codebase
  • REFACTOR-#000: Refactor remote function class for MPI backend (#410)

Contributors

@YarShev
@Retribution98
@arunjose696

unidist 0.5.1

This release contains an important bugfix for MPI backend and
adds psutil as a required dependency for unidist[mpi] package.

Key Features and Updates

• Stability and Bugfixes
  • FIX-#402: Fix MPICH version check (#403)
  • FIX-#404: Add psutil as a required dependency for unidist[mpi] (#406)

Contributors

@YarShev
@Retribution98

unidist 0.5.0

This release adds support for a shared object store for MPI backend,
which can be enabled with UNIDIST_MPI_SHARED_OBJECT_STORE environment variable,
contains a lot of performance enhancements, includes several features,
important documentation updates and a bunch of bugfixes.

Key Features and Updates

• Stability and Bugfixes
  • FIX-#329: Allow remaining communication requests to complete (#330)
  • FIX-#343: Get rid of the hacks in the MPI backend (#344)
  • FIX-#345: Test both modes of MPI execution on Linux only (#346)
  • FIX-#349: Remove the warning regarding MpiPickleThreshold (#350)
  • FIX-#354: Fix unidist.wait for MPI backend (#355)
  • FIX-#363: Fix DataID usage between different MPI processes (#364)
  • FIX-#371: Fix check for out of bounds reserved memory in shared storage (#372)
  • FIX-#375: Fix running an MPI cluster without using shared memory (#377)
  • FIX-#340: Fix hangs at low UNIDIST_MPI_SHARED_OBJECT_STORE_THRESHOLD (#380)
  • FIX-#376: Partition buffers of size larger than the _bigmpi.blocksize into blocks while sending and receiving with MPI (#383)
  • FIX-#337: Fix MpiHosts parameter usage (#378)
  • FIX-#000: Revert PR #387 (#388)
  • FEAT-#392: Raise a warning if shared object storage is enabled but the MPI library does not support it (#393)
  • FIX-#389: Revert async get changes made in #359 (#390)
  • FIX-#394: Fix CI running on Windows (#395)
  • FIX-#396: Fix Split_type call for Unidist on MSMPI (#397)
• Performance enhancements
  • PERF-#365: Cache info about sent metadata of shared data (#366)
  • PERF-#360: Implement asyncronous get request (#359)
  • PERF-#367: Use std::fill to fill service shared buffer with a given value (#373)
• New Features
  • FEAT-#285: Implement shared object storage for MPI backend (#286)
  • FEAT-#341: Add the ability to specify environment variables for MPI workers (#352)
  • FEAT-#347: Get rid of mpi4py-mpich dependency (#358)
  • FEAT-#386: Make MPI shared object store default option (#387)
• Refactor Codebase
  • REFACTOR-#000: Remove unnecessary build step in CI (#339)
  • REFACTOR-#000: Introduce task_info metadata package (#348)
  • REFACTOR-#000: Remove unncessary else branch in mpi init (#351)
  • REFACTOR-#000: Change mpi_state.comm to mpi_state.global_comm (#379)
  • REFACTOR-#000: Change IsMpiSpawnWorkers to MpiSpawn to make it more concise (#382)
• Documentation improvements
  • DOCS-#000: Update the workflow status badge (#332)
  • DOCS-#000: Add the downloads badge in readme (#333)
  • DOCS-#384: Fix warnings when building documentation (#385)
  • DOCS-#336: Add documentation about shared object storage (#391)

Contributors

@YarShev
@Retribution98
@arunjose696

unidist 0.4.1

This release contains an important bugfix for MPI backend.

Key Features and Updates

• Stability and Bugfixes
  • FIX-#329: Allow remaining communication requests to complete (#330)

Contributors

@YarShev

unidist 0.4.0

This release makes MPI the default backend, includes new features and
important documentation updates for it, as well as contains a bunch of bugfixes.

Key Features and Updates

• Stability and Bugfixes
  • FIX-#275: Pass python flags into workers (#276)
  • FIX-#281: Do not shutdown MPI backend multiple times (#282)
  • FIX-#283: Exit monitor and worker processes in SPMD mode (#284)
  • FIX-#294: Fix MPI backend launch for Open MPI in a cluster (#295)
  • FIX-#242: Fix behavior of unidist.wait for MPI backend (#291)
  • FIX-#000: Add licence header to request_store.py (#307)
  • FIX-#296: Make workers exit the loop correctly (#312)
  • FIX-#318: Do not initialize MPI in serialization.py (#319)
  • FIX-#320: Improve CI (#321)
  • FIX-#322: Disable actor tests for Ray (#323)
  • FIX-#324: Pin pydantic<2 to fix CI (#325)
  • FIX-#326: Set I_MPI_SPAWN for Intel MPI when using dynamic spawn (#327)
• Performance enhancements
  • PERF-#287: Optimize python list appends (#290)
• New Features
  • FEAT-#288: Make MPI as default backend (#293)
  • FEAT-#305: Introduce MpiBackoff configuration value (#306)
  • FEAT-#316: Introduce MpiLog configuration value (#317)
• Refactor Codebase
  • REFACTOR-#000: Refactor request store to make it clearer (#311)
• Documentation improvements
  • DOCS-#000: Clarify the phrases regarding pyseq backend (#272)
  • DOCS-#277: Add notes regarding installation of MPI implementations (#278)
  • DOCS-#279: Advise original MPI binaries rather than conda-forge packages (#280)
  • DOCS-#299: Extend troubleshooting page as to MPI errors (#300)
  • DOCS-#301: Add Optimization Notes section to the docs (#302)
  • DOCS-#303: Fix a typo in unidist_on_mpi.rst (#304)
  • DOCS-#314: Add troubleshooting block for 'OpenSSL version mismatch' error (#315)

Contributors

@YarShev
@Retribution98
@arunjose696
@luweizheng

unidist 0.3.0

This release contains a few performance enhancements for MPI backend,
the renaming of backends "multiprocessing" to "pymp" and "python" to "pyseq",
includes a couple of important updates for documentation and multiple bugfixes.

Key Features and Updates

• Stability and Bugfixes
  • FIX-#170: Fix serialization/deserialization for Ray actors (#171)
  • FIX-#172: Fix serialization/deserialization for Dask actors (#173)
  • FIX-#174: Fix serialization/deserialization for MPI actors (#177)
  • FIX-#179: Handle None return values correctly on MPI backend (#180)
  • FIX-#175; FIX-#178: Fix nested remote calls of remote tasks and actor methods (#176)
  • FIX-#187: Make Data_ID unique for different workers on MPI backend (#188)
  • FIX-#183: Fix serialization of dict_keys for MPI backend (#184)
  • FIX-#189: Fix Actor scheduling (#190)
  • FIX-#191: Fix pendinq response after blocking GET request (#192)
  • FIX-#193: Fix actor async execution (#194)
  • FIX-#197: Initialize a backend when doing unidist.put (#198)
  • FIX-#201: Fix warnings to be aligned with latest flake8 (#202)
  • FIX-#211: Fix dask[complete] syntax in conda environment file (#212)
  • FIX-#213: Fix NamedTuple serialization (#215)
  • FIX-#209: Fix set up of conda in CI to make it work properly (#210)
  • FIX-#216: Fix development environment for ray==2.1.0 (#217)
  • FIX-#218: Install Ray from PyPI to make CI work (#220)
  • FIX-#224: Fix recursion in processing remote task arguments and serialization of dictionary that has a tuple key (#225)
  • FIX-#228: Add missing configs to workers initialization (#229)
  • FIX-#233: Fix passing MpiHosts config to workers initialization (#234)
  • FIX-#232: Fix sending out-of-band data over 1 GB (#231)
  • FIX-#235: Adding handler only if logger does not have any handlers (#236)
  • FIX-#246: Slight changes to file formatting to match new version of black (#247)
  • FIX-#253: Append worker rank to logger name (#254)
  • FIX-#256: Fix sending a cleanup command from a worker to itself (#257)
  • FIX-#263: Update the year in the licence header (#264)
  • FIX-#259: Fix cleanup data in the workers (#260)
  • FIX-#269: Update min supported Ray version (#270)
• Performance enhancements
  • PERF-#226: Do not use pickle5 for small object serialization (#227)
  • PERF-#243: Using non-blocking send (#244)
  • PERF-#195: Fix serialization for numpy objects (#200)
  • PERF-#195: Fix serialization for pandas and numpy objects (#196)
• New Features
  • FEAT-#185: Add communication logging (#186)
  • FEAT-#204: Allow PERF prefix in the commit messages (#205)
  • FIX-#248: Rename backends "multiprocessing" to "pymp" and "python" to "pyseq" (#265)
• Refactor Codebase
  • REFACTOR-#237: Replace len_buffers with buffer_count (#238)
• Documentation improvements
  • DOCS-#207: Add a doc section regarding installation of Intel MPI (#208)
  • DOCS-#266: Fix documentation build (#267)
  • DOCS-#221: Add mentioning about Modin and unidist integration in to docs (#222)
• Update testing suite
  • TEST-#181: Added async actor tests (#182)
  • TEST-#89: Set up testing for unidist on push (#206)
  • FIX-#250: Require format for PR descriptions instead of commit descriptions (#251)
  • FIX-#250: Replace lint-commit to check-pr-title in ci.yml (#252)

Contributors

@YarShev
@Retribution98
@arunjose696

unidist 0.2.2

This release contains a few important bugfixes for MPI backend.

Key Features and Updates

• Stability and Bugfixes
  • FIX-#201: Fix warnings to be aligned with latest flake8 (#202)
  • FIX-#195: Fix serialization for numpy objects (#200)
  • FIX-#211: Fix dask[complete] syntax in conda environment file (#212)
  • FIX-#213: Fix NamedTuple serialization (#215)
  • FIX-#224: Fix recursion in processing remote task arguments and serialization of dictionary that has a tuple key (#225)
  • FIX-#228: Add missing configs to workers initialization (#229)

Contributors

@YarShev
@Retribution98

unidist 0.2.1

This release contains multiple important bugfixes for MPI backend,
which makes it be more functional and stable.

Key Features and Updates

• Stability and Bugfixes
  • FIX-#170: Fix serialization/deserialization for Ray actors (#171)
  • FIX-#172: Fix serialization/deserialization for Dask actors (#173)
  • FIX-#174: Fix serialization/deserialization for MPI actors (#177)
  • FIX-#179: Handle None return values correctly on MPI backend (#180)
  • FIX-#175: FIX-#178: Fix nested remote calls of remote tasks and actor methods (#176)
  • FIX-#187: Make Data_ID unique for different workers on MPI backend (#188)
  • FIX-#183: Fix serialization of dict_keys for MPI backend (#184)
  • FEAT-#185: Add communication logging (#186)
  • FIX-#189: Fix Actor scheduling (#190)
  • FIX-#191: Fix pendinq response after blocking GET request (#192)
  • FIX-#195: Fix serialization for pandas and numpy objects (#196)
  • FIX-#193: Fix actor async execution (#194)
  • FIX-#197: Initialize a backend when doing unidist.put (#198)
• Update testing suite
  • TEST-#181: Added async actor tests (#182)

Contributors

@YarShev
@Retribution98