feat!: Live, laugh, Shwap #3675
Open
+13,996
−9,614
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This is the foundation of new ~era~ sampling protocol and storage.
Adds in-memory file implementation
…r Share (#3012) Reworks `Share` method of `Store` to return shareWithProof containing axis field.
This PR implements the foundational types and functionalities for the Shwap system as defined in [CIP-19](https://github.com/celestiaorg/CIPs/blob/main/cips/cip-19.md). It introduces several key components essential for the operation of the shwap containers and their identification within the Celestia network. Here are the primary additions: **Features Implemented:** 1. **Base Shwap Container Types:** - **Entities Added:** - `Sample`: Represents a share with its corresponding Merkle proof. - `Row`: Denotes a part of a row in an Extended Data Square (EDS), including either its left or right half. - `RowNamespaceData `: Handles rows with specified namespaces. - **Functionalities:** - Protobuf encoding for each container type. - Validation functions to check the integrity and correctness of the data structures received from network. - Inclusion verification methods to validate each entity against a provided data root. 2. **Shwap ID Types:** - **Details:** - Declaration of unique identifier types for shwap entities,. - **Encoding:** - Implementation of binary encoding for each ID type. **Side Note** If you encounter any confusing elements in this PR, please highlight them. After working on this project for so long, it's possible I've overlooked what might be unclear to others. I'm eager to add comments wherever necessary to make everything as clear as possible. Your feedback is invaluable in enhancing the understandability of our work. --------- Co-authored-by: Wondertan <[email protected]>
Adds tests for shwap types and associated functions. There are no non-inclusion tests, because the test function `NamespacedRowFromShares` for creating test data does not support the creation of non-inclusion proofs. The non-inclusion logic itself exists and will be covered at the level of integration tests. `NamespacedRowFromShares` is not used in the shwap and may only be needed for more convenient testing. The Store implementation will use its own proof-building engine, which supports non-inclusion proofs. For now, I see no reason to fuss with this. To add these unit tests, either a lot of duplicative NMT code needs to be written, or access to the NMT tree inside the app wrapper must be provided. The app wrapper will need to be redesigned later for a “clean” implementation of proof caching. Based on #3383
Updates file interface to latest prototype version
- Reintroduce file interface as eds interface. Change aims to allow
usage of EDS interface outside of storage package and to be high level
interface of EDS methods.
- Renames of eds interface methods to align with returned shwap types
names
- Share() -> Sample
- Data -> Row data
- Extracts New<shwap_type_name>FromEDS functions to eds file methods
- moves associated tests to eds pkg
**Additional refactoring:**
- **Change Interface Name**: Realized that 'EDS' is a terrible name for
an interface. Renamed `eds.EDS` to `eds.Accessor` to more accurately
reflect its functionality rather than its internal content.
- **Separate Closer**: Extracted `Closer` from `Accessor`. Now it is
available in a new composite interface `AccessorCloser`.
- **Rename InMem**: Renamed `InMem` to `rsmt2d` to better align with its
usage.
- **Decouple NamespacedData**: Separated `NamespacedData` from the
`rsmt2d` implementation. It is now a standalone function.
- **Update EDS Method**: Replaced the `EDS()` method with `Flattened`,
similar to `rsmt2d`. Considered introducing two separate methods,
`Flattened` and `FlattenedODS`, with the latter to be potentially added
later. Proposed to park this suggestion in an issue for future
consideration.
Adds generalised testing framework for file interface implementaions Also adds non-inclusion proofs handling in RowNamespaceDataFromShares and closes #3428
Adds middleware to protect AccessorCloser from multiple Closes. Based on #3425
This PR adds ods file, that implements eds.Accessor interface. The file stores ods part of eds on disk and lazily reads data upon request. If requested data is from Q4, it reads full ods in single read and stores it in-memory for later re-use.
Co-authored-by: Wondertan <[email protected]>
In #3545, I requested to rename `DataHash` to `DataRoot` to match the name of the original name in the Core's RawHeader. However, I realized it is called `DataHash` in the header, so this PR reverts that change. I think this confusion comes from the fact that we are always referring to this field as `DataRoot` in our discussions, so it feels like this is the correct phrasing, but it's not. It actually does not matter how it's called, honestly, and both are correct in some way, but it's good to be consistent in the code and avoid confusion with tautology.
Add versioning for discovery, so we can separate shwap network from older one. Metrics will still use tag instead of topic, so we can support multiple visions with same dashboard
cristaloleg
reviewed
Sep 11, 2024
cristaloleg
reviewed
Sep 11, 2024
| return | ||
| } | ||
| if ctx.Err() != nil { | ||
| ctx = context.Background() |
There was a problem hiding this comment.
Same here, infinite ctx.
Otherwise, it has additional `/` e.g. `//celestia/private/shwap/ipfs/bitswap/1.2.0`
Adds cel-shed swap subcommand for swap-related utilities. The introduced one decodes shwap cids and prints its information
Signed-off-by: Wondertan <[email protected]>
We need to check that requested ID is equal to received ID. This was the case, but implicitly in the Bitswap layer. This change adds another explicit check for that, ensuring IDs are printed on error Signed-off-by: Wondertan <[email protected]>
Signed-off-by: Wondertan <[email protected]>
This reverts commit 75de820.
Skip empty blocks in generateNonEmptyBlocks
There were two error logs per every Bitswap request for a not synced EDS. Not anymore.
Avoidы storing q4 for data outside of availability window
This reverts commit 15ebe17.
With Bitswap moving to using blockstore.Has soon, we need a way to check if the Accessor is present on the cache before reading the disk and this change does exactly that.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR aims to merge Shwap into the
mainbranch.Commits must be merged without squashing.