-
Notifications
You must be signed in to change notification settings - Fork 353
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[CELEBORN-1490][CIP-6] Impl worker write process for Flink Hybrid Shuffle #2741
base: main
Are you sure you want to change the base?
Conversation
081b5fd
to
6b0bf5e
Compare
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #2741 +/- ##
==========================================
+ Coverage 33.11% 33.12% +0.01%
==========================================
Files 314 314
Lines 18373 18435 +62
Branches 1683 1691 +8
==========================================
+ Hits 6082 6104 +22
- Misses 11951 11989 +38
- Partials 340 342 +2 ☔ View full report in Codecov by Sentry. |
worker/src/main/scala/org/apache/celeborn/service/deploy/worker/WorkerSource.scala
Show resolved
Hide resolved
worker/src/main/java/org/apache/celeborn/service/deploy/worker/storage/PartitionDataWriter.java
Outdated
Show resolved
Hide resolved
worker/src/main/scala/org/apache/celeborn/service/deploy/worker/PushDataHandler.scala
Outdated
Show resolved
Hide resolved
...org/apache/celeborn/service/deploy/worker/storage/segment/SegmentMapPartitionFileWriter.java
Outdated
Show resolved
Hide resolved
.../src/main/java/org/apache/celeborn/service/deploy/worker/storage/MapPartitionDataWriter.java
Outdated
Show resolved
Hide resolved
Co-authored-by: Xu Huang <[email protected]>
...org/apache/celeborn/service/deploy/worker/storage/segment/SegmentMapPartitionFileWriter.java
Outdated
Show resolved
Hide resolved
...org/apache/celeborn/service/deploy/worker/storage/segment/SegmentMapPartitionFileWriter.java
Outdated
Show resolved
Hide resolved
...org/apache/celeborn/service/deploy/worker/storage/segment/SegmentMapPartitionFileWriter.java
Show resolved
Hide resolved
@reswqa, could you provide the grafana metric dashboard renderings to help confirm whether the |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM.
What changes were proposed in this pull request?
Impl worker write process for Flink Hybrid Shuffle.
Why are the changes needed?
We supports tiered producer write data from flink to worker. In this PR, we enable the worker to write this kind of data to storage.
Does this PR introduce any user-facing change?
no
How was this patch tested?
no need.