Add 1K and 5K RayCluster/RayJob scalability test results #2218
Conversation
| - | | ||
| #!/bin/sh | ||
|
|
||
| ray job logs $RAY_JOB_SUBMISSION_ID --address=http://$RAY_DASHBOARD_ADDRESS --follow || \ |
There was a problem hiding this comment.
This is a temporary workaroud until ray job submit doesn't error on already existing submission IDs
| apiVersion: apps/v1 | ||
| kind: DaemonSet | ||
| metadata: | ||
| name: ray-image-preloader |
There was a problem hiding this comment.
For 1K and 5K tests, a lot of time was spent on pulling Ray images so I added pre-loader daemonset that runs before the tests
| @@ -0,0 +1,17 @@ | |||
| #!/bin/bash | |||
There was a problem hiding this comment.
Both 1000-raycluster and 5000-raycluster directories have wait-for-raycluster.sh. Could we unify them?
There was a problem hiding this comment.
Yeah we could, can I do it in a follow-up when we introduce the 10K tests?
| command: | ||
| - "bash" | ||
| - "1000-raycluster/wait-for-rayclusters.sh" | ||
| - name: Wait for pods to be running |
There was a problem hiding this comment.
What does this mean? Does it execute after "Wait for RayClusters ready"? If a RayCluster is "ready", all Pods in the RayCluster should be running and ready. It is a bit weird for me to measure "pods to be running" after RayCluster is ready.
There was a problem hiding this comment.
Ah, this step is not actually waiting for pdos to be ready, it's measuring the WaitForControlledPodsRunning metric, which we capture after waitign for all RayClusters to be ready. I updated the name to better reflect what this is doing.
| Method: WaitForControlledPodsRunning | ||
| Params: | ||
| action: gather | ||
| - name: Measure pod startup latency |
There was a problem hiding this comment.
The definition is here: https://github.com/kubernetes/community/blob/master/sig-scalability/slos/pod_startup_latency.md
There was a problem hiding this comment.
Now that you mention it, we're probably not that interested in startup latency because it excludes image pull times and init containers. I'll remove this in a follow-up PR
| kind: RayCluster | ||
| labelSelector: app.kubernetes.io/created-by = kuberay-operator | ||
| operationTimeout: 120s | ||
| - Identifier: JobLifecycleLatency |
There was a problem hiding this comment.
What does JobLifecycleLatency mean? Does it refer to the submitter K8s Job?
There was a problem hiding this comment.
yes, it's measuring time for Job to complete. In our case it measures submitter job time to completion, which roughly represents how long the training job took to complete
Signed-off-by: Andrew Sy Kim <[email protected]>
Signed-off-by: Andrew Sy Kim <[email protected]>
I updated the README with some references on how to understand the results |
Why are these changes needed?
Per #2069, adds 1K and 5K RayCluster / RayJob test results
Related issue number
#2069
Checks