Support pod template for Spark 3.x applications #2141
Conversation
Signed-off-by: Yi Chen <[email protected]>
Signed-off-by: Yi Chen <[email protected]>
Signed-off-by: Yi Chen <[email protected]>
ChenYi015
force-pushed
the
feature/pod-template
branch
from
07e334a
to
dc671cb
Compare
|
/hold |
| // Spark version >= 3.0.0 is required. | ||
| // Ref: https://spark.apache.org/docs/latest/running-on-kubernetes.html#pod-template. | ||
| // +optional | ||
| Template *corev1.PodTemplateSpec `json:"template,omitempty"` |
There was a problem hiding this comment.
Will review the rest of the PR as well, but adding this as a struct rather than a binary/string field increases the number of properties in the resulting CRD by quite a bit. Is there any downside/considerations on the K8s side for very large CRDs?
There was a problem hiding this comment.
Good catch. I added a kubebuilder marker +kubebuilder:validation:Schemaless to mark the pod template field as schemaless to reduce the size of CRD.
Signed-off-by: Yi Chen <[email protected]>
ChenYi015
force-pushed
the
feature/pod-template
branch
from
27696b4
to
a7c5002
Compare
| @@ -302,6 +304,12 @@ func driverConfOption(app *v1beta2.SparkApplication) ([]string, error) { | |||
| property = fmt.Sprintf(common.SparkKubernetesDriverLabelTemplate, common.LabelLaunchedBySparkOperator) | |||
| args = append(args, "--conf", fmt.Sprintf("%s=%s", property, "true")) | |||
|
|
|||
| // If Spark version is less than 3.0.0 or driver pod template is not defined, then the driver pod needs to be mutated by the webhook. | |||
There was a problem hiding this comment.
As discussed during the community call, we can think about a future release (maybe 2.1) where we announce the deprecation of web hook and support for Spark 2.x
There was a problem hiding this comment.
Would we take the approach of moving to users only being able to specify a pod template, or map the existing struct fields and construct a pod template on the fly during submission?
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: yuchaoran2011 The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
Signed-off-by: Chaoran Yu <[email protected]>
Purpose of this PR
Close #2101
Close #1690
Proposed changes:
spec.driver.templateandspec.executor.templateto SparkApplication CRDsparkoperator.k8s.io/mutated-by-spark-operator="true", and this label will be added by controller as needed.spark-pi-pod-template.yamlChange Category
Indicate the type of change by marking the applicable boxes:
Rationale
Checklist
Before submitting your PR, please review the following:
Additional Notes