Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

piped panics when using K8S_BASELINE_ROLLOUT with spec.planner.alwaysPipelineSync: true on v0.47.3-rc0 #4995

Open
ffjlabo opened this issue Jun 25, 2024 · 0 comments
Assignees
Labels
kind/bug Something isn't working

Comments

@ffjlabo
Copy link
Member

ffjlabo commented Jun 25, 2024

What happened:

When adding the k8s app with app.pipecd.yaml which has spec.planner.alwaysPipelineSync: true and K8S_BASELINE_ROLLOUT, piped fails with panic like this↓.
The piped keeps failing until it is canceled on the UI.

gx6v1_zp09_2tx_x/T/workspace1130338840/35fa4ef2-dd6f-40c0-acf2-d9a944b32d50-scheduler-192449208", "stage-name": "K8S_CANARY_ROLLOUT", "app-dir": "/var/folders/th/pq_q9v6j6_n_0fgx6v1_zp09_2tx_x/T/workspace1130338840/35fa4ef2-dd6f-40c0-acf2-d9a944b32d50-scheduler-192449208/target-deploysource/deploysource3508451428/repo-1/kubernetes/analysis-with-baseline"}
there are 1 planned/running deployments for scheduling	{"count": 1}
start executing kubernetes stage	{"deployment-id": "35fa4ef2-dd6f-40c0-acf2-d9a944b32d50", "app-id": "c6c93b81-0afa-47f3-a79d-2bbff89d7723", "project-id": "pipecd", "app-kind": "KUBERNETES", "working-dir": "/var/folders/th/pq_q9v6j6_n_0fgx6v1_zp09_2tx_x/T/workspace1130338840/35fa4ef2-dd6f-40c0-acf2-d9a944b32d50-scheduler-192449208", "stage-name": "K8S_BASELINE_ROLLOUT", "app-dir": "/var/folders/th/pq_q9v6j6_n_0fgx6v1_zp09_2tx_x/T/workspace1130338840/35fa4ef2-dd6f-40c0-acf2-d9a944b32d50-scheduler-192449208/target-deploysource/deploysource3508451428/repo-2/kubernetes/analysis-with-baseline"}
successfully reported 6 events about application live state	{"platform-provider": "kubernetes-dev"}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x2 addr=0x18 pc=0x1046345a4]

goroutine 17863 [running]:
github.com/pipe-cd/pipecd/pkg/app/piped/executor/kubernetes.(*deployExecutor).ensureBaselineRollout(0x14000842800, {0x10514cf98, 0x14000f3e320})
	/Users/s14218/oss/pipe-cd/pipecd/pkg/app/piped/executor/kubernetes/baseline.go:46 +0x1b4
github.com/pipe-cd/pipecd/pkg/app/piped/executor/kubernetes.(*deployExecutor).Execute(0x14000842800, {0x10514d580, 0x140018eee70})
	/Users/s14218/oss/pipe-cd/pipecd/pkg/app/piped/executor/kubernetes/kubernetes.go:156 +0xb00
github.com/pipe-cd/pipecd/pkg/app/piped/controller.(*scheduler).executeStage(0x14000acb408, {0x10514d580, 0x140018eee70}, {{{}, {}, {}, 0x0}, 0x0, {0x0, 0x0, ...}, ...}, ...)
	/Users/s14218/oss/pipe-cd/pipecd/pkg/app/piped/controller/scheduler.go:541 +0xde4
github.com/pipe-cd/pipecd/pkg/app/piped/controller.(*scheduler).Run.func2()
	/Users/s14218/oss/pipe-cd/pipecd/pkg/app/piped/controller/scheduler.go:300 +0xb8
created by github.com/pipe-cd/pipecd/pkg/app/piped/controller.(*scheduler).Run in goroutine 17606
	/Users/s14218/oss/pipe-cd/pipecd/pkg/app/piped/controller/scheduler.go:299 +0xc50
exit status 2
PipeCD

K8S_BASELINE_ROLLOUT requires the previous running commit, which was deployed previously.
https://github.com/pipe-cd/pipecd/blob/master/pkg/app/piped/executor/kubernetes/baseline.go#L46

func (e *deployExecutor) ensureBaselineRollout(ctx context.Context) model.StageStatus {
	var (
		runningCommit   = e.Deployment.RunningCommitHash
		options         = e.StageConfig.K8sBaselineRolloutStageOptions
		variantLabel    = e.appCfg.VariantLabel.Key
		baselineVariant = e.appCfg.VariantLabel.BaselineValue
	)
	if options == nil {
		e.LogPersister.Errorf("Malformed configuration for stage %s", e.Stage.Name)
		return model.StageStatus_STAGE_FAILURE
	}

	// Load running manifests at the most successful deployed commit.
	e.LogPersister.Infof("Loading running manifests at commit %s for handling", runningCommit)
	ds, err := e.RunningDSP.Get(ctx, e.LogPersister)
	if err != nil {
		e.LogPersister.Errorf("Failed to prepare running deploy source (%v)", err)
		return model.StageStatus_STAGE_FAILURE
	}

But there isn't the one when adding the app first.
So e.RunningDSP is nil and causes panic.

This bug is created by the fix in #4916

What you expected to happen:

It should fail with an error on the stage when first adding the app and deploying it as PipelineSync.

How to reproduce it:

Create and add k8s app below.

app.pipecd.yaml
deployment.yaml
service.yaml

app.pipecd.yaml

apiVersion: pipecd.dev/v1beta1
kind: KubernetesApp
spec:
  name: analysis-with-baseline
  labels:
    env: example
    team: product
  planner:
    alwaysUsePipeline: true
  pipeline:
    stages:
      - name: K8S_CANARY_ROLLOUT
        with:
          replicas: 10%
      - name: K8S_BASELINE_ROLLOUT
        with:
          replicas: 10%
      - name: ANALYSIS
        with:
          duration: 10m
          threshold: 2
      - name: K8S_PRIMARY_ROLLOUT
      - name: K8S_CANARY_CLEAN
      - name: K8S_BASELINE_CLEAN

deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: analysis-with-baseline
  labels:
    app: analysis-with-baseline
spec:
  replicas: 3
  selector:
    matchLabels:
      app: analysis-with-baseline
      pipecd.dev/variant: primary
  template:
    metadata:
      labels:
        app: analysis-with-baseline
        pipecd.dev/variant: primary
    spec:
      containers:
      - name: helloworld
        image: ghcr.io/pipe-cd/helloworld:v0.30.0
        args:
          - server
        ports:
        - containerPort: 9085

service.yaml

apiVersion: v1
kind: Service
metadata:
  name: analysis-with-baseline
spec:
  selector:
    app: analysis-with-baseline
  ports:
    - protocol: TCP
      port: 9085
      targetPort: 9085

Environment:

  • piped version: v0.47.3-rc0
  • control-plane version:
  • Others:
@ffjlabo ffjlabo added the kind/bug Something isn't working label Jun 25, 2024
@ffjlabo ffjlabo self-assigned this Jun 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Something isn't working
Projects
Status: 🌮 Postponed
Status: No status
Development

No branches or pull requests

1 participant