Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

piped panics when using K8S_PRIMARY_ROLLOUT with spec.planner.alwaysPipelineSync: true and its stage option prune:true on v0.47.3-rc0 #4996

Open
ffjlabo opened this issue Jun 25, 2024 · 0 comments
Assignees
Labels
kind/bug Something isn't working

Comments

@ffjlabo
Copy link
Member

ffjlabo commented Jun 25, 2024

What happened:

When adding the k8s app with app.pipecd.yaml which has spec.planner.alwaysPipelineSync: true and K8S_PRIMARY_ROLLOUT with the option prune:true, piped fails with panic like this↓.
The piped keeps failing until it is canceled on the UI.

found out 3 valid unregistered applications in repository "ffjlabo-dev"
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x2 addr=0x18 pc=0x103eda994]

goroutine 6677 [running]:
github.com/pipe-cd/pipecd/pkg/app/piped/executor/kubernetes.(*deployExecutor).ensurePrimaryRollout(0x140020c0e00, {0x1049e8f98, 0x14002003310})
	/Users/s14218/oss/pipe-cd/pipecd/pkg/app/piped/executor/kubernetes/primary.go:161 +0xc34
github.com/pipe-cd/pipecd/pkg/app/piped/executor/kubernetes.(*deployExecutor).Execute(0x140020c0e00, {0x1049e9580, 0x1400218c360})
	/Users/s14218/oss/pipe-cd/pipecd/pkg/app/piped/executor/kubernetes/kubernetes.go:147 +0xb2c
github.com/pipe-cd/pipecd/pkg/app/piped/controller.(*scheduler).executeStage(0x140001d7188, {0x1049e9580, 0x1400218c360}, {{{}, {}, {}, 0x0}, 0x0, {0x0, 0x0, ...}, ...}, ...)
	/Users/s14218/oss/pipe-cd/pipecd/pkg/app/piped/controller/scheduler.go:541 +0xde4
github.com/pipe-cd/pipecd/pkg/app/piped/controller.(*scheduler).Run.func2()
	/Users/s14218/oss/pipe-cd/pipecd/pkg/app/piped/controller/scheduler.go:300 +0xb8
created by github.com/pipe-cd/pipecd/pkg/app/piped/controller.(*scheduler).Run in goroutine 6674
	/Users/s14218/oss/pipe-cd/pipecd/pkg/app/piped/controller/scheduler.go:299 +0xc50
exit status 2
PipeCD

K8S_PRIMARY_ROLLOUT requires the previous running commit, which was deployed previously when prune option is enabled.

	// Wait for all applied manifests to be stable.
	// In theory, we don't need to wait for them to be stable before going to the next step
	// but waiting for a while reduces the number of Kubernetes changes in a short time.
	e.LogPersister.Info("Waiting for the applied manifests to be stable")
	select {
	case <-time.After(15 * time.Second):
		break
	case <-ctx.Done():
		break
	}

	// Find the running resources that are not defined in Git.
	e.LogPersister.Info("Start finding all running PRIMARY resources but no longer defined in Git")
	// Load running manifests at the most successful deployed commit.
	e.LogPersister.Infof("Loading running manifests at commit %s for handling", e.Deployment.RunningCommitHash)
	ds, err := e.RunningDSP.Get(ctx, e.LogPersister) 

if !options.Prune {
e.LogPersister.Info("Resource GC was skipped because sync.prune was not configured")
return model.StageStatus_STAGE_SUCCESS
}
// Wait for all applied manifests to be stable.
// In theory, we don't need to wait for them to be stable before going to the next step
// but waiting for a while reduces the number of Kubernetes changes in a short time.
e.LogPersister.Info("Waiting for the applied manifests to be stable")
select {
case <-time.After(15 * time.Second):
break
case <-ctx.Done():
break
}
// Find the running resources that are not defined in Git.
e.LogPersister.Info("Start finding all running PRIMARY resources but no longer defined in Git")
// Load running manifests at the most successful deployed commit.
e.LogPersister.Infof("Loading running manifests at commit %s for handling", e.Deployment.RunningCommitHash)
ds, err := e.RunningDSP.Get(ctx, e.LogPersister)
if err != nil {
e.LogPersister.Errorf("Failed to prepare running deploy source (%v)", err)
return model.StageStatus_STAGE_FAILURE
}

But there isn't the one when adding the app first.
So e.RunningDSP is nil and causes panic.

This bug is created by the fix in #4916

What you expected to happen:

It should fail with an error on the stage when first adding the app and deploying it as PipelineSync.

How to reproduce it:

Create and add k8s app below.

app.pipecd.yaml
deployment.yaml
service.yaml

app.pipecd.yaml

apiVersion: pipecd.dev/v1beta1
kind: KubernetesApp
spec:
  name: primary-rollout
  labels:
    env: example
    team: product
  planner:
    alwaysUsePipeline: true
  pipeline:
    stages:
      - name: WAIT
        with:
          duration: 5s
      - name: K8S_PRIMARY_ROLLOUT
        with:
          prune: true

deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: primary-rollout
  labels:
    app: primary-rollout
spec:
  replicas: 3
  selector:
    matchLabels:
      app: primary-rollout
      pipecd.dev/variant: primary
  template:
    metadata:
      labels:
        app: primary-rollout
        pipecd.dev/variant: primary
    spec:
      containers:
      - name: helloworld
        image: ghcr.io/pipe-cd/helloworld:v0.30.0
        args:
          - server
        ports:
        - containerPort: 9085

service.yaml

apiVersion: v1
kind: Service
metadata:
  name: primary-rollout
spec:
  selector:
    app: primary-rollout
  ports:
    - protocol: TCP
      port: 9085
      targetPort: 9085

Environment:

  • piped version: v0.47.3-rc0
  • control-plane version:
  • Others:
@ffjlabo ffjlabo added the kind/bug Something isn't working label Jun 25, 2024
@ffjlabo ffjlabo self-assigned this Jun 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Something isn't working
Projects
Status: 🌮 Postponed
Status: No status
Development

No branches or pull requests

1 participant