Spinnaker: Pipeline template not rendering correctly in 1.11.7

[sdpoueme]

Issue Summary:

We are using pipeline templates to update/re-create our deployment pipelines on Spinnaker. At the moment, we are experiencing a very strange issue. Since the upgrade to 1.11.7, our templates relying on helm charts are not rendering pipelines correctly.

Cloud Provider(s):

GCP

Environment:

Spinnaker 1.11.7

Feature Area:

Pipeline Templates

Description:

The helm release pipelines are composed of two stages:

1. bake, where we render the manifests using Helm2 then produce a base64 artifacts.
2. deploy, where the base64 artifact is deployed to k8s inside a dedicated namespace

The issue we are experimenting is with the bake stage, even though we are specifying an expected artifact in the template artifact. The artifact is not displayed in the deck ui.

And if the pipeline executes, it will fail with the following error:

{"exception":{"details":{"error":"Unexpected Task Failure","errors":["01D1XY4MG9NRMXJX2BADSJCM2Q: Input artifact 5384bf5f-4638-4bae-afbf-ba22f2bed72b could not be found in the execution"],"stackTrace":"java.lang.IllegalArgumentException: 01D1XY4MG9NRMXJX2BADSJCM2Q: Input artifact 5384bf5f-4638-4bae-afbf-ba22f2bed72b could not be found in the execution\n\tat com.netflix.spinnaker.orca.bakery.tasks.manifests.CreateBakeManifestTask.lambda$execute$0(CreateBakeManifestTask.java:81)\n\tat java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)\n\tat java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)\n\tat java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)\n\tat java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)\n\tat java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)\n\tat java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)\n\tat java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)\n\tat com.netflix.spinnaker.orca.bakery.tasks.manifests.CreateBakeManifestTask.execute(CreateBakeManifestTask.java:85)\n\tat com.netflix.spinnaker.orca.q.handler.RunTaskHandler$handle$1$2$1.invoke(RunTaskHandler.kt:100)\n\tat com.netflix.spinnaker.orca.q.handler.RunTaskHandler$handle$1$2$1.invoke(RunTaskHandler.kt:56)\n\tat com.netflix.spinnaker.orca.q.handler.RunTaskHandler.withLoggingContext(RunTaskHandler.kt:311)\n\tat com.netflix.spinnaker.orca.q.handler.RunTaskHandler.access$withLoggingContext(RunTaskHandler.kt:56)\n\tat com.netflix.spinnaker.orca.q.handler.RunTaskHandler$handle$1$2.invoke(RunTaskHandler.kt:99)\n\tat com.netflix.spinnaker.orca.q.handler.RunTaskHandler$handle$1$2.invoke(RunTaskHandler.kt:56)\n\tat com.netflix.spinnaker.orca.q.handler.AuthenticationAware$sam$java_util_concurrent_Callable$0.call(AuthenticationAware.kt)\n\tat com.netflix.spinnaker.security.AuthenticatedRequest.lambda$propagate$0(AuthenticatedRequest.java:97)\n\tat com.netflix.spinnaker.orca.q.handler.AuthenticationAware$DefaultImpls.withAuth(AuthenticationAware.kt:49)\n\tat com.netflix.spinnaker.orca.q.handler.RunTaskHandler.withAuth(RunTaskHandler.kt:56)\n\tat com.netflix.spinnaker.orca.q.handler.RunTaskHandler$handle$1.invoke(RunTaskHandler.kt:98)\n\tat com.netflix.spinnaker.orca.q.handler.RunTaskHandler$handle$1.invoke(RunTaskHandler.kt:56)\n\tat com.netflix.spinnaker.orca.q.handler.RunTaskHandler$withTask$1.invoke(RunTaskHandler.kt:182)\n\tat com.netflix.spinnaker.orca.q.handler.RunTaskHandler$withTask$1.invoke(RunTaskHandler.kt:56)\n\tat com.netflix.spinnaker.orca.q.handler.OrcaMessageHandler$withTask$1.invoke(OrcaMessageHandler.kt:49)\n\tat com.netflix.spinnaker.orca.q.handler.OrcaMessageHandler$withTask$1.invoke(OrcaMessageHandler.kt:33)\n\tat com.netflix.spinnaker.orca.q.handler.OrcaMessageHandler$withStage$1.invoke(OrcaMessageHandler.kt:59)\n\tat com.netflix.spinnaker.orca.q.handler.OrcaMessageHandler$withStage$1.invoke(OrcaMessageHandler.kt:33)\n\tat com.netflix.spinnaker.orca.q.handler.OrcaMessageHandler$DefaultImpls.withExecution(OrcaMessageHandler.kt:68)\n\tat com.netflix.spinnaker.orca.q.handler.RunTaskHandler.withExecution(RunTaskHandler.kt:56)\n\tat com.netflix.spinnaker.orca.q.handler.OrcaMessageHandler$DefaultImpls.withStage(OrcaMessageHandler.kt:55)\n\tat com.netflix.spinnaker.orca.q.handler.RunTaskHandler.withStage(RunTaskHandler.kt:56)\n\tat com.netflix.spinnaker.orca.q.handler.OrcaMessageHandler$DefaultImpls.withTask(OrcaMessageHandler.kt:42)\n\tat com.netflix.spinnaker.orca.q.handler.RunTaskHandler.withTask(RunTaskHandler.kt:56)\n\tat com.netflix.spinnaker.orca.q.handler.RunTaskHandler.withTask(RunTaskHandler.kt:175)\n\tat com.netflix.spinnaker.orca.q.handler.RunTaskHandler.handle(RunTaskHandler.kt:69)\n\tat com.netflix.spinnaker.orca.q.handler.RunTaskHandler.handle(RunTaskHandler.kt:56)\n\tat com.netflix.spinnaker.q.MessageHandler$DefaultImpls.invoke(MessageHandler.kt:36)\n\tat com.netflix.spinnaker.orca.q.handler.OrcaMessageHandler$DefaultImpls.invoke(OrcaMessageHandler.kt)\n\tat com.netflix.spinnaker.orca.q.handler.RunTaskHandler.invoke(RunTaskHandler.kt:56)\n\tat com.netflix.spinnaker.orca.q.audit.ExecutionTrackingMessageHandlerPostProcessor$ExecutionTrackingMessageHandlerProxy.invoke(ExecutionTrackingMessageHandlerPostProcessor.kt:47)\n\tat com.netflix.spinnaker.q.QueueProcessor$pollOnce$1$1.run(QueueProcessor.kt:82)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tat java.lang.Thread.run(Thread.java:748)\n"},"exceptionType":"IllegalArgumentException","operation":"createBake","shouldRetry":false,"timestamp":1548266656338}

We first thought it was related to https://github.com/spinnaker/spinnaker/issues/3737#issuecomment-448080940 but it seems to be a different issue. Is there any workaround we could apply to temporary solve that issue ?

Steps to Reproduce:

1. Create a bake (helm 2 rendering) and deploy pipeline template with the corresponding config
2. Save the pipeline
3. Run the pipeline

Additional Details:

---

*

[DavidGereb]

I've upgraded from 1.10.5 to 1.11.7, and everything seemed fine with the pipeline executions for a few hours after the upgrade. However, now our pipelines fail immediately with status TERMINAL with the same Input artifact <artifactID> could not be found in the execution. Also the notifications for the configuration part and the template artifacts for HELM2 bake stage not showing up on the deck, but it was the same for 1.10.5. Aside from the deck's bugs, the pipelines worked correctly, until the input artifact not found in execution issue came to life.

Additional info:
Cloud Provider: AWS
spinnaker deployed in kubernetes (AWS EKS)
artifacts are loaded from S3

[ezimanyi]

I believe this should be fixed in 1.11.8 by https://github.com/spinnaker/orca/pull/2621

[DavidGereb]

@ezimanyi thanks! confirmed its fixed in Orca 2.3.0 tried with 1.12.0