Spinnaker: [Solved] Bake Manifest RetrofitError 500 when using pipeline-template v1

[port3000]

Issue Summary:

Create a pipeline using pipeline-template, which has a Bake Manifest stage.
The stage uses HELM2 to bake a helm charts from GCS.
Pipeline created normally works fine, while piepline created using pipeline-template v1 terminated at Bake stage giving RetrofitError

Here is the pipeline template: https://gist.github.com/kcatstack/e9d1bca4bc0c7275e2d5f9dd0893393e
It gets triggered from docker, bakes a HELM manifest then deploy

Update 1: I tried deleted the second stage and leave only Bake stage as in https://gist.github.com/kcatstack/b6c3a70de8b677cd500733194adab926, the pipeline still failed

Update 2: I tried putting the Helm Charts in github instead of GCS as in https://gist.github.com/kcatstack/f84d45e3da55a273e2ff62fcf679f07c. The pipeline still failed with the same error

Cloud Provider(s):

Kubernetes

Environment:

GCP

Feature Area (if this issue is UI/UX related, please tag @spinnaker/ui-ux-team):

Description:

Error view from Source:

      "context": {
        "exception": {
          "details": {
            "error": "Internal Server Error",
            "errors": [
              "No message available"
            ],
            "kind": "HTTP",
            "responseBody": "{\"timestamp\":1556701481355,\"status\":500,\"error\":\"Internal Server Error\",\"exception\":\"java.lang.NullPointerException\",\"message\":\"No message available\"}",
            "rootException": "java.lang.NullPointerException",
            "status": 500,
            "url": "http://spin-rosco.spinnaker:8087/api/v2/manifest/bake/helm"
          },
          "exceptionType": "RetrofitError",
          "operation": "createBake",
          "shouldRetry": false,
          "timestamp": 1556701481359
        },
        "expectedArtifacts": [
          {
            "defaultArtifact": {},
            "id": "bakedManifestArtifact",
            "matchArtifact": {
              "kind": "base64",
              "name": "baked-charts",
              "type": "embedded/base64"
            },
            "useDefaultArtifact": false
          }
        ],
        "id": "67473048-a26c-4616-bec2-3e819d4c3d2e",
        "inputArtifacts": [
          {
            "account": "ops-internal-gcs-artifact-account",
            "id": "helmChartsArtifact"
          }
        ],
        "namespace": "staging",
        "outputName": "spinnaker-demo-node",
        "templateRenderer": "HELM2"
      }

Rosco log:

---> HTTP PUT http://spin-clouddriver.spinnaker:7002/artifacts/fetch/
<--- HTTP 200 http://spin-clouddriver.spinnaker:7002/artifacts/fetch/ (134ms)
2019-05-01 09:54:14.872 ERROR 1 --- [.0-8087-exec-10] c.n.s.k.w.e.GenericExceptionHandlers     : Internal Server Error

java.lang.NullPointerException: null
        at com.netflix.spinnaker.rosco.manifests.helm.HelmTemplateUtils.buildBakeRecipe(HelmTemplateUtils.java:54) ~[rosco-manifests.jar:na]
        at com.netflix.spinnaker.rosco.manifests.helm.HelmBakeManifestService.bake(HelmBakeManifestService.java:43) ~[rosco-manifests.jar:na]
        at com.netflix.spinnaker.rosco.controllers.V2BakeryController.doBake(V2BakeryController.java:22) ~[rosco-web.jar:na]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_191]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_191]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_191]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_191]
        at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) ~[spring-web-4.3.20.RELEASE.jar:4.3.20.RELEASE]
        at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133) ~[spring-web-4.3.20.RELEASE.jar:4.3.20.RELEASE]
        at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97) ~[spring-webmvc-4.3.20.RELEASE.jar:4.3.20.RELEASE]
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:849) ~[spring-webmvc-4.3.20.RELEASE.jar:4.3.20.RELEASE]
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:760) ~[spring-webmvc-4.3.20.RELEASE.jar:4.3.20.RELEASE]
        at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85) ~[spring-webmvc-4.3.20.RELEASE.jar:4.3.20.RELEASE]
        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967) ~[spring-webmvc-4.3.20.RELEASE.jar:4.3.20.RELEASE]
        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901) ~[spring-webmvc-4.3.20.RELEASE.jar:4.3.20.RELEASE]
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970) [spring-webmvc-4.3.20.RELEASE.jar:4.3.20.RELEASE]
        at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872) [spring-webmvc-4.3.20.RELEASE.jar:4.3.20.RELEASE]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:661) [tomcat-embed-core-8.5.34.jar:8.5.34]
        at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846) [spring-webmvc-4.3.20.RELEASE.jar:4.3.20.RELEASE]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:742) [tomcat-embed-core-8.5.34.jar:8.5.34]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) [tomcat-embed-core-8.5.34.jar:8.5.34]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)[tomcat-embed-core-8.5.34.jar:8.5.34]
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) [tomcat-embed-websocket-8.5.34.jar:8.5.34]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.34.jar:8.5.34]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)[tomcat-embed-core-8.5.34.jar:8.5.34]
        at org.springframework.boot.web.filter.ApplicationContextHeaderFilter.doFilterInternal(ApplicationContextHeaderFilter.java:55) [spring-boot-1.5.17.RELEASE.jar:1.5.17.RELEASE]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.20.RELEASE.jar:4.3.20.RELEASE]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.34.jar:8.5.34]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)[tomcat-embed-core-8.5.34.jar:8.5.34]
        at org.springframework.web.filter.ShallowEtagHeaderFilter.doFilterInternal(ShallowEtagHeaderFilter.java:110) [spring-web-4.3.20.RELEASE.jar:4.3.20.RELEASE]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.20.RELEASE.jar:4.3.20.RELEASE]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.34.jar:8.5.34]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)[tomcat-embed-core-8.5.34.jar:8.5.34]
        at javax.servlet.FilterChain$doFilter.call(Unknown Source) [tomcat-embed-core-8.5.34.jar:8.5.34]
        at com.netflix.spinnaker.rosco.filters.SimpleCORSFilter.doFilter(SimpleCORSFilter.groovy:34)[rosco-web.jar:na]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.34.jar:8.5.34]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)[tomcat-embed-core-8.5.34.jar:8.5.34]
        at org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:111) [spring-boot-actuator-1.5.17.RELEASE.jar:1.5.17.RELEASE]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.20.RELEASE.jar:4.3.20.RELEASE]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.34.jar:8.5.34]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)[tomcat-embed-core-8.5.34.jar:8.5.34]
        at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) [spring-web-4.3.20.RELEASE.jar:4.3.20.RELEASE]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.20.RELEASE.jar:4.3.20.RELEASE]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.34.jar:8.5.34]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)[tomcat-embed-core-8.5.34.jar:8.5.34]
        at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:109) [spring-web-4.3.20.RELEASE.jar:4.3.20.RELEASE]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.20.RELEASE.jar:4.3.20.RELEASE]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.34.jar:8.5.34]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)[tomcat-embed-core-8.5.34.jar:8.5.34]
        at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:93) [spring-web-4.3.20.RELEASE.jar:4.3.20.RELEASE]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.20.RELEASE.jar:4.3.20.RELEASE]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.34.jar:8.5.34]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)[tomcat-embed-core-8.5.34.jar:8.5.34]
        at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197) [spring-web-4.3.20.RELEASE.jar:4.3.20.RELEASE]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.20.RELEASE.jar:4.3.20.RELEASE]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.34.jar:8.5.34]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)[tomcat-embed-core-8.5.34.jar:8.5.34]
        at org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:103) [spring-boot-actuator-1.5.17.RELEASE.jar:1.5.17.RELEASE]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.20.RELEASE.jar:4.3.20.RELEASE]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.34.jar:8.5.34]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)[tomcat-embed-core-8.5.34.jar:8.5.34]
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198) [tomcat-embed-core-8.5.34.jar:8.5.34]
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) [tomcat-embed-core-8.5.34.jar:8.5.34]
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:493) [tomcat-embed-core-8.5.34.jar:8.5.34]
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140) [tomcat-embed-core-8.5.34.jar:8.5.34]
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81) [tomcat-embed-core-8.5.34.jar:8.5.34]
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) [tomcat-embed-core-8.5.34.jar:8.5.34]
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342) [tomcat-embed-core-8.5.34.jar:8.5.34]
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:800) [tomcat-embed-core-8.5.34.jar:8.5.34]
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) [tomcat-embed-core-8.5.34.jar:8.5.34]
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:806) [tomcat-embed-core-8.5.34.jar:8.5.34]
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1498) [tomcat-embed-core-8.5.34.jar:8.5.34]
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-embed-core-8.5.34.jar:8.5.34]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_191]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_191]
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-8.5.34.jar:8.5.34]
        at java.lang.Thread.run(Thread.java:748) [na:1.8.0_191]

2019-05-01 09:54:14.882  WARN 1 --- [.0-8087-exec-10] .m.m.a.ExceptionHandlerExceptionResolver : Resolved [java.lang.NullPointerException]

Orca log:

2019-05-01 09:54:14.899  INFO 1 --- [    handlers-13] c.n.s.orca.bakery.api.BakeryService      : [[email protected]] <--- HTTP 500 http://spin-rosco.spinnaker:8087/api/v2/manifest/bake/helm (194ms)
2019-05-01 09:54:14.908 ERROR 1 --- [    handlers-13] c.n.s.orca.q.handler.RunTaskHandler      : [] Error running CreateBakeManifestTask for pipeline[01D9SD1SWR6RWN45W7WB34FP2W]
retrofit.RetrofitError: 500
        at retrofit.RetrofitError.httpError(RetrofitError.java:40)
        at retrofit.RestAdapter$RestHandler.invokeRequest(RestAdapter.java:388)
        at retrofit.RestAdapter$RestHandler.invoke(RestAdapter.java:240)
        at com.sun.proxy.$Proxy102.bakeManifest(Unknown Source)
        at com.netflix.spinnaker.orca.bakery.tasks.manifests.CreateBakeManifestTask.execute(CreateBakeManifestTask.java:108)
        at com.netflix.spinnaker.orca.q.handler.RunTaskHandler$handle$1$2$1.invoke(RunTaskHandler.kt:100)
        at com.netflix.spinnaker.orca.q.handler.RunTaskHandler$handle$1$2$1.invoke(RunTaskHandler.kt:56)
        at com.netflix.spinnaker.orca.q.handler.RunTaskHandler.withLoggingContext(RunTaskHandler.kt:311)
        at com.netflix.spinnaker.orca.q.handler.RunTaskHandler.access$withLoggingContext(RunTaskHandler.kt:56)
        at com.netflix.spinnaker.orca.q.handler.RunTaskHandler$handle$1$2.invoke(RunTaskHandler.kt:99)
        at com.netflix.spinnaker.orca.q.handler.RunTaskHandler$handle$1$2.invoke(RunTaskHandler.kt:56)
        at com.netflix.spinnaker.orca.q.handler.AuthenticationAware$sam$java_util_concurrent_Callable$0.call(AuthenticationAware.kt)
        at com.netflix.spinnaker.security.AuthenticatedRequest.lambda$propagate$0(AuthenticatedRequest.java:97)
        at com.netflix.spinnaker.orca.q.handler.AuthenticationAware$DefaultImpls.withAuth(AuthenticationAware.kt:49)
        at com.netflix.spinnaker.orca.q.handler.RunTaskHandler.withAuth(RunTaskHandler.kt:56)
        at com.netflix.spinnaker.orca.q.handler.RunTaskHandler$handle$1.invoke(RunTaskHandler.kt:98)
        at com.netflix.spinnaker.orca.q.handler.RunTaskHandler$handle$1.invoke(RunTaskHandler.kt:56)
        at com.netflix.spinnaker.orca.q.handler.RunTaskHandler$withTask$1.invoke(RunTaskHandler.kt:182)
        at com.netflix.spinnaker.orca.q.handler.RunTaskHandler$withTask$1.invoke(RunTaskHandler.kt:56)
        at com.netflix.spinnaker.orca.q.handler.OrcaMessageHandler$withTask$1.invoke(OrcaMessageHandler.kt:49)
        at com.netflix.spinnaker.orca.q.handler.OrcaMessageHandler$withTask$1.invoke(OrcaMessageHandler.kt:33)
        at com.netflix.spinnaker.orca.q.handler.OrcaMessageHandler$withStage$1.invoke(OrcaMessageHandler.kt:59)
        at com.netflix.spinnaker.orca.q.handler.OrcaMessageHandler$withStage$1.invoke(OrcaMessageHandler.kt:33)
        at com.netflix.spinnaker.orca.q.handler.OrcaMessageHandler$DefaultImpls.withExecution(OrcaMessageHandler.kt:68)
        at com.netflix.spinnaker.orca.q.handler.RunTaskHandler.withExecution(RunTaskHandler.kt:56)
        at com.netflix.spinnaker.orca.q.handler.OrcaMessageHandler$DefaultImpls.withStage(OrcaMessageHandler.kt:55)
        at com.netflix.spinnaker.orca.q.handler.RunTaskHandler.withStage(RunTaskHandler.kt:56)
        at com.netflix.spinnaker.orca.q.handler.OrcaMessageHandler$DefaultImpls.withTask(OrcaMessageHandler.kt:42)
        at com.netflix.spinnaker.orca.q.handler.RunTaskHandler.withTask(RunTaskHandler.kt:56)
        at com.netflix.spinnaker.orca.q.handler.RunTaskHandler.withTask(RunTaskHandler.kt:175)
        at com.netflix.spinnaker.orca.q.handler.RunTaskHandler.handle(RunTaskHandler.kt:69)
        at com.netflix.spinnaker.orca.q.handler.RunTaskHandler.handle(RunTaskHandler.kt:56)
        at com.netflix.spinnaker.q.MessageHandler$DefaultImpls.invoke(MessageHandler.kt:36)
        at com.netflix.spinnaker.orca.q.handler.OrcaMessageHandler$DefaultImpls.invoke(OrcaMessageHandler.kt)
        at com.netflix.spinnaker.orca.q.handler.RunTaskHandler.invoke(RunTaskHandler.kt:56)
        at com.netflix.spinnaker.orca.q.audit.ExecutionTrackingMessageHandlerPostProcessor$ExecutionTrackingMessageHandlerProxy.invoke(ExecutionTrackingMessageHandlerPostProcessor.kt:47)
        at com.netflix.spinnaker.q.QueueProcessor$pollOnce$1$1.run(QueueProcessor.kt:82)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Steps to Reproduce:

1. Create a pipeline from pipeline-template
2. Manually run the pipeline with certain docker image tag
3. The pipeline stops at the first stage (Bake Manifest) due to RetrofitError

Additional Details:

[port3000]

Finally figure out, I need to add

overrides: {}

in the Bake stage

[icoolchn]

@kcatstack Hey ,How to solve of, I also encountered the same problem, thank you.

[port3000]

@icoolchn Probably you forgot some fields in the pipeline-template. For me, I forgot the overrides field in Bake stage.