Argo: failed to save outputs: verify serviceaccount argo:default has necessary privileges

[ghost]

Is this a BUG REPORT or FEATURE REQUEST?:

What happened:
when I submit the example hello specify node in some nodes , it happen this, message: failed to save outputs: verify serviceaccount argo:default has necessary privileges"
time="2018-09-27T05:31:07Z" level=info msg="No more retries left. Failing..." namespace=argo workflow=hello-7g25v

and I try to , it did not work

kubectl create rolebinding default-admin --clusterrole=admin --serviceaccount=default:default

argo version 2.2

my yaml has not any outpout

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: hello-world-
spec:
  entrypoint: whalesay
  templates:
  - name: whalesay
    container:
      image: docker/whalesay:latest
      command: [cowsay]
      args: ["hello world"]
    tolerations:
      - key: CriticalAddonsOnly
        operator: Exists
    affinity:
      nodeAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
            - matchExpressions:
                - key: kubernetes.io/hostname
                  operator: In
                  values:
                    - mtlab-xm-10-255-0-78

[niekoost]

I had to add the namespace (--namespace=argo) to the kubectl create rolebinding command as well.

kubectl create rolebinding default-admin --clusterrole=admin --serviceaccount=argo:default --namespace=argo

This is in a situation where I explicitly provide the namespace to the argo submit command

 argo submit --watch ./workflow.yaml --namespace=argo

[jessesuen]

The error "failed to save outputs: verify serviceaccount argo:default has necessary privileges" is misleading since it fails to capture a lot of issues that fail in the wait sidecar.

We'll need the wait sidecar logs to capture the true error.

[thundergolfer]

For me, checking the sidecar logs yielded:

time="2019-06-17T08:11:49Z" level=fatal msg="pods \"REDACTED-phzm5-2352864016\" is forbidden: User \"system:serviceaccount:argo:default\" cannot get resource \"pods\" in API group \"\" in the namespace 
\"argo\"\ngithub.com/argoproj/argo/errors.Wrap\n\t/root/go/src/github.com/argoproj/argo/errors/errors.go:87\ngithub.com/argoproj/argo/errors.InternalWrapError\n\t/root/go/src/github.com/argoproj/argo/errors/errors.go:70\ngithub.com/argoproj/argo/workflow/executor.
(*WorkflowExecutor).getPod\n\t/root/go/src/github.com/argoproj/argo/workflow/executor/executor.go:506\ngithub.com/argoproj/argo/workflow/executor.
(*WorkflowExecutor).GetMainContainerStatus\n\t/root/go/src/github.com/argoproj/argo/workflow/executor/executor.go:548\ngithub.com/argoproj/argo/workflow/executor.
(*WorkflowExecutor).GetMainContainerID\n\t/root/go/src/github.com/argoproj/argo/workflow/executor/executor.go:565\ngithub.com/argoproj/argo/workflow/executor.
(*WorkflowExecutor).SaveParameters\n\t/root/go/src/github.com/argoproj/argo/workflow/executor/executor.go:332\ngithub.com/argoproj/argo/cmd/argoexec/commands.waitContainer\n\t/root/go/src/github.com/argoproj/argo/cmd/argoexec/commands/wait.go:49\ngithub.com/argoproj/argo/cmd/argoexec/commands.glob..func4\n\t/root/go/src/github.com/argoproj/argo/cmd/argoexec/commands/wait.go:19\ngithub.com/argoproj/argo/vendor/github.com/spf13/cobra.
(*Command).execute\n\t/root/go/src/github.com/argoproj/argo/vendor/github.com/spf13/cobra/command.go:766\ngithub.com/argoproj/argo/vendor/github.com/spf13/cobra.
(*Command).ExecuteC\n\t/root/go/src/github.com/argoproj/argo/vendor/github.com/spf13/cobra/command.go:852\ngithub.com/argoproj/argo/vendor/github.com/spf13/cobra.
(*Command).Execute\n\t/root/go/src/github.com/argoproj/argo/vendor/github.com/spf13/cobra/command.go:800\nmain.main\n\t/root/go/src/github.com/argoproj/argo/cmd/argoexec/main.go:15\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:198\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:2361"

[thundergolfer]

This turned out to be because we hadn't granted the pods in our workflows serviceAccount: argo, so they had the default service account which didn't have permissions. 😖

[BobLiu20]

@thundergolfer do you know why our pods working in default, but not argo?

[thundergolfer]

You'd have to clarify what you mean by this:

why our pods working in default

Which pods are these? The Argo workflow pods are working in default, but don't work if you try run those same pods in the argo namespace?

[RaviHari]

I had similar issue tried running initially in both default and argo namespaces:

time="2019-10-11T08:13:13Z" level=fatal msg="pods \"hello-world-kmnsr\" is forbidden: User \"system:serviceaccount:argo:default\" cannot get resource \"pods\" in API group \"\" in the namespace \"argo\"\ngithub.com/argoproj/argo/errors.Wrap\n\t/go/src/github.com/argoproj/argo/errors/errors.go:88\ngithub.com/argoproj/argo/errors.InternalWrapError\n\t/go/src/github.com/argoproj/argo/errors/errors.go:71\ngithub.com/argoproj/argo/workflow/executor.(WorkflowExecutor).getPod\n\t/go/src/github.com/argoproj/argo/workflow/executor/executor.go:639\ngithub.com/argoproj/argo/workflow/executor.(WorkflowExecutor).KillSidecars\n\t/go/src/github.com/argoproj/argo/workflow/executor/executor.go:1088\ngithub.com/argoproj/argo/cmd/argoexec/commands.waitContainer\n\t/go/src/github.com/argoproj/argo/cmd/argoexec/commands/wait.go:56\ngithub.com/argoproj/argo/cmd/argoexec/commands.NewWaitCommand.func1\n\t/go/src/github.com/argoproj/argo/cmd/argoexec/commands/wait.go:16\ngithub.com/spf13/cobra.(Command).execute\n\t/go/src/github.com/spf13/cobra/command.go:766\ngithub.com/spf13/cobra.(Command).ExecuteC\n\t/go/src/github.com/spf13/cobra/command.go:852\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/src/github.com/spf13/cobra/command.go:800\nmain.main\n\t/go/src/github.com/argoproj/argo/cmd/argoexec/main.go:17\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:201\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1333"

But after creating a new minikube cluster (by deleting old minikube instance with this command: minikube delete) and giving the following permissions in the default ns:
kubectl create rolebinding default-admin --clusterrole=admin --serviceaccount=default:default

it started working

Name: hello-world-zr99l
Namespace: default
ServiceAccount: default
Status: Succeeded
Created: Fri Oct 11 14:07:24 +0530 (7 seconds ago)
Started: Fri Oct 11 14:07:24 +0530 (7 seconds ago)
Finished: Fri Oct 11 14:07:31 +0530 (now)
Duration: 7 seconds

STEP PODNAME DURATION MESSAGE
✔ hello-world-zr99l hello-world-zr99l 6s

[thundergolfer]

kubectl create rolebinding default-admin --clusterrole=admin --serviceaccount=default:default

Doing that will make it work but do note that it's not good security-wise.

[dcnsakthi]

kubectl create rolebinding default-admin --clusterrole=admin --serviceaccount=argo:default --namespace=argo