Argo: Dashboard Is Showing False Negative Results

Created on 28 Jan 2020  路  1Comment  路  Source: argoproj/argo

Checklist:

  • [x] I've included the version.
  • [x] I've included reproduction steps.
  • [x] I've included the workflow YAML.
  • [x] I've included the logs.

What happened:

Ran a basic hello-world workflow

---
apiVersion: argoproj.io/v1alpha1
kind: Workflow                  # new type of k8s spec
metadata:
  generateName: hello-world-    # name of the workflow spec
  namespace: octopus
spec:
  entrypoint: whalesay          # invoke the whalesay template
  templates:
  - name: whalesay              # name of the template
    container:
      image: docker/whalesay
      command: [cowsay]
      args: ["hello world"]
      resources:                # limit the resources
        limits:
          memory: 32Mi
          cpu: 100m

dashboard

workflow

What you expected to happen:
Successful reading both on the UI dashboard as well as in the context of k8s workflow

How to reproduce it (as minimally and precisely as possible):

---
apiVersion: v1
kind: Namespace
metadata:
  name: octopus
spec:
  finalizers:
  - kubernetes
status:
  phase: Active
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: workflows.argoproj.io
  namespace: octopus
spec:
  additionalPrinterColumns:
  - JSONPath: .status.phase
    description: Status of the workflow
    name: Status
    type: string
  - JSONPath: .status.startedAt
    description: When the workflow was started
    format: date-time
    name: Age
    type: date
  group: argoproj.io
  names:
    kind: Workflow
    plural: workflows
    shortNames:
    - wf
  scope: Namespaced
  version: v1alpha1
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: workflowtemplates.argoproj.io
  namespace: octopus
spec:
  group: argoproj.io
  names:
    kind: WorkflowTemplate
    plural: workflowtemplates
    shortNames:
    - wftmpl
  scope: Namespaced
  version: v1alpha1
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: workflow-controller
  namespace: octopus
spec:
  selector:
    matchLabels:
      app: workflow-controller
  template:
    metadata:
      namespace: octopus
      labels:
        app: workflow-controller
    spec:
      containers:
      - args:
        - --configmap
        - workflow-controller-configmap
        - --executor-image
        - argoproj/argoexec:v2.5.0-rc2
        command:
        - workflow-controller
        image: argoproj/workflow-controller:v2.5.0-rc2
        name: workflow-controller
      serviceAccountName: argo
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: workflow-controller-configmap
  namespace: octopus
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: argo
  namespace: octopus
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: cronworkflows.argoproj.io
  namespace: octopus
spec:
  group: argoproj.io
  names:
    kind: CronWorkflow
    plural: cronworkflows
    shortNames:
      - cronwf
      - cwf
  scope: Namespaced
  version: v1alpha1
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: argo-server
  namespace: octopus
spec:
  selector:
    matchLabels:
      app: argo-server
  template:
    metadata:
      namespace: octopus
      labels:
        app: argo-server
    spec:
      containers:
      - args:
        - server
        image: argoproj/argocli:v2.5.0-rc2
        name: argo-server
        ports:
        - containerPort: 2746
        readinessProbe:
          httpGet:
            path: /
            port: 2746
            scheme: HTTP
          initialDelaySeconds: 5
          periodSeconds: 30
      serviceAccountName: argo-server
---
apiVersion: v1
kind: Service
metadata:
  name: argo-server
  namespace: octopus
spec:
  ports:
  - port: 2746
    targetPort: 2746
  selector:
    app: argo-server
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: argo-server
  namespace: octopus
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: argo-role
  namespace: octopus
rules:
- apiGroups:
  - ""
  resources:
  - secrets
  verbs:
  - get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  namespace: octopus
  labels:
    rbac.authorization.k8s.io/aggregate-to-admin: "true"
  name: argo-aggregate-to-admin
rules:
- apiGroups:
  - argoproj.io
  resources:
  - workflows
  - workflows/finalizers
  - workflowtemplates
  - workflowtemplates/finalizers
  - cronworkflows
  - cronworkflows/finalizers
  verbs:
  - create
  - delete
  - deletecollection
  - get
  - list
  - patch
  - update
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  namespace: octopus
  labels:
    rbac.authorization.k8s.io/aggregate-to-edit: "true"
  name: argo-aggregate-to-edit
rules:
- apiGroups:
  - argoproj.io
  resources:
  - workflows
  - workflows/finalizers
  - workflowtemplates
  - workflowtemplates/finalizers
  - cronworkflows
  - cronworkflows/finalizers
  verbs:
  - create
  - delete
  - deletecollection
  - get
  - list
  - patch
  - update
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  namespace: octopus
  labels:
    rbac.authorization.k8s.io/aggregate-to-view: "true"
  name: argo-aggregate-to-view
rules:
- apiGroups:
  - argoproj.io
  resources:
  - workflows
  - workflows/finalizers
  - workflowtemplates
  - workflowtemplates/finalizers
  - cronworkflows
  - cronworkflows/finalizers
  verbs:
  - get
  - list
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: argo-cluster-role
  namespace: octopus
rules:
- apiGroups:
  - ""
  resources:
  - pods
  - pods/exec
  verbs:
  - create
  - get
  - list
  - watch
  - update
  - patch
  - delete
- apiGroups:
  - ""
  resources:
  - configmaps
  verbs:
  - get
  - watch
  - list
- apiGroups:
  - ""
  resources:
  - persistentvolumeclaims
  verbs:
  - create
  - delete
- apiGroups:
  - argoproj.io
  resources:
  - workflows
  - workflows/finalizers
  verbs:
  - get
  - list
  - watch
  - update
  - patch
  - delete
  - create
- apiGroups:
  - argoproj.io
  resources:
  - workflowtemplates
  - workflowtemplates/finalizers
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - serviceaccounts
  verbs:
  - get
  - list
- apiGroups:
  - argoproj.io
  resources:
  - cronworkflows
  - cronworkflows/finalizers
  verbs:
  - get
  - list
  - watch
  - update
  - patch
  - delete
- apiGroups:
  - ""
  resources:
  - events
  verbs:
  - create
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: argo-server-cluster-role
  namespace: octopus
rules:
- apiGroups:
  - ""
  resources:
  - configmaps
  verbs:
  - get
  - watch
  - list
- apiGroups:
  - ""
  resources:
  - secrets
  verbs:
  - get
- apiGroups:
  - ""
  resources:
  - pods
  - pods/exec
  - pods/log
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - secrets
  verbs:
  - get
- apiGroups:
  - argoproj.io
  resources:
  - workflows
  - workflowtemplates
  - cronworkflows
  verbs:
  - create
  - get
  - list
  - watch
  - update
  - patch
  - delete
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: argo-binding
  namespace: octopus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: argo-cluster-role
subjects:
- kind: ServiceAccount
  name: argo
  namespace: octopus
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: argo-server-binding
  namespace: octopus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: argo-server-cluster-role
subjects:
- kind: ServiceAccount
  name: argo-server
  namespace: octopus
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: argo-binding
  namespace: octopus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: argo-role
subjects:
- kind: ServiceAccount
  name: argo
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: default-admin
  namespace: octopus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: admin
subjects:
- kind: ServiceAccount
  name: default
  namespace: octopus
- kind: ServiceAccount
  name: argo
  namespace: octopus
- kind: ServiceAccount
  name: argo-server
  namespace: octopus

Anything else we need to know?:

Environment:

  • Argo version: 
argoproj/workflow-controller:v2.5.0-rc2
argoproj/argocli:v2.5.0-rc2
  • Kubernetes version :
Server Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.5", GitCommit:"0e9fcb426b100a2aea5ed5c25b3d8cfbb01a8acf", GitTreeState:"clean", BuildDate:"2019-08-05T09:13:08Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}

Other debugging information (if applicable):

  • workflow result:
hello-world-8fb2g   Error    10m
  • executor logs:
kubectl logs hello-world-8fb2g -c init
kubectl logs hello-world-8fb2g -c wait
  • workflow-controller logs:
kubectl logs -n argo $(kubectl get pods -l app=workflow-controller -n argo -o name)

Logs

kubectl get wf -n octopus
NAME                STATUS   AGE
hello-world-8fb2g   Error    10m

kubectl logs hello-world-8fb2g -c wait
time="2020-01-28T19:31:02Z" level=info msg="Creating a docker executor"
time="2020-01-28T19:31:02Z" level=info msg="Executor (version: v0.0.0+unknown, build_date: 1970-01-01T00:00:00Z) initialized (pod: octopus/hello-world-8fb2g) with template:\n{\"name\":\"whalesay\",\"arguments\":{},\"inputs\":{},\"outputs\":{},\"metadata\":{},\"container\":{\"name\":\"\",\"image\":\"docker/whalesay\",\"command\":[\"cowsay\"],\"args\":[\"hello world\"],\"resources\":{\"limits\":{\"cpu\":\"100m\",\"memory\":\"32Mi\"}}}}"
time="2020-01-28T19:31:02Z" level=info msg="Waiting on main container"
time="2020-01-28T19:31:07Z" level=info msg="main container started with container ID: d663ace322b2c3f40de96a82447f85c8d73304d148b17c7bb243fed3edc56158"
time="2020-01-28T19:31:07Z" level=info msg="Starting annotations monitor"
time="2020-01-28T19:31:07Z" level=info msg="docker wait d663ace322b2c3f40de96a82447f85c8d73304d148b17c7bb243fed3edc56158"
time="2020-01-28T19:31:07Z" level=info msg="Starting deadline monitor"
time="2020-01-28T19:31:07Z" level=info msg="Main container completed"
time="2020-01-28T19:31:07Z" level=info msg="No output parameters"
time="2020-01-28T19:31:07Z" level=info msg="No output artifacts"
time="2020-01-28T19:31:07Z" level=info msg="No Script output reference in workflow. Capturing script output ignored"
time="2020-01-28T19:31:07Z" level=info msg="Annotations monitor stopped"
time="2020-01-28T19:31:07Z" level=info msg="Killing sidecars"
time="2020-01-28T19:31:07Z" level=info msg="Alloc=7229 TotalAlloc=12521 Sys=70846 NumGC=4 Goroutines=9"

kubectl logs hello-world-8fb2g -c main
 _____________
< hello world >
 -------------
    \
     \
      \
                    ##        .
              ## ## ##       ==
           ## ## ## ##      ===
       /""""""""""""""""___/ ===
  ~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ /  ===- ~~~
       \______ o          __/
        \    \        __/
          \____\______/


kubectl logs -n octopus $(kubectl get pods -l app=workflow-controller -n octopus -o name)
time="2020-01-28T19:37:25Z" level=warning msg="ConfigMap 'workflow-controller-configmap' does not have key 'config'"
time="2020-01-28T19:37:25Z" level=info msg="Starting CronWorkflow controller"
time="2020-01-28T19:37:25Z" level=info msg="Workflow Controller (version: v0.0.0+unknown) starting"
time="2020-01-28T19:37:25Z" level=info msg="Workers: workflow: 8, pod: 8"
time="2020-01-28T19:37:25Z" level=info msg="Watch Workflow controller config map updates"
time="2020-01-28T19:37:25Z" level=info msg="Starting workflow TTL controller (resync 20m0s)"
time="2020-01-28T19:37:25Z" level=info msg="Performing periodic GC every 5m0s"
time="2020-01-28T19:37:25Z" level=info msg="Detected ConfigMap update. Updating the controller config."
time="2020-01-28T19:37:25Z" level=warning msg="ConfigMap 'workflow-controller-configmap' does not have key 'config'"
time="2020-01-28T19:37:25Z" level=info msg="Started workflow TTL worker"

Workflow controller ran into a CrashLoopBackOff

workflow-controller.log


Message from the maintainers:

If you are impacted by this bug please add a 馃憤 reaction to this issue! We often sort issues this way to know what to prioritize.

bug
Source
picture derekahn

Most helpful comment

This is fixed in rc4

picture alexec on 28 Jan 2020
🎉1 👍1

>All comments

This is fixed in rc4

alexec picture alexec on 28 Jan 2020
🎉1 👍1
Was this page helpful?
0 / 5 - 0 ratings

Related issues

nelsonfassis picture nelsonfassis  路  4Comments
kounoike picture kounoike  路  4Comments
hden picture hden  路  3Comments
vicaire picture vicaire  路  4Comments
0xdevalias picture 0xdevalias  路  3Comments