Argo: Default Archive Location Error

Created on 17 Apr 2020  Â·  26Comments  Â·  Source: argoproj/argo

I followed the default getting started document on argo website :
https://argoproj.github.io/docs/argo/getting-started.html

Followed until step 7 ( after minio setup )
Error:

Name:                artifact-passing-mwwct
Namespace:           default
ServiceAccount:      default
Status:              Failed
Message:             child 'artifact-passing-mwwct[0].generate-artifact' errored
Created:             Fri Apr 17 20:16:42 +0500 (10 seconds from now)
Started:             Fri Apr 17 20:16:42 +0500 (10 seconds from now)
Finished:            Fri Apr 17 20:16:42 +0500 (10 seconds from now)
Duration:            0 seconds

STEP                       TEMPLATE          PODNAME                            DURATION  MESSAGE
 ✖ artifact-passing-mwwct  artifact-example                                               child 'artifact-passing-mwwct[0].generate-artifact' errored
 â””---âš  generate-artifact   whalesay          artifact-passing-mwwct-3765465502  0s        controller is not configured with a default archive location

I'm using Azure Kuberenetes Service (AKS)

Argo Version:

argo: v2.7.0
  BuildDate: 2020-03-31T23:35:04Z
  GitCommit: 4d1175eb68f6578ed5d599f877be9b4855d33ce9
  GitTreeState: clean
  GitTag: v2.7.0
  GoVersion: go1.13.4
  Compiler: gc
  Platform: linux/amd64

Let me know if more info is needed to look into this issue.

Most helpful comment

Configuration:\nartifactRepository: {}\nfeatureFlags: {}\nmetricsConfig:\n disableLegacy: false\npodSpecLogStrategy: {}\ntelemetryConfig:\n disableLegacy: false\n"

It is not loading the config. This is a problem with your installation, but I suspect your controller is not running in the correct namespace. If it is, are you running latest? If so, try changing your config:

# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
data:
    # DELETE `config:`
    artifactRepository:  |
      s3:
        bucket: my-bucket
        endpoint: argo-artifacts.default:9000
        insecure: true
        # accessKeySecret and secretKeySecret are secret selectors.
        # It references the k8s secret named 'argo-artifacts'
        # which was created during the minio helm install. The keys,
        # 'accesskey' and 'secretkey', inside that secret are where the
        # actual minio credentials are stored.
        accessKeySecret:
          name: argo-artifacts
          key: accesskey
        secretKeySecret:
          name: argo-artifacts
          key: secretkey
kind: ConfigMap
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"ConfigMap","metadata":{"annotations":{},"name":"workflow-controller-configmap","namespace":"argo"}}
  creationTimestamp: "2020-04-17T15:03:15Z"
  name: workflow-controller-configmap
  namespace: argo
  resourceVersion: "2250"
  selfLink: /api/v1/namespaces/argo/configmaps/workflow-controller-configmap
  uid: 2aa67b8c-9ef5-4ebc-9b5a-a8342b40acd0

All 26 comments

Could you please check the controller logs for errors?

After getting the error, these are logs of the controller:

time="2020-04-20T06:12:46Z" level=info msg="Processing workflow" namespace=default workflow=output-artifact-s3-hjspp
time="2020-04-20T06:12:46Z" level=info msg="Updated phase  -> Running" namespace=default workflow=output-artifact-s3-hjspp
time="2020-04-20T06:12:46Z" level=info msg="Pod node {output-artifact-s3-hjspp output-artifact-s3-hjspp output-artifact-s3-hjspp Pod whalesay nil    Pending   2020-04-20 06:12:46.457068818 +0000 UTC 0001-01-01 00:00:00 +0000 UTC   <nil
> nil nil [] []} initialized Pending" namespace=default workflow=output-artifact-s3-hjspp
time="2020-04-20T06:12:46Z" level=error msg="Mark error node output-artifact-s3-hjspp: controller is not configured with a default archive location\ngithub.com/argoproj/argo/errors.New\n\t/go/src/github.com/argoproj/argo/errors/errors.
go:49\ngithub.com/argoproj/argo/errors.Errorf\n\t/go/src/github.com/argoproj/argo/errors/errors.go:55\ngithub.com/argoproj/argo/workflow/controller.(*wfOperationCtx).addArchiveLocation\n\t/go/src/github.com/argoproj/argo/workflow/contr
oller/workflowpod.go:937\ngithub.com/argoproj/argo/workflow/controller.(*wfOperationCtx).createWorkflowPod\n\t/go/src/github.com/argoproj/argo/workflow/controller/workflowpod.go:165\ngithub.com/argoproj/argo/workflow/controller.(*wfOpe
rationCtx).executeContainer\n\t/go/src/github.com/argoproj/argo/workflow/controller/operator.go:1749\ngithub.com/argoproj/argo/workflow/controller.(*wfOperationCtx).executeTemplate\n\t/go/src/github.com/argoproj/argo/workflow/controlle
r/operator.go:1432\ngithub.com/argoproj/argo/workflow/controller.(*wfOperationCtx).operate\n\t/go/src/github.com/argoproj/argo/workflow/controller/operator.go:281\ngithub.com/argoproj/argo/workflow/controller.(*WorkflowController).proc
essNextItem\n\t/go/src/github.com/argoproj/argo/workflow/controller/controller.go:411\ngithub.com/argoproj/argo/workflow/controller.(*WorkflowController).runWorker\n\t/go/src/github.com/argoproj/argo/workflow/controller/controller.go:3
25\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/ut
il/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:88\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357" namespace=default workflow=output-ar
tifact-s3-hjspp
time="2020-04-20T06:12:46Z" level=info msg="node &NodeStatus{ID:output-artifact-s3-hjspp,Name:output-artifact-s3-hjspp,DisplayName:output-artifact-s3-hjspp,Type:Pod,TemplateName:whalesay,TemplateRef:nil,Phase:Pending,BoundaryID:,Messag
e:,StartedAt:2020-04-20 06:12:46.457068818 +0000 UTC,FinishedAt:0001-01-01 00:00:00 +0000 UTC,PodIP:,Daemoned:nil,Inputs:nil,Outputs:nil,Children:[],OutboundNodes:[],StoredTemplateID:,WorkflowTemplateName:,TemplateScope:,ResourcesDurat
ion:ResourcesDuration{},} phase Pending -> Error" namespace=default workflow=output-artifact-s3-hjspp
time="2020-04-20T06:12:46Z" level=info msg="node &NodeStatus{ID:output-artifact-s3-hjspp,Name:output-artifact-s3-hjspp,DisplayName:output-artifact-s3-hjspp,Type:Pod,TemplateName:whalesay,TemplateRef:nil,Phase:Error,BoundaryID:,Message:
,StartedAt:2020-04-20 06:12:46.457068818 +0000 UTC,FinishedAt:0001-01-01 00:00:00 +0000 UTC,PodIP:,Daemoned:nil,Inputs:nil,Outputs:nil,Children:[],OutboundNodes:[],StoredTemplateID:,WorkflowTemplateName:,TemplateScope:,ResourcesDuratio
n:ResourcesDuration{},} message: controller is not configured with a default archive location" namespace=default workflow=output-artifact-s3-hjspp
time="2020-04-20T06:12:46Z" level=info msg="node &NodeStatus{ID:output-artifact-s3-hjspp,Name:output-artifact-s3-hjspp,DisplayName:output-artifact-s3-hjspp,Type:Pod,TemplateName:whalesay,TemplateRef:nil,Phase:Error,BoundaryID:,Message:
controller is not configured with a default archive location,StartedAt:2020-04-20 06:12:46.457068818 +0000 UTC,FinishedAt:2020-04-20 06:12:46.457966194 +0000 UTC,PodIP:,Daemoned:nil,Inputs:nil,Outputs:nil,Children:[],OutboundNodes:[],S
toredTemplateID:,WorkflowTemplateName:,TemplateScope:,ResourcesDuration:ResourcesDuration{},} finished: 2020-04-20 06:12:46.457966194 +0000 UTC" namespace=default workflow=output-artifact-s3-hjspp
time="2020-04-20T06:12:46Z" level=error msg="output-artifact-s3-hjspp error in entry template execution: controller is not configured with a default archive location\ngithub.com/argoproj/argo/errors.New\n\t/go/src/github.com/argoproj/a
rgo/errors/errors.go:49\ngithub.com/argoproj/argo/errors.Errorf\n\t/go/src/github.com/argoproj/argo/errors/errors.go:55\ngithub.com/argoproj/argo/workflow/controller.(*wfOperationCtx).addArchiveLocation\n\t/go/src/github.com/argoproj/a
rgo/workflow/controller/workflowpod.go:937\ngithub.com/argoproj/argo/workflow/controller.(*wfOperationCtx).createWorkflowPod\n\t/go/src/github.com/argoproj/argo/workflow/controller/workflowpod.go:165\ngithub.com/argoproj/argo/workflow/
controller.(*wfOperationCtx).executeContainer\n\t/go/src/github.com/argoproj/argo/workflow/controller/operator.go:1749\ngithub.com/argoproj/argo/workflow/controller.(*wfOperationCtx).executeTemplate\n\t/go/src/github.com/argoproj/argo/
workflow/controller/operator.go:1432\ngithub.com/argoproj/argo/workflow/controller.(*wfOperationCtx).operate\n\t/go/src/github.com/argoproj/argo/workflow/controller/operator.go:281\ngithub.com/argoproj/argo/workflow/controller.(*Workfl
owController).processNextItem\n\t/go/src/github.com/argoproj/argo/workflow/controller/controller.go:411\ngithub.com/argoproj/argo/workflow/controller.(*WorkflowController).runWorker\n\t/go/src/github.com/argoproj/argo/workflow/controll
er/controller.go:325\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.
16.7-beta.0/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:88\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357" namespace=default
workflow=output-artifact-s3-hjspp
time="2020-04-20T06:12:46Z" level=info msg="Workflow update successful" namespace=default phase=Running resourceVersion=281296 workflow=output-artifact-s3-hjspp
time="2020-04-20T06:12:47Z" level=info msg="Processing workflow" namespace=default workflow=output-artifact-s3-hjspp
time="2020-04-20T06:12:47Z" level=info msg="Updated phase Running -> Error" namespace=default workflow=output-artifact-s3-hjspp
time="2020-04-20T06:12:47Z" level=info msg="Updated message  -> controller is not configured with a default archive location" namespace=default workflow=output-artifact-s3-hjspp
time="2020-04-20T06:12:47Z" level=info msg="Marking workflow completed" namespace=default workflow=output-artifact-s3-hjspp
time="2020-04-20T06:12:47Z" level=info msg="Checking daemoned children of " namespace=default workflow=output-artifact-s3-hjspp
time="2020-04-20T06:12:47Z" level=info msg="Workflow update successful" namespace=default phase=Error resourceVersion=281300 workflow=output-artifact-s3-hjspp

OK. You need to configure the default location in the workflow-controller-configmap:

https://github.com/argoproj/argo/blob/master/docs/workflow-controller-configmap.yaml

I did, as described on step 6, and this is my configmap:

# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
data:
  config: |
    artifactRepository: |
      s3:
        bucket: my-bucket
        endpoint: argo-artifacts.default:9000
        insecure: true
        # accessKeySecret and secretKeySecret are secret selectors.
        # It references the k8s secret named 'argo-artifacts'
        # which was created during the minio helm install. The keys,
        # 'accesskey' and 'secretkey', inside that secret are where the
        # actual minio credentials are stored.
        accessKeySecret:
          name: argo-artifacts
          key: accesskey
        secretKeySecret:
          name: argo-artifacts
          key: secretkey
kind: ConfigMap
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"ConfigMap","metadata":{"annotations":{},"name":"workflow-controller-configmap","namespace":"argo"}}
  creationTimestamp: "2020-04-17T15:03:15Z"
  name: workflow-controller-configmap
  namespace: argo
  resourceVersion: "2250"
  selfLink: /api/v1/namespaces/argo/configmaps/workflow-controller-configmap
  uid: 2aa67b8c-9ef5-4ebc-9b5a-a8342b40acd0

The error is pretty explicit, so I'm a bit confused. Can you check that the controller logs show the correct configuration please?

Ok so now I copied the secret in both default and argo namespace and tried to run artifact sample and error remains same, here are the controller logs:

time="2020-04-20T16:07:50Z" level=info msg="Processing workflow" namespace=default workflow=artifact-passing-8l5dt
time="2020-04-20T16:07:50Z" level=info msg="Updated phase  -> Running" namespace=default workflow=artifact-passing-8l5dt
time="2020-04-20T16:07:50Z" level=info msg="Steps node {artifact-passing-8l5dt artifact-passing-8l5dt artifact-passing-8l5dt Steps artifact-example nil    Running   2020-04-20 16:07:50.689468384 +0000 UTC 0001-01-01 00:00:00 +0000 UTC
  <nil> nil nil [] []} initialized Running" namespace=default workflow=artifact-passing-8l5dt
time="2020-04-20T16:07:50Z" level=info msg="StepGroup node {artifact-passing-8l5dt-134184679 artifact-passing-8l5dt[0] [0] StepGroup artifact-example nil    Running artifact-passing-8l5dt  2020-04-20 16:07:50.689575493 +0000 UTC 0001-0
1-01 00:00:00 +0000 UTC   <nil> nil nil [] []} initialized Running" namespace=default workflow=artifact-passing-8l5dt
time="2020-04-20T16:07:50Z" level=info msg="Pod node {artifact-passing-8l5dt-1146599573 artifact-passing-8l5dt[0].generate-artifact generate-artifact Pod whalesay nil    Pending artifact-passing-8l5dt  2020-04-20 16:07:50.690059235 +00
00 UTC 0001-01-01 00:00:00 +0000 UTC   <nil> nil nil [] []} initialized Pending" namespace=default workflow=artifact-passing-8l5dt
time="2020-04-20T16:07:50Z" level=error msg="Mark error node artifact-passing-8l5dt[0].generate-artifact: controller is not configured with a default archive location\ngithub.com/argoproj/argo/errors.New\n\t/go/src/github.com/argoproj/
argo/errors/errors.go:49\ngithub.com/argoproj/argo/errors.Errorf\n\t/go/src/github.com/argoproj/argo/errors/errors.go:55\ngithub.com/argoproj/argo/workflow/controller.(*wfOperationCtx).addArchiveLocation\n\t/go/src/github.com/argoproj/
argo/workflow/controller/workflowpod.go:937\ngithub.com/argoproj/argo/workflow/controller.(*wfOperationCtx).createWorkflowPod\n\t/go/src/github.com/argoproj/argo/workflow/controller/workflowpod.go:165\ngithub.com/argoproj/argo/workflow
/controller.(*wfOperationCtx).executeContainer\n\t/go/src/github.com/argoproj/argo/workflow/controller/operator.go:1749\ngithub.com/argoproj/argo/workflow/controller.(*wfOperationCtx).executeTemplate\n\t/go/src/github.com/argoproj/argo
/workflow/controller/operator.go:1432\ngithub.com/argoproj/argo/workflow/controller.(*wfOperationCtx).executeStepGroup\n\t/go/src/github.com/argoproj/argo/workflow/controller/steps.go:223\ngithub.com/argoproj/argo/workflow/controller.(
*wfOperationCtx).executeSteps\n\t/go/src/github.com/argoproj/argo/workflow/controller/steps.go:86\ngithub.com/argoproj/argo/workflow/controller.(*wfOperationCtx).executeTemplate\n\t/go/src/github.com/argoproj/argo/workflow/controller/o
perator.go:1434\ngithub.com/argoproj/argo/workflow/controller.(*wfOperationCtx).operate\n\t/go/src/github.com/argoproj/argo/workflow/controller/operator.go:281\ngithub.com/argoproj/argo/workflow/controller.(*WorkflowController).process
NextItem\n\t/go/src/github.com/argoproj/argo/workflow/controller/controller.go:411\ngithub.com/argoproj/argo/workflow/controller.(*WorkflowController).runWorker\n\t/go/src/github.com/argoproj/argo/workflow/controller/controller.go:325\
nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/
wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:88\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357" namespace=default workflow=artifact-pas
sing-8l5dt
time="2020-04-20T16:07:50Z" level=info msg="node &NodeStatus{ID:artifact-passing-8l5dt-1146599573,Name:artifact-passing-8l5dt[0].generate-artifact,DisplayName:generate-artifact,Type:Pod,TemplateName:whalesay,TemplateRef:nil,Phase:Pendi
ng,BoundaryID:artifact-passing-8l5dt,Message:,StartedAt:2020-04-20 16:07:50.690059235 +0000 UTC,FinishedAt:0001-01-01 00:00:00 +0000 UTC,PodIP:,Daemoned:nil,Inputs:nil,Outputs:nil,Children:[],OutboundNodes:[],StoredTemplateID:,Workflow
TemplateName:,TemplateScope:,ResourcesDuration:ResourcesDuration{},} phase Pending -> Error" namespace=default workflow=artifact-passing-8l5dt
time="2020-04-20T16:07:50Z" level=info msg="node &NodeStatus{ID:artifact-passing-8l5dt-1146599573,Name:artifact-passing-8l5dt[0].generate-artifact,DisplayName:generate-artifact,Type:Pod,TemplateName:whalesay,TemplateRef:nil,Phase:Error
,BoundaryID:artifact-passing-8l5dt,Message:,StartedAt:2020-04-20 16:07:50.690059235 +0000 UTC,FinishedAt:0001-01-01 00:00:00 +0000 UTC,PodIP:,Daemoned:nil,Inputs:nil,Outputs:nil,Children:[],OutboundNodes:[],StoredTemplateID:,WorkflowTe
mplateName:,TemplateScope:,ResourcesDuration:ResourcesDuration{},} message: controller is not configured with a default archive location" namespace=default workflow=artifact-passing-8l5dt
time="2020-04-20T16:07:50Z" level=info msg="node &NodeStatus{ID:artifact-passing-8l5dt-1146599573,Name:artifact-passing-8l5dt[0].generate-artifact,DisplayName:generate-artifact,Type:Pod,TemplateName:whalesay,TemplateRef:nil,Phase:Error
,BoundaryID:artifact-passing-8l5dt,Message:controller is not configured with a default archive location,StartedAt:2020-04-20 16:07:50.690059235 +0000 UTC,FinishedAt:2020-04-20 16:07:50.690536476 +0000 UTC,PodIP:,Daemoned:nil,Inputs:nil
,Outputs:nil,Children:[],OutboundNodes:[],StoredTemplateID:,WorkflowTemplateName:,TemplateScope:,ResourcesDuration:ResourcesDuration{},} finished: 2020-04-20 16:07:50.690536476 +0000 UTC" namespace=default workflow=artifact-passing-8l5
dt
time="2020-04-20T16:07:50Z" level=info msg="Step group node &NodeStatus{ID:artifact-passing-8l5dt-134184679,Name:artifact-passing-8l5dt[0],DisplayName:[0],Type:StepGroup,TemplateName:artifact-example,TemplateRef:nil,Phase:Running,Bound
aryID:artifact-passing-8l5dt,Message:,StartedAt:2020-04-20 16:07:50.689575493 +0000 UTC,FinishedAt:0001-01-01 00:00:00 +0000 UTC,PodIP:,Daemoned:nil,Inputs:nil,Outputs:nil,Children:[],OutboundNodes:[],StoredTemplateID:,WorkflowTemplate
Name:,TemplateScope:,ResourcesDuration:ResourcesDuration{},} deemed errored due to child artifact-passing-8l5dt[0].generate-artifact error: controller is not configured with a default archive location" namespace=default workflow=artifa
ct-passing-8l5dt
time="2020-04-20T16:07:50Z" level=info msg="node &NodeStatus{ID:artifact-passing-8l5dt-134184679,Name:artifact-passing-8l5dt[0],DisplayName:[0],Type:StepGroup,TemplateName:artifact-example,TemplateRef:nil,Phase:Running,BoundaryID:artif
act-passing-8l5dt,Message:,StartedAt:2020-04-20 16:07:50.689575493 +0000 UTC,FinishedAt:0001-01-01 00:00:00 +0000 UTC,PodIP:,Daemoned:nil,Inputs:nil,Outputs:nil,Children:[artifact-passing-8l5dt-1146599573],OutboundNodes:[],StoredTempla
teID:,WorkflowTemplateName:,TemplateScope:,ResourcesDuration:ResourcesDuration{},} phase Running -> Error" namespace=default workflow=artifact-passing-8l5dt
time="2020-04-20T16:07:50Z" level=info msg="node &NodeStatus{ID:artifact-passing-8l5dt-134184679,Name:artifact-passing-8l5dt[0],DisplayName:[0],Type:StepGroup,TemplateName:artifact-example,TemplateRef:nil,Phase:Error,BoundaryID:artifac
t-passing-8l5dt,Message:,StartedAt:2020-04-20 16:07:50.689575493 +0000 UTC,FinishedAt:0001-01-01 00:00:00 +0000 UTC,PodIP:,Daemoned:nil,Inputs:nil,Outputs:nil,Children:[artifact-passing-8l5dt-1146599573],OutboundNodes:[],StoredTemplate
ID:,WorkflowTemplateName:,TemplateScope:,ResourcesDuration:ResourcesDuration{},} message: child 'artifact-passing-8l5dt[0].generate-artifact' errored" namespace=default workflow=artifact-passing-8l5dt
time="2020-04-20T16:07:50Z" level=info msg="node &NodeStatus{ID:artifact-passing-8l5dt-134184679,Name:artifact-passing-8l5dt[0],DisplayName:[0],Type:StepGroup,TemplateName:artifact-example,TemplateRef:nil,Phase:Error,BoundaryID:artifac
t-passing-8l5dt,Message:child 'artifact-passing-8l5dt[0].generate-artifact' errored,StartedAt:2020-04-20 16:07:50.689575493 +0000 UTC,FinishedAt:2020-04-20 16:07:50.691062221 +0000 UTC,PodIP:,Daemoned:nil,Inputs:nil,Outputs:nil,Childre
n:[artifact-passing-8l5dt-1146599573],OutboundNodes:[],StoredTemplateID:,WorkflowTemplateName:,TemplateScope:,ResourcesDuration:ResourcesDuration{},} finished: 2020-04-20 16:07:50.691062221 +0000 UTC" namespace=default workflow=artifac
t-passing-8l5dt
time="2020-04-20T16:07:50Z" level=info msg="step group artifact-passing-8l5dt-134184679 was unsuccessful: child 'artifact-passing-8l5dt[0].generate-artifact' errored" namespace=default workflow=artifact-passing-8l5dt
time="2020-04-20T16:07:50Z" level=info msg="Outbound nodes of artifact-passing-8l5dt-1146599573 is [artifact-passing-8l5dt-1146599573]" namespace=default workflow=artifact-passing-8l5dt
time="2020-04-20T16:07:50Z" level=info msg="Outbound nodes of artifact-passing-8l5dt is [artifact-passing-8l5dt-1146599573]" namespace=default workflow=artifact-passing-8l5dt
time="2020-04-20T16:07:50Z" level=info msg="node &NodeStatus{ID:artifact-passing-8l5dt,Name:artifact-passing-8l5dt,DisplayName:artifact-passing-8l5dt,Type:Steps,TemplateName:artifact-example,TemplateRef:nil,Phase:Running,BoundaryID:,Me
ssage:,StartedAt:2020-04-20 16:07:50.689468384 +0000 UTC,FinishedAt:0001-01-01 00:00:00 +0000 UTC,PodIP:,Daemoned:nil,Inputs:nil,Outputs:nil,Children:[artifact-passing-8l5dt-134184679],OutboundNodes:[artifact-passing-8l5dt-1146599573],
StoredTemplateID:,WorkflowTemplateName:,TemplateScope:,ResourcesDuration:ResourcesDuration{},} phase Running -> Failed" namespace=default workflow=artifact-passing-8l5dt
time="2020-04-20T16:07:50Z" level=info msg="node &NodeStatus{ID:artifact-passing-8l5dt,Name:artifact-passing-8l5dt,DisplayName:artifact-passing-8l5dt,Type:Steps,TemplateName:artifact-example,TemplateRef:nil,Phase:Failed,BoundaryID:,Mes
sage:,StartedAt:2020-04-20 16:07:50.689468384 +0000 UTC,FinishedAt:0001-01-01 00:00:00 +0000 UTC,PodIP:,Daemoned:nil,Inputs:nil,Outputs:nil,Children:[artifact-passing-8l5dt-134184679],OutboundNodes:[artifact-passing-8l5dt-1146599573],S
toredTemplateID:,WorkflowTemplateName:,TemplateScope:,ResourcesDuration:ResourcesDuration{},} message: child 'artifact-passing-8l5dt[0].generate-artifact' errored" namespace=default workflow=artifact-passing-8l5dt
time="2020-04-20T16:07:50Z" level=info msg="node &NodeStatus{ID:artifact-passing-8l5dt,Name:artifact-passing-8l5dt,DisplayName:artifact-passing-8l5dt,Type:Steps,TemplateName:artifact-example,TemplateRef:nil,Phase:Failed,BoundaryID:,Mes
sage:child 'artifact-passing-8l5dt[0].generate-artifact' errored,StartedAt:2020-04-20 16:07:50.689468384 +0000 UTC,FinishedAt:2020-04-20 16:07:50.691537362 +0000 UTC,PodIP:,Daemoned:nil,Inputs:nil,Outputs:nil,Children:[artifact-passing
-8l5dt-134184679],OutboundNodes:[artifact-passing-8l5dt-1146599573],StoredTemplateID:,WorkflowTemplateName:,TemplateScope:,ResourcesDuration:ResourcesDuration{},} finished: 2020-04-20 16:07:50.691537362 +0000 UTC" namespace=default wor
kflow=artifact-passing-8l5dt
time="2020-04-20T16:07:50Z" level=info msg="Checking daemoned children of artifact-passing-8l5dt" namespace=default workflow=artifact-passing-8l5dt
time="2020-04-20T16:07:50Z" level=info msg="Updated phase Running -> Failed" namespace=default workflow=artifact-passing-8l5dt
time="2020-04-20T16:07:50Z" level=info msg="Updated message  -> child 'artifact-passing-8l5dt[0].generate-artifact' errored" namespace=default workflow=artifact-passing-8l5dt
time="2020-04-20T16:07:50Z" level=info msg="Marking workflow completed" namespace=default workflow=artifact-passing-8l5dt
time="2020-04-20T16:07:50Z" level=info msg="Checking daemoned children of " namespace=default workflow=artifact-passing-8l5dt
time="2020-04-20T16:07:50Z" level=info msg="Workflow update successful" namespace=default phase=Failed resourceVersion=325259 workflow=artifact-passing-8l5dt
time="2020-04-20T16:08:29Z" level=info msg="Alloc=4534 TotalAlloc=179789 Sys=70080 NumGC=2195 Goroutines=101"

The workflow controller prints it's configuration in the first few log lines - can you please attach those?

These are logs from the start:

time="2020-04-17T15:03:29Z" level=info msg="config map" name=workflow-controller-configmap
time="2020-04-17T15:03:29Z" level=info msg="Configuration:\nartifactRepository: {}\nfeatureFlags: {}\nmetricsConfig:\n  disableLegacy: false\npodSpecLogStrategy: {}\ntelemetryConfig:\n  disableLegacy: false\n"
time="2020-04-17T15:03:29Z" level=info msg="Persistence configuration disabled"
time="2020-04-17T15:03:29Z" level=info msg="Starting CronWorkflow controller"
time="2020-04-17T15:03:29Z" level=info msg="Starting Workflow Controller" version=vv2.7.4+50b209c.dirty
time="2020-04-17T15:03:29Z" level=info msg="Workers: workflow: 8, pod: 8"
time="2020-04-17T15:03:29Z" level=info msg="Starting workflow TTL controller (resync 20m0s)"
time="2020-04-17T15:03:29Z" level=info msg="Performing periodic GC every 5m0s"
time="2020-04-17T15:03:29Z" level=info msg="Started workflow TTL worker"
time="2020-04-17T15:04:52Z" level=info msg="Processing workflow" namespace=default workflow=loops-maps-9cv79
time="2020-04-17T15:04:52Z" level=info msg="Updated phase  -> Running" namespace=default workflow=loops-maps-9cv79
time="2020-04-17T15:04:52Z" level=info msg="Steps node {loops-maps-9cv79 loops-maps-9cv79 loops-maps-9cv79 Steps loop-map-example nil    Running   2020-04-17 15:04:52.684566096 +0000 UTC 0001-01-01 00:00:00 +0000 UTC   <nil> nil nil []
 []} initialized Running" namespace=default workflow=loops-maps-9cv79
time="2020-04-17T15:04:52Z" level=info msg="StepGroup node {loops-maps-9cv79-2560827875 loops-maps-9cv79[0] [0] StepGroup loop-map-example nil    Running loops-maps-9cv79  2020-04-17 15:04:52.684628 +0000 UTC 0001-01-01 00:00:00 +0000
UTC   <nil> nil nil [] []} initialized Running" namespace=default workflow=loops-maps-9cv79
time="2020-04-17T15:04:52Z" level=info msg="Pod node {loops-maps-9cv79-2603470149 loops-maps-9cv79[0].test-linux(0:image:debian,tag:9.1) test-linux(0:image:debian,tag:9.1) Pod cat-os-release nil    Pending loops-maps-9cv79  2020-04-17
15:04:52.685107735 +0000 UTC 0001-01-01 00:00:00 +0000 UTC   <nil> nil nil [] []} initialized Pending" namespace=default workflow=loops-maps-9cv79
time="2020-04-17T15:04:52Z" level=info msg="Created pod: loops-maps-9cv79[0].test-linux(0:image:debian,tag:9.1) (loops-maps-9cv79-2603470149)" namespace=default workflow=loops-maps-9cv79
time="2020-04-17T15:04:52Z" level=info msg="Pod node {loops-maps-9cv79-4146441795 loops-maps-9cv79[0].test-linux(1:image:debian,tag:8.9) test-linux(1:image:debian,tag:8.9) Pod cat-os-release nil    Pending loops-maps-9cv79  2020-04-17
15:04:52.70311084 +0000 UTC 0001-01-01 00:00:00 +0000 UTC   <nil> nil nil [] []} initialized Pending" namespace=default workflow=loops-maps-9cv79
time="2020-04-17T15:04:52Z" level=info msg="Created pod: loops-maps-9cv79[0].test-linux(1:image:debian,tag:8.9) (loops-maps-9cv79-4146441795)" namespace=default workflow=loops-maps-9cv79
time="2020-04-17T15:04:52Z" level=info msg="Pod node {loops-maps-9cv79-2502400288 loops-maps-9cv79[0].test-linux(2:image:alpine,tag:3.6) test-linux(2:image:alpine,tag:3.6) Pod cat-os-release nil    Pending loops-maps-9cv79  2020-04-17
15:04:52.711068117 +0000 UTC 0001-01-01 00:00:00 +0000 UTC   <nil> nil nil [] []} initialized Pending" namespace=default workflow=loops-maps-9cv79
time="2020-04-17T15:04:52Z" level=info msg="Created pod: loops-maps-9cv79[0].test-linux(2:image:alpine,tag:3.6) (loops-maps-9cv79-2502400288)" namespace=default workflow=loops-maps-9cv79
time="2020-04-17T15:04:52Z" level=info msg="Pod node {loops-maps-9cv79-3240014489 loops-maps-9cv79[0].test-linux(3:image:ubuntu,tag:17.10) test-linux(3:image:ubuntu,tag:17.10) Pod cat-os-release nil    Pending loops-maps-9cv79  2020-04
-17 15:04:52.724372382 +0000 UTC 0001-01-01 00:00:00 +0000 UTC   <nil> nil nil [] []} initialized Pending" namespace=default workflow=loops-maps-9cv79
time="2020-04-17T15:04:52Z" level=info msg="Created pod: loops-maps-9cv79[0].test-linux(3:image:ubuntu,tag:17.10) (loops-maps-9cv79-3240014489)" namespace=default workflow=loops-maps-9cv79
time="2020-04-17T15:04:52Z" level=info msg="Workflow step group node &NodeStatus{ID:loops-maps-9cv79-2560827875,Name:loops-maps-9cv79[0],DisplayName:[0],Type:StepGroup,TemplateName:loop-map-example,TemplateRef:nil,Phase:Running,Boundar
yID:loops-maps-9cv79,Message:,StartedAt:2020-04-17 15:04:52.684628 +0000 UTC,FinishedAt:0001-01-01 00:00:00 +0000 UTC,PodIP:,Daemoned:nil,Inputs:nil,Outputs:nil,Children:[loops-maps-9cv79-2603470149 loops-maps-9cv79-4146441795 loops-ma
ps-9cv79-2502400288 loops-maps-9cv79-3240014489],OutboundNodes:[],StoredTemplateID:,WorkflowTemplateName:,TemplateScope:,ResourcesDuration:ResourcesDuration{},} not yet completed" namespace=default workflow=loops-maps-9cv79
time="2020-04-17T15:04:52Z" level=info msg="Workflow update successful" namespace=default phase=Running resourceVersion=1302 workflow=loops-maps-9cv79
time="2020-04-17T15:04:53Z" level=info msg="Processing workflow" namespace=default workflow=loops-maps-9cv79

Configuration:\nartifactRepository: {}\nfeatureFlags: {}\nmetricsConfig:\n disableLegacy: false\npodSpecLogStrategy: {}\ntelemetryConfig:\n disableLegacy: false\n"

It is not loading the config. This is a problem with your installation, but I suspect your controller is not running in the correct namespace. If it is, are you running latest? If so, try changing your config:

# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
data:
    # DELETE `config:`
    artifactRepository:  |
      s3:
        bucket: my-bucket
        endpoint: argo-artifacts.default:9000
        insecure: true
        # accessKeySecret and secretKeySecret are secret selectors.
        # It references the k8s secret named 'argo-artifacts'
        # which was created during the minio helm install. The keys,
        # 'accesskey' and 'secretkey', inside that secret are where the
        # actual minio credentials are stored.
        accessKeySecret:
          name: argo-artifacts
          key: accesskey
        secretKeySecret:
          name: argo-artifacts
          key: secretkey
kind: ConfigMap
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"ConfigMap","metadata":{"annotations":{},"name":"workflow-controller-configmap","namespace":"argo"}}
  creationTimestamp: "2020-04-17T15:03:15Z"
  name: workflow-controller-configmap
  namespace: argo
  resourceVersion: "2250"
  selfLink: /api/v1/namespaces/argo/configmaps/workflow-controller-configmap
  uid: 2aa67b8c-9ef5-4ebc-9b5a-a8342b40acd0

Okay, removing the keyword config: and restarting the controller pod did the trick.
It works fine now. Thanks.

This is the 3rd exact issue for this and yet it doesn't work, as shown on the Getting Started doc. Maybe someone should actually update the source documentation instead of answering the same question (at least) 3 times in issues? Just a thought...

thanks alot, I got the same issue,for the doc use config:
https://argoproj.github.io/docs/argo/configure-artifact-repository.html#s3-compatible-artifact-repository-bucket-such-as-aws-gcs-and-minio
@talhairfanbentley

we should get rid of those docs

@alexmt can you help me? Can we get rid of those docs ?

Hi, I just started to learn and use Argo and K8s. I also hit this issue and the answers here (# DELETE config:\) helps. Thank you!

I also hit two other issues

  • The argo namespace is not used consistently through this getting-started tutorial. This is potentially confusing. For example, I couldn't see workflows I submitted, hello-world for example, after launching Argo UI (which is with argo namespace) since the instruction submitted it with default namespace.
  • I couldn't get minio artifacts to work with the instruction, endpoint: argo-artifacts.default:9000 specifically. I got Failed to put file: Get http://my-minio-endpoint.default:9000/my-bucket/?location=: dial tcp: lookup my-minio-endpoint.default on 10.96.0.10:53: no such host. I got it work by doing endpoint: argo-artifacts:9000.

I'm wondering if I could help with the getting-started documentation? Or do you have other big changes planned for this tutorial? @alexec

I'm wondering if I could help with the getting-started documentation? Or do you have other big changes planned for this tutorial? @alexec

Yes please! We'd love to see someone refresh these!

I'm wondering if I could help with the getting-started documentation? Or do you have other big changes planned for this tutorial? @alexec

Yes please! We'd love to see someone refresh these!

Thank you @alexec . Here is my PR: https://github.com/argoproj/argo/pull/3099

I couldn't get minio artifacts to work with the instruction, endpoint: argo-artifacts.default:9000 specifically. I got Failed to put file: Get http://my-minio-endpoint.default:9000/my-bucket/?location=: dial tcp: lookup my-minio-endpoint.default on 10.96.0.10:53: no such host. I got it work by doing endpoint: argo-artifacts:9000.

The docs assume you deploy Minio in the default namespace, so we should probably add some context to what the command is doing and not presume that a new Argo user understands the nuance of Kubernetes networking. I made a comment on the PR @yunhailuo

@yunhailuo
Any reason this error might pop up with a gcs configuration block but not a s3 block?

@yunhailuo
Any reason this error might pop up with a gcs configuration block but not a s3 block?

I'm not quite sure how you got this error. And I'm not sure the difference between your gcs and s3 block. So I can't really say anything.

The specific reason I got this before is namespace mismatch. I was following the previous version and used argo namespace while the artifact configure written there assume it's default. Fixing that solved my problem

we should get rid of those docs

@alexec not just the docs have to change.. the official helm chart is also broken in that regard:
https://github.com/argoproj/argo-helm/blob/master/charts/argo/templates/workflow-controller-config-map.yaml

we should get rid of those docs

@alexec not just the docs have to change.. the official helm chart is also broken in that regard:
https://github.com/argoproj/argo-helm/blob/master/charts/argo/templates/workflow-controller-config-map.yaml

How does that affect the helm chart?

we should get rid of those docs

@alexec not just the docs have to change.. the official helm chart is also broken in that regard:
https://github.com/argoproj/argo-helm/blob/master/charts/argo/templates/workflow-controller-config-map.yaml

How does that affect the helm chart?

It automatically adds the artifactRepository key inside config, which if I understand correctly is an error.

It automatically adds the artifactRepository key inside config, which if I understand correctly is an error.

Fair. But I guess my point was, what the helm chart had to do with deleting the docs? Or, more succinctly, how deleting those docs fixes the helm chart. Not trying to be argumentative, I just don't see the correlation. If the docs and the chart is wrong, it seems like they should both be fixed and/or removed.

@ntwrkguru my rhetorical point was that it wasn't the only "leftover" regarding this breaking change.
On one hand you are right that this should have been opened as an issue on the relevant repository and it's somehow unrelated here, and on the other I was trying to enlighten the fact that breaking changes should be communicated better and pushed across the board on all side repositories as well.
I was using "the docs" as a mere example of something that was accidentally overlooked during a breaking change, and since this isn't the first time I'm encountering breaking differences between argo products I do think that it's worth mentioning out load.

@iMoses Thanks for the explanation! I didn't catch the rhetorical bit, so my bad on that one. Agreed on your other points.

Was this page helpful?
0 / 5 - 0 ratings

Related issues

kounoike picture kounoike  Â·  4Comments

nelsonfassis picture nelsonfassis  Â·  4Comments

hden picture hden  Â·  3Comments

0xdevalias picture 0xdevalias  Â·  3Comments

srados picture srados  Â·  3Comments