Argo: Argo getting started fails with Kind

Created on 6 Mar 2020 · 6Comments · Source: argoproj/argo

Checklist:

[x] I've included the version.
[x] I've included reproduction steps.
[x] I've included the workflow YAML.
[x] I've included the logs.

What happened:

I'm trying Argo for the first time with Kind v0.7. The Hello World example in the "getting started" guide hangs indefinitely

https://github.com/argoproj/argo/blob/master/docs/getting-started.md

How to reproduce it (as minimally and precisely as possible):

Here are the commands I'm running:

$ kind version
kind v0.7.0 go1.13.4 linux/amd64
nick@dopey:~$ kind create cluster
Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.17.0) 🖼
 ✓ Preparing nodes 📦  
 ✓ Writing configuration 📜 
 ✓ Starting control-plane 🕹️ 
 ✓ Installing CNI 🔌 
 ✓ Installing StorageClass 💾 
Set kubectl context to "kind-kind"
You can now use your cluster with:

kubectl cluster-info --context kind-kind

Not sure what to do next? 😅 Check out https://kind.sigs.k8s.io/docs/user/quick-start/
nick@dopey:~$ kubectl create namespace argo
namespace/argo created
nick@dopey:~$ kubectl apply -n argo -f https://raw.githubusercontent.com/argoproj/argo/stable/manifests/install.yaml
customresourcedefinition.apiextensions.k8s.io/cronworkflows.argoproj.io created
customresourcedefinition.apiextensions.k8s.io/workflows.argoproj.io created
customresourcedefinition.apiextensions.k8s.io/workflowtemplates.argoproj.io created
serviceaccount/argo created
serviceaccount/argo-server created
role.rbac.authorization.k8s.io/argo-role created
clusterrole.rbac.authorization.k8s.io/argo-aggregate-to-admin created
clusterrole.rbac.authorization.k8s.io/argo-aggregate-to-edit created
clusterrole.rbac.authorization.k8s.io/argo-aggregate-to-view created
clusterrole.rbac.authorization.k8s.io/argo-cluster-role created
clusterrole.rbac.authorization.k8s.io/argo-server-cluster-role created
rolebinding.rbac.authorization.k8s.io/argo-binding created
clusterrolebinding.rbac.authorization.k8s.io/argo-binding created
clusterrolebinding.rbac.authorization.k8s.io/argo-server-binding created
configmap/workflow-controller-configmap created
service/argo-server created
deployment.apps/argo-server created
deployment.apps/workflow-controller created
nick@dopey:~$ kubectl create rolebinding default-admin --clusterrole=admin --serviceaccount=default:default
rolebinding.rbac.authorization.k8s.io/default-admin created
nick@dopey:~$ argo submit --watch https://raw.githubusercontent.com/argoproj/argo/master/examples/hello-world.yaml
Name:                hello-world-r6ljw
Namespace:           default
ServiceAccount:      default
Status:              Running
Created:             Thu Mar 05 20:46:22 -0500 (46 seconds ago)
Started:             Thu Mar 05 20:46:32 -0500 (36 seconds ago)
Duration:            36 seconds

STEP                             PODNAME            DURATION  MESSAGE
 ◷ hello-world-r6ljw (whalesay)  hello-world-r6ljw  36s       ContainerCreating

The job never completes. Here's the output of kubectl describe pods

$ kubectl describe pods
Name:           hello-world-r6ljw
Namespace:      default
Priority:       0
Node:           kind-control-plane/172.17.0.3
Start Time:     Thu, 05 Mar 2020 20:46:32 -0500
Labels:         workflows.argoproj.io/completed=false
                workflows.argoproj.io/workflow=hello-world-r6ljw
Annotations:    workflows.argoproj.io/node-name: hello-world-r6ljw
                workflows.argoproj.io/template:
                  {"name":"whalesay","inputs":{},"outputs":{},"metadata":{},"container":{"name":"","image":"docker/whalesay:latest","command":["cowsay"],"ar...
Status:         Pending
IP:             
IPs:            <none>
Controlled By:  Workflow/hello-world-r6ljw
Containers:
  wait:
    Container ID:  
    Image:         argoproj/argoexec:v2.6.1
    Image ID:      
    Port:          <none>
    Host Port:     <none>
    Command:
      argoexec
      wait
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Environment:
      ARGO_POD_NAME:  hello-world-r6ljw (v1:metadata.name)
    Mounts:
      /argo/podmetadata from podmetadata (rw)
      /var/run/docker.sock from docker-sock (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-kp476 (ro)
  main:
    Container ID:  
    Image:         docker/whalesay:latest
    Image ID:      
    Port:          <none>
    Host Port:     <none>
    Command:
      cowsay
    Args:
      hello world
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-kp476 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  podmetadata:
    Type:  DownwardAPI (a volume populated by information about the pod)
    Items:
      metadata.annotations -> annotations
  docker-sock:
    Type:          HostPath (bare host directory volume)
    Path:          /var/run/docker.sock
    HostPathType:  Socket
  default-token-kp476:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-kp476
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason       Age                From                         Message
  ----     ------       ----               ----                         -------
  Normal   Scheduled    30s                default-scheduler            Successfully assigned default/hello-world-r6ljw to kind-control-plane
  Warning  FailedMount  14s (x6 over 29s)  kubelet, kind-control-plane  MountVolume.SetUp failed for volume "docker-sock" : hostPath type check failed: /var/run/docker.sock is not a socket file

Anything else we need to know?:

As far as I can tell, Argo is trying to mount a Docker socket. This obviously doesn't work on Kind, because it uses a containerd runtime. Is Argo not expected to work on containerd? What should I be doing instead?

Environment:

Argo version:

$ argo version
argo: v2.6.1
  BuildDate: 2020-03-04T21:25:34Z
  GitCommit: 842739d7831cc5b417c4f524ed85288408a32bbf
  GitTreeState: clean
  GitTag: v2.6.1
  GoVersion: go1.13.4
  Compiler: gc
  Platform: linux/amd64

Kubernetes version :

$ kubectl version -o yaml
clientVersion:
  buildDate: "2020-02-12T13:43:46Z"
  compiler: gc
  gitCommit: 06ad960bfd03b39c8310aaf92d1e7c12ce618213
  gitTreeState: clean
  gitVersion: v1.17.3
  goVersion: go1.13.7
  major: "1"
  minor: "17"
  platform: linux/amd64
serverVersion:
  buildDate: "2020-01-14T00:09:19Z"
  compiler: gc
  gitCommit: 70132b0f130acc0bed193d9ba59dd186f0e634cf
  gitTreeState: clean
  gitVersion: v1.17.0
  goVersion: go1.13.4
  major: "1"
  minor: "17"
  platform: linux/amd64

Message from the maintainers:

If you are impacted by this bug please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.

Source

nicks

Most helpful comment

OK, I:

downloaded https://raw.githubusercontent.com/argoproj/argo/stable/manifests/install.yaml locally
replaced the resource:

apiVersion: v1
kind: ConfigMap
metadata:
  name: workflow-controller-configmap

with this

apiVersion: v1
kind: ConfigMap
metadata:
  name: workflow-controller-configmap
data:
  config: |
    containerRuntimeExecutor: k8sapi

And it worked OK.

Would it make sense to add these instructions to the Getting Started Guide? I couldn't find anything about this on
https://github.com/argoproj/argo/blob/master/docs/getting-started.md

nicks on 6 Mar 2020

👍2

All 6 comments

Would changing your Workflow executor help here?

simster7 on 6 Mar 2020

OK, I:

downloaded https://raw.githubusercontent.com/argoproj/argo/stable/manifests/install.yaml locally
replaced the resource:

apiVersion: v1
kind: ConfigMap
metadata:
  name: workflow-controller-configmap

with this

apiVersion: v1
kind: ConfigMap
metadata:
  name: workflow-controller-configmap
data:
  config: |
    containerRuntimeExecutor: k8sapi

And it worked OK.

Would it make sense to add these instructions to the Getting Started Guide? I couldn't find anything about this on
https://github.com/argoproj/argo/blob/master/docs/getting-started.md

nicks on 6 Mar 2020

👍2

Feel free to update that document for executor selection.