Argo-cd: Issue while connecting repo on kubernetes running on minikube + ec2 instance

[continuum-vikrant-tyagi]

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
Steps to reproduce the behavior:

1. Go to 'connect repo via cli/ui'
2. Click on '....'
3. Scroll down to '....'
4. See error

FATA[0030] rpc error: code = InvalidArgument desc = application spec is invalid: InvalidSpecError: No credentials available for source repository and repository is not publicly accessible: Get https://github.com/argoproj/argocd-example-apps.git/info/refs?service=git-upload-pack: dial tcp: i/o timeout

And while adding from UI geeting below error

Unable to connect repository: Get https://github.com/argoproj/argocd-example-apps.git/info/refs?service=git-upload-pack: dial tcp: lookup github.com on 10.96.0.10:53: read udp 172.17.0.9:45016->10.96.0.10:53: read: connection refused

Expected behavior
expected this is public repository and able to clone without any cred.

[alexec]

@continuum-vikrant-tyagi any chance you can provide the relevant section of the argocd-cm for this? I'd like to check it against https://argoproj.github.io/argo-cd/operator-manual/declarative-setup/#repositories

[continuum-vikrant-tyagi]

@alexec i've refer this https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

and i guess this is the relevant section for argocd-cm
apiVersion: v1
kind: ConfigMap
metadata:
labels:
app.kubernetes.io/name: argocd-cm
app.kubernetes.io/part-of: argocd
name: argocd-cm

Well some how i've seen this error via UI and CLI both

FATA[0015] rpc error: code = InvalidArgument desc = application spec is invalid: InvalidSpecError: No credentials available for source repository and repository is not publicly accessible: Get https://github.com/argoproj/argocd-example-apps.git/info/refs?service=git-upload-pack: dial tcp: lookup github.com on 10.96.0.10:53: read udp 172.17.0.9:53576->10.96.0.10:53: read: connection refused

[alexec]

It looks like from the error you have an egress rule blocking outbound connections.

You could try and kubectl exec into the pod and execute git clone REPO?

[continuum-vikrant-tyagi]

hi any specific pod?

[alexec]

repo server

[continuum-vikrant-tyagi]

Cloning into 'argocd-example-apps'...
fatal: unable to access 'https://github.com/argoproj/argocd-example-apps.git/': Could not resolve host: github.com

[continuum-vikrant-tyagi]

hi @alexec well have setup argocd again in another cluster with ingress configuration. but there is some wiered behaviour

not able to browse and even not connect with one of the worker node to the argocd-server .
Below is the curl output for the same.

curl -v 100.64.20.226:8080

• Rebuilt URL to: 100.64.20.226:8080/
• Trying 100.64.20.226...
• TCP_NODELAY set
• connect to 100.64.20.226 port 8080 failed: Connection timed out
• Failed to connect to 100.64.20.226 port 8080: Connection timed out
• Closing connection 0
  curl: (7) Failed to connect to 100.64.20.226 port 8080: Connection timed out

Also sharing below logs of the container argocd-server

#

LOGS

#

kubectl logs -f argocd-server-7c5cdd88f8-lpwjw -n argocd
time="2019-04-26T11:22:34Z" level=info msg="Starting configmap/secret informers"
time="2019-04-26T11:22:34Z" level=info msg="Configmap/secret informer synced"
time="2019-04-26T11:22:34Z" level=info msg="Initialized server signature"
time="2019-04-26T11:22:34Z" level=warning msg="Unable to parse updated settings: admin.password is missing"
time="2019-04-26T11:22:34Z" level=warning msg="Unable to parse updated settings: admin.password is missing"
time="2019-04-26T11:22:34Z" level=warning msg="Unable to parse updated settings: admin.password is missing"
time="2019-04-26T11:22:34Z" level=warning msg="Unable to parse updated settings: admin.password is missing"
time="2019-04-26T11:22:34Z" level=warning msg="Unable to parse updated settings: admin.password is missing"
time="2019-04-26T11:22:34Z" level=info msg="Initialized admin password"
time="2019-04-26T11:22:34Z" level=info msg="Initialized TLS certificate"
W0426 11:22:34.928872 1 listers.go:63] can not retrieve list of objects using index : Index with name namespace does not exist
time="2019-04-26T11:22:34Z" level=info msg="Starting configmap/secret informers"
time="2019-04-26T11:22:34Z" level=info msg="configmap informer cancelled"
time="2019-04-26T11:22:34Z" level=info msg="secrets informer cancelled"
time="2019-04-26T11:22:35Z" level=info msg="Configmap/secret informer synced"
2019/04/26 11:22:35 [Model:]
2019/04/26 11:22:35 r.r: sub, res, act, obj
2019/04/26 11:22:35 p.p: sub, res, act, obj, eft
2019/04/26 11:22:35 e.e: some(where (p_eft == allow)) && !some(where (p_eft == deny))
2019/04/26 11:22:35 m.m: g(r_sub, p_sub) && keyMatch(r_res, p_res) && keyMatch(r_act, p_act) && keyMatch(r_obj, p_obj)
2019/04/26 11:22:35 g.g: _, _
2019/04/26 11:22:35 [Policy:]
2019/04/26 11:22:35 [p : sub, res, act, obj, eft : []]
2019/04/26 11:22:35 [g : _, _ : []]
2019/04/26 11:22:35 [Role links for: g]
time="2019-04-26T11:22:35Z" level=info msg="argocd v0.12.2+ff0c23f serving on port 8080 (url: , tls: true, namespace: argocd, sso: false)"
time="2019-04-26T11:22:35Z" level=info msg="0xc000554720 subscribed to settings updates"
time="2019-04-26T11:22:35Z" level=info msg="Starting rbac config informer"
time="2019-04-26T11:22:35Z" level=info msg="RBAC ConfigMap 'argocd-rbac-cm' added"

#

kubectl get pods -n argocd -o wide

[alexec]

Your logs do hint that your cluster is blocking outbound connections to GitHub. This is unlikely to be an issue with Argo CD. Can you check your egress configuration?

[continuum-vikrant-tyagi]

well that not the case for sure, as it's a kops k8 cluster and already in service, we have running pipeline for CI/CD on certain environment.

It looks i might missing some configuration, we did configure with ingress rule.

[continuum-vikrant-tyagi]

this has been resolved with ingress rule, and update the deployment.yaml file as mentioned in ingress rule configuration

[Pradyumn]

@continuum-vikrant-tyagi can you share your ingress rule configuration.

[continuum-vikrant-tyagi]

we have user helm chart.

below is the rule, need to define in values.yaml. argo-cd default provide subdomain but in our case it was /argocd

ingress:
  enabled: true
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/backend-protocol: HTTP
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
    nginx.ingress.kubernetes.io/rewrite-target: /
  path: /argocd
  hosts:
    - www.example.com

and this i have defined under templates >> argocd-server-deployment.yaml

containers: - name: argocd-server image: {{ .Values.server.image.repository }}:{{ .Values.server.image.tag }} imagePullPolicy: {{ .Values.server.image.pullPolicy }} command: - argocd-server - --staticassets - /shared/app - --repo-server - argocd-repo-server:8081 - --insecure - --basehref - /argocd

[sivaguru-nathan]

@alexmt @alexec @continuum-vikrant-tyagi
ENV: ec2 , accessed by LoadBalancer
version:1.6.2

while trying to create an app, initially faced "client: time out exceeded error"

checked cloning repo inside argocd-server, argocd-repo-server not able to clone

changed DNSpolicy of all the deployment pods to Default. then able to clone inside container. but facing "transport: Error while dialing dial tcp: lookup argocd-repo-server on 172.31.0.2:53: nosuch host"

could anyone help me to solve this problem!!!

[srinivasraojangiti]

Can someone help on the same. while establishing connection to git repository I see this error[repository url is correct -here its masked]
Unable to connect HTTPS repository: Get "https://github.com/<>/argocd-demo/info/refs?service=git-upload-pack": dial tcp: lookup github.com on 10.96.0.10:53: read udp 10.244.2.211:53884->10.96.0.10:53: read: connection refused

Unable to connect HTTPS repository: Get "https://github.com/<>/argocd-demo/info/refs?service=git-upload-pack": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

And whats the reason , Cluster is reflecting as unknown