Argo-cd: Setting proxy in argocd server fails to start

[ksaito1125]

Checklist:

• [ ] I've searched in the docs and FAQ for my answer: http://bit.ly/argocd-faq.
• [x] I've included steps to reproduce the bug.
• [x] I've pasted the output of argocd version.

Describe the bug

I run Argo CD behind a proxy server.

I set proxy to environment variable of argocd-repo-server and installed ArgoCD. It worked fine, but argocd-server could not communicate with github.

After configuring proxy in the same way for argocd-server and installing ArgoCD, argocd-server failed to start.

To Reproduce

Set proxy in argocd-repo-server and start.

$ helm repo add argo https://argoproj.github.io/argo-helm
$ kubectl create namespace argocd
$ kubectl ns argocd
$ diff -u <(helm inspect values argo/argo-cd) values-proxy.yaml
...
@@ -494,7 +494,13 @@

   ## Environment variables to pass to argocd-repo-server
   ##
-  env: []
+  env:
+  - name: http_proxy
+    value: http://proxy:8080/
+  - name: https_proxy
+    value: http://proxy:8080/
+  - name: no_proxy
+    value: argocd-repo-server,argocd-application-controller,argocd-metrics,argocd-server,argocd-server-metrics,argocd-redis,10.0.0.0/8
...
$ helm install --values=values-proxy.yaml argocd argo/argo-cd --namespace=argocd
$ kubectl get pod
NAME                                             READY   STATUS    RESTARTS   AGE
argocd-application-controller-8466f9bd5b-mkwhd   1/1     Running   0          57s
argocd-dex-server-865d7bc898-nmldc               1/1     Running   0          57s
argocd-redis-59dd8cb6f6-2nvqq                    1/1     Running   0          57s
argocd-repo-server-5784d4dcf5-srmtz              1/1     Running   0          57s
argocd-server-7f4dc4cd64-kclcp                   1/1     Running   0          57s

Once, uninstall argocd.

$ helm uninstall argocd --namespace=argocd

Set proxy in argocd-server and install again.

$ git diff
diff --git a/values-proxy.yaml b/values-proxy.yaml
index 306ae14..2a54a15 100644
--- a/values-proxy.yaml
+++ b/values-proxy.yaml
@@ -260,7 +260,13 @@ server:

   ## Environment variables to pass to argocd-server
   ##
-  env: []
+  env:
+  - name: http_proxy
+    value: http://proxy:8080/
+  - name: https_proxy
+    value: http://proxy:8080/
+  - name: no_proxy
+    value: argocd-repo-server,argocd-application-controller,argocd-metrics,argocd-server,argocd-server-metrics,argocd-redis,10.0.0.0/8

   ## Argo server log level
   logLevel: info
$ helm install --values=values-proxy.yaml argocd argo/argo-cd --namespace=argocd
$ kubectl get pod
NAME                                             READY   STATUS    RESTARTS   AGE
argocd-application-controller-8466f9bd5b-zt6s5   1/1     Running   0          62s
argocd-dex-server-865d7bc898-z8c8n               0/1     Error     3          62s
argocd-redis-59dd8cb6f6-qgg5x                    1/1     Running   0          62s
argocd-repo-server-5784d4dcf5-f5d5x              1/1     Running   0          62s

argocd-server-7f5cf69ff6-skqnd                   0/1     Running   1          62s

Expected behavior

I expected that github.com repository could be used by setting proxy in argocd-server
Looking at the log, it seems that argocd-server could not get the configmap required for startup.
I think no_proxy needs additional settings, but I didn't know what to set.

Version

$ argocd version
argocd: v1.3.6+89be1c9
  BuildDate: 2019-12-10T22:46:45Z
  GitCommit: 89be1c9ce6db0f727c81277c1cfdfb1e385bf248
  GitTreeState: clean
  GoVersion: go1.12.6
  Compiler: gc
  Platform: linux/amd64
argocd-server: v1.3.6+89be1c9
  BuildDate: 2019-12-10T22:47:48Z
  GitCommit: 89be1c9ce6db0f727c81277c1cfdfb1e385bf248
  GitTreeState: clean
  GoVersion: go1.12.6
  Compiler: gc
  Platform: linux/amd64
  Ksonnet Version: v0.13.1
  Kustomize Version: Version: {Version:kustomize/v3.2.1 GitCommit:d89b448c745937f0cf1936162f26a5aac688f840 BuildDate:2019-09-27T00:10:52Z GoOs:linux GoArch:amd64}
  Helm Version: v2.15.2
  Kubectl Version: v1.14.0

Logs

Log at normal startup with proxy setting only for argocd-repo-server.

$ stern argocd-server
+ argocd-server-7f4dc4cd64-kclcp › server
argocd-server-7f4dc4cd64-kclcp server time="2020-01-09T04:56:50Z" level=info msg="Starting configmap/secret informers"
argocd-server-7f4dc4cd64-kclcp server time="2020-01-09T04:56:50Z" level=info msg="Configmap/secret informer synced"
argocd-server-7f4dc4cd64-kclcp server time="2020-01-09T04:56:50Z" level=info msg="Initialized server signature"
argocd-server-7f4dc4cd64-kclcp server time="2020-01-09T04:56:50Z" level=info msg="Initialized admin password"
argocd-server-7f4dc4cd64-kclcp server time="2020-01-09T04:56:50Z" level=info msg="Initialized TLS certificate"
argocd-server-7f4dc4cd64-kclcp server time="2020-01-09T04:56:50Z" level=info msg="configmap informer cancelled"
argocd-server-7f4dc4cd64-kclcp server time="2020-01-09T04:56:50Z" level=info msg="Starting configmap/secret informers"
argocd-server-7f4dc4cd64-kclcp server time="2020-01-09T04:56:51Z" level=info msg="Configmap/secret informer synced"
argocd-server-7f4dc4cd64-kclcp server time="2020-01-09T04:56:51Z" level=info msg="secrets informer cancelled"
argocd-server-7f4dc4cd64-kclcp server 2020/01/09 04:56:51 [Model:]
argocd-server-7f4dc4cd64-kclcp server 2020/01/09 04:56:51 r.r: sub, res, act, obj
argocd-server-7f4dc4cd64-kclcp server 2020/01/09 04:56:51 p.p: sub, res, act, obj, eft
argocd-server-7f4dc4cd64-kclcp server 2020/01/09 04:56:51 e.e: some(where (p_eft == allow)) && !some(where (p_eft == deny))
argocd-server-7f4dc4cd64-kclcp server 2020/01/09 04:56:51 m.m: g(r_sub, p_sub) && keyMatch(r_res, p_res) && keyMatch(r_act, p_act) && keyMatch(r_obj, p_obj)
argocd-server-7f4dc4cd64-kclcp server 2020/01/09 04:56:51 g.g: _, _
argocd-server-7f4dc4cd64-kclcp server 2020/01/09 04:56:51 [Policy:]
argocd-server-7f4dc4cd64-kclcp server 2020/01/09 04:56:51 [p :  sub, res, act, obj, eft :  []]
argocd-server-7f4dc4cd64-kclcp server 2020/01/09 04:56:51 [g :  _, _ :  []]
argocd-server-7f4dc4cd64-kclcp server 2020/01/09 04:56:51 [Role links for: g]
argocd-server-7f4dc4cd64-kclcp server time="2020-01-09T04:56:51Z" level=info msg="argocd v1.3.6+89be1c9 serving on port 8080 (url: https://argocd.example.com, tls: true, namespace: argocd, sso: false)"
argocd-server-7f4dc4cd64-kclcp server time="2020-01-09T04:56:51Z" level=info msg="0xc00081faa0 subscribed to settings updates"
argocd-server-7f4dc4cd64-kclcp server time="2020-01-09T04:56:51Z" level=info msg="Starting rbac config informer"
argocd-server-7f4dc4cd64-kclcp server time="2020-01-09T04:56:51Z" level=info msg="RBAC ConfigMap 'argocd-rbac-cm' added"

Failure log.

$ stern argocd-server
+ argocd-server-7f5cf69ff6-skqnd › server
argocd-server-7f5cf69ff6-skqnd server time="2020-01-09T05:00:50Z" level=info msg="Starting configmap/secret informers"
argocd-server-7f5cf69ff6-skqnd server I0109 05:01:09.661074       1 trace.go:82] Trace[692801166]: "Reflector github.com/argoproj/argo-cd/util/settings/settings.go:544 ListAndWatch" (started: 2020-01-09 05:00:50.169021218 +0000 UTC m=+0.147664721) (total time: 19.491985533s):
argocd-server-7f5cf69ff6-skqnd server Trace[692801166]: [19.491985533s] [19.491985533s] END
argocd-server-7f5cf69ff6-skqnd server E0109 05:01:09.661099       1 reflector.go:126] github.com/argoproj/argo-cd/util/settings/settings.go:544: Failed to list *v1.Secret: an error on the server ("") has prevented the request from succeeding (get secrets)
argocd-server-7f5cf69ff6-skqnd server I0109 05:01:10.701287       1 trace.go:82] Trace[1087694162]: "Reflector github.com/argoproj/argo-cd/util/settings/settings.go:540 ListAndWatch" (started: 2020-01-09 05:00:50.168314313 +0000 UTC m=+0.146957813) (total time: 20.532916909s):
argocd-server-7f5cf69ff6-skqnd server Trace[1087694162]: [20.532916909s] [20.532916909s] END
argocd-server-7f5cf69ff6-skqnd server E0109 05:01:10.701308       1 reflector.go:126] github.com/argoproj/argo-cd/util/settings/settings.go:540: Failed to list *v1.ConfigMap: an error on the server ("") has prevented the request from succeeding (get configmaps)

[Kyrklund]

Hi
I seem to have stumbled into the same rabbit hole with a proxy.
I can't connect to Github repos and have added the proxy information in env for the "argocd-repo-server" but no difference.
I then added the same config to the "argocd-server" and in the logs i got the following errors:

reflector.go:123] github.com/argoproj/argo-cd/util/settings/settings.go:600: Failed to list *v1.Secret: Get "https://172.17.0.1:443/api/v1/namespaces/argocd/secrets?limit=500&resourceVersion=0": Forbidden

So i then went ahead and added the 172.17.0.1 address.
The "argocd-server" pod now starts, but i get denied by our proxy instead

Failed to query provider "https://argocd.example.com/api/dex": 403 Forbidden:
And even if I add this host to no_proxy list I still get denied by the proxy.

I will continue to test and see if i can figure out a working config.

[toVersus]

@Kyrklund Could you check if argocd-dex-server is addded in NO_PROXY env value. I think the 403 status code would be returned when argocd-server is trying to connect to dex server (http://argocd-dex-server:5556) for SSO. The argocd-dex-server value is missing in https://github.com/argoproj/argo-cd/issues/2954#issue-547270068 's suggested proxy setting.

  env:
  - name: http_proxy
    value: http://proxy:8080/
  - name: https_proxy
    value: http://proxy:8080/
  - name: NO_PROXY
     value: argocd-repo-server,argocd-application-controller,argocd-metrics,argocd-server,argocd-server-metrics,argocd-redis,argocd-dex-server,10.0.0.0/8

[Kyrklund]

@toVersus That could absolutely be something missing in the config. At the moment we settled for just letting our developers use our in-house private repositories.
But if we change our mind and go back trying to implement external access i'll be sure to check if the Dex server is present.

Thank you so much for the tip