Argo-cd: Timeouts / Possible memory issues?

[andye2004]

This isn't a feature request, there is already one of those in place but a question (maybe bug report?) regarding upgrading the helm binary to version 3.0.2.

I have followed the details here that outline how to change the binary version of helm but when I run argo in the cluster and try to connect to a private github repo I get the following error:

time="2020-01-09T10:13:08Z" level=error msg="finished unary call with code Unknown" error="Get https://github.com/private-repo/info/refs?service=git-upload-pack: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" grpc.code=Unknown grpc.method=Create grpc.service=repository.RepositoryService grpc.start_time="2020-01-09T10:12:53Z" grpc.time_ms=15001.043 span.kind=server system=grpc

Is there something else going on under the covers that prevents a simple upgrade of the helm binary, or, is this possibly a bug?

EDIT: This appears to be something else entirely. I have reverted to a standard install and I'm receiving what appears to be memory / timeout errors.

From argo-repo-server I'm seeing messages like:

time="2020-01-09T12:40:11Z" level=info msg="Alloc=3633 TotalAlloc=14805 Sys=70592 NumGC=19 Goroutines=9"

and from the argocd-server I'm seeing:

time="2020-01-09T12:41:54Z" level=info msg="finished unary call with code InvalidArgument" error="rpc error: code = InvalidArgument desc = application spec is invalid: InvalidSpecError: repository not accessible: Get https://github.com/privaterepoinfo/refs?service=git-upload-pack: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" grpc.code=InvalidArgument grpc.method=Create grpc.service=application.ApplicationService grpc.start_time="2020-01-09T12:41:39Z" grpc.time_ms=15006.049 span.kind=server system=grpc

This was trying to create an application AFTER about 30 attempts to actually create the github repo.

[andye2004]

I worked out what was going on here, it's a side effect of the init command being removed in helm3. It appears that ArgoCD calls helm init as part of the load process and it was the failure of this command that was leading to my issues. Closing this ticket.

[alexmt]

thank you for update @andye2004

[omgapuppy]

@andye2004 out of curiosity, were you able to get helm3 working with Argo? If so, how did you get around the lack of init?
Want to start using Argo, but we're not going to use helm2 out of security concerns.

[andye2004]

@omgapuppy Not by simply upgrading it as per the link at the top. It _should_ be possible to do it as a plugin although I didn't try, there were no specific features of Helm3 we were using so we could use Helm2.

If you were to do a Helm3 plugin you would lose the 'nice' feature of being able to over-ride the values in the argo UI, this was primarily the reason I didn't attempt the plugin route. We want users to be able to edit certain properties.

Regards, the security concerns around Helm2, argo-cd doesn't actually use tiller, it only runs helm inside a container called argocd-repo and uses helm init and helm template commands to generate complete manifests that are then applied to the cluster. I did wonder about the init command but I can't find a tiller instance running in my cluster so my assumption is that it is being run with the –client-only flag in order to set $HELM-HOME.

The argo guys have now teamed up with the weaveworks flux guys to combine their efforts to work on gitops-engine, maybe we'll need to wait for that to have helm3 coverage?

EDIT: Thinking about this a little more, it might (or should) be possible to add/update helm3 and proxy the helm calls. E.g. have a shell script that performs the helm init but then delegates other commands to the helm3 instance. This would then make it possible to use the newer features of helm3 but still be able to edit values in the argo UI. Mmmmmm., might take a loot at that later.

FURTHER EDIT: As if by magic my old friend ;) @travisghansen has saved me the bother and produced a wrapper script. See https://github.com/argoproj/argo-cd/issues/2383#issuecomment-576083328