Argo-cd: Helm S3 Support

[theeltahir]

Summary

Repo server should leverage the Helm CLI for some of the calls to enable better Helm plugin support

Motivation

Currently our team uses S3 to store Helm charts. We are excited to use the 1st class Helm support features slated to arrive in Argo CD 1.3. After building a custom image with the Helm S3 plugin, adding an S3 bucket as a Helm repo results in the following errors

time="2019-10-24T15:52:24Z" level=error msg="finished unary call with code Unknown" error="Get s3://<s3-bucket-name>/charts/index.yaml: unsupported protocol scheme \"s3\"" grpc.code=Unknown grpc.method=GetHelmCharts grpc.request.deadline="2019-10-24T15:53:24Z" grpc.service=repository.RepoServerService grpc.start_time="2019-10-24T15:52:24Z" grpc.time_ms=0.084 span.kind=server system=grpc

After digging in the code further, looks like the GetIndex method acquires the chart index via http calls and not the Helm CLI:
https://github.com/argoproj/argo-cd/blob/5706a171550415308c5574322a1aa93ee5cd7ce2/util/helm/client.go#L153-L201

This would lead me to believe that our image with the Helm S3 plugin is ineffective as the http client does not understand the S3 calls.

Proposal

I have not spent enough time with the code base to know if it is feasible but if we leveraged the Helm CLI for some of these calls it might allow the Helm S3 to function properly

[jannfis]

Hm, personally I like the idea that ArgoCD could access Helm charts or repositories residing in S3 buckets, but I feel using helm CLI for fetching data from the repository might not be the right way to do it, for various reasons, i.e. you would have to build a custom image (as you did), containing your AWS credentials and additional Helm plugins. Also, ArgoCD certificate & credential management might not work with Helm CLI and custom plugins.

Maybe native S3 support for accessing repositories hosted on S3 in ArgoCD could work, i.e. by leveraging the Minio S3 client (https://github.com/minio/minio-go). This way, we could manage the credentials just like every other repository credentials (i.e. access key as username, secret key as password) and also use custom TLS certificates, in case someone hosts on private Minio instances exposing the S3 API.

What do you think about this idea?

[theeltahir]

Yeah i was not sure how much of a demand is for this feature, that is why I tried to go the S3 plugin route which we use in our current CI process (and Helmsman prior to Argo migration).

If the team is able to add S3 protocol support via minio, I would be super happy as it would reduce the need for us to build a custom image for this feature.

Edit: I would prefer that we also allow the possibility of using IAM roles for accessing the S3 repo (the argo repo server would be granted read-only access), it would make it less stressful for us for key rotation.

[jannfis]

Hm, I've just had a look at our Helm integration code, and it seems we use golang's http.Client for fetching the index.yaml from the chart repository, and then helm fetch ... for getting the actual chart. So it might actually work (as in, a workaround) for you that you put up the index.yaml on a web server, and the actual charts on S3.

On a second thought, maybe using Chart Museum would be another viable choice, since it supports many backends for storing the charts already.

Finally, I can understand the desire for using IAM roles for authentication & authorization, but I think managing those is a little out of scope for ArgoCD, since they're very vendor-specific. But that's just my opinion, there might be others.

@alexec You know the Helm stuff way better than me, so I'd love to hear your thoughts on this topic.

[theeltahir]

That is an interesting thought, I'll play around with the permissions on the index.yaml on the bucket to see if that is the only hurdle. That would mean that the S3 plug-in is still needed for the helm fetch operation.

Perhaps Chart Museum is a better product, however that would require us to retool around it. The attractive piece about using S3 is we no longer need to manage the basic auth credentials, and relieve the stress on the key rotation and secrets management.

I am not advocating for the Argo CD to have special provisions for IAM roles (any more than it has now), we would leverage the IAM roles granted to the pods via Kube2IAM/KAIM/IRSA.

[theeltahir]

So after updating the permissions on the bucket, looks like using the https URL instead of the S3 URL works, provided the Helm S3 Plugin is installed in our custom image.
https://<chart-repo-bucket>.s3.amazonaws.com/charts/

after adding the chart repo i was able to pull down an existing chart and the UI displays all the values.yaml fields appropriately. I have not tried deploying a release yet but it looks like this is a potential work around.

[jannfis]

So after updating the permissions on the bucket, looks like using the https URL instead of the S3 URL works, provided the Helm S3 Plugin is installed in our custom image.
https://<chart-repo-bucket>.s3.amazonaws.com/charts/

after adding the chart repo i was able to pull down an existing chart and the UI displays all the values.yaml fields appropriately. I have not tried deploying a release yet but it looks like this is a potential work around.

Having a workaround is some good news, I hope the installation of charts works that way as well.

I was thinking a little more about native S3 support meanwhile, and I guess it becomes quite difficult when dealing with fetching dependencies, although I have not looked at the code yet how (or if) we resolve them currently. I'm not an avid user of Helm, so I'm unsure whether helm fetch actually resolves and fetches dependencies. If it just fetches the chart's tarball, my naive assumption would be that we could replace it by a custom fetch method as well. But then we'd have to interpret the index.yaml of the Helm repository as well to get the right URL for the chart to fetch. I don't know for if that's what we want, so I'll wait on @alexec 's judgement for that approach.

As a further workaround, instead of using a custom built argocd image (which may or may not be something you are happy to do), you could also opt for an init container that performs a helm plugin install ... for the s3 plugin, possibly from a private location containing a mirror of the plugin in case your ArgoCD pods do not have access to the Internet (or for other reasons you do not want to modify your pod from Internet resources).

[alexec]

This is correct. Our code does not currently support S3.

Argo CD is un-opinionated on what cloud provider you use, and import an S3 library would break that principle.

Our first preference would be to recommend people use HTTP rather than a custom protocol. Have you read this?

https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.html#access-bucket-intro

[theeltahir]

Yeah I figured that would be the case. It looks like the work around works for us right now, register the S3 bucket via the HTTPS URL and ensure that Helm S3 is present in the Argo CD repo server.

I tried using the init container, ultimately it turned out to be cleaner to build the images locally. Main reason turned out that i would need to build a custom image that contains the plugin, just seemed cleaner to build it directly into the image.

FROM argoproj/argocd:v1.3.0-rc2

USER root
RUN apt-get update && \
    apt-get install -y \
        curl && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
USER argocd
RUN mkdir -p $(helm home)/plugins
ENV HELM_PLUGIN="/home/argocd/.helm/plugins"
ENV AWS_REGION=us-east-1
ARG HELM_S3_VERSION="0.8.0"
ARG S3_PLUGIN_REPO="https://github.com/hypnoglow/helm-s3.git"
RUN helm plugin install ${S3_PLUGIN_REPO} --version ${HELM_S3_VERSION}

[alexec]

See #1105 .

[alexec]

@theelthair it would be amazing if you could document this for other users, just edit this page:

https://github.com/argoproj/argo-cd/edit/master/docs/user-guide/helm.md

[theeltahir]

Of course, I have some bandwidth coming up that I can contribute an example.

[tomjohnburton]

@theeltahir do you have an updated way of doing this with helm3?

[jansony1]

Yeah I figured that would be the case. It looks like the work around works for us right now, register the S3 bucket via the HTTPS URL and ensure that Helm S3 is present in the Argo CD repo server.

I tried using the init container, ultimately it turned out to be cleaner to build the images locally. Main reason turned out that i would need to build a custom image that contains the plugin, just seemed cleaner to build it directly into the image.

FROM argoproj/argocd:v1.3.0-rc2

USER root
RUN apt-get update && \
    apt-get install -y \
        curl && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
USER argocd
RUN mkdir -p $(helm home)/plugins
ENV HELM_PLUGIN="/home/argocd/.helm/plugins"
ENV AWS_REGION=us-east-1
ARG HELM_S3_VERSION="0.8.0"
ARG S3_PLUGIN_REPO="https://github.com/hypnoglow/helm-s3.git"
RUN helm plugin install ${S3_PLUGIN_REPO} --version ${HELM_S3_VERSION}

Hi, after build this customized image, argocd still could not acess s3 hosted helm, with 403 forbidden. Do you how it access iam credentials? I have exec into argo-repo container which do have a iam service account attached with amazon s3 full access policy:

$ aws sts get-caller-identity { "UserId": "AROAT5QZODQXI2HWQPMCV:botocore-session-1594812324", "Account": "269562551342", "Arn": "arn:aws:sts::269562551342:assumed-role/eksctl-test-eks-addon-iamserviceaccount-argo-Role1-1S6RXJRRIXTS7/botocore-session-1594812324" }

Any idea?

[jpke]

Yeah I figured that would be the case. It looks like the work around works for us right now, register the S3 bucket via the HTTPS URL and ensure that Helm S3 is present in the Argo CD repo server.
I tried using the init container, ultimately it turned out to be cleaner to build the images locally. Main reason turned out that i would need to build a custom image that contains the plugin, just seemed cleaner to build it directly into the image.

FROM argoproj/argocd:v1.3.0-rc2

USER root
RUN apt-get update && \
    apt-get install -y \
        curl && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
USER argocd
RUN mkdir -p $(helm home)/plugins
ENV HELM_PLUGIN="/home/argocd/.helm/plugins"
ENV AWS_REGION=us-east-1
ARG HELM_S3_VERSION="0.8.0"
ARG S3_PLUGIN_REPO="https://github.com/hypnoglow/helm-s3.git"
RUN helm plugin install ${S3_PLUGIN_REPO} --version ${HELM_S3_VERSION}

Hi, after build this customized image, argocd still could not acess s3 hosted helm, with 403 forbidden. Do you how it access iam credentials? I have exec into argo-repo container which do have a iam service account attached with amazon s3 full access policy:

$ aws sts get-caller-identity { "UserId": "AROAT5QZODQXI2HWQPMCV:botocore-session-1594812324", "Account": "269562551342", "Arn": "arn:aws:sts::269562551342:assumed-role/eksctl-test-eks-addon-iamserviceaccount-argo-Role1-1S6RXJRRIXTS7/botocore-session-1594812324" }

Any idea?

a few things to look for:
-does your k8s node running argo have an iam policy attached to it's role allowing access to the helm s3 repo bucket? (Possibly relevant - migrating from service-role/AmazonEC2RoleforSSM to AmazonSSMManagedInstanceCore requires adding another policy granting s3 access.)
-is static site hosting enabled for the bucket?
-if the bucket is not publicly accessible, is there a bucket policy in place to whitelist the ip of the node (or the nat gateway if running in a vpc?)

[victorboissiere]

Its not working without making index.html public which is not an option for us.
We will have to work with a custom plugin integration

[victorboissiere]

I've added a proxy (not great) bit at least it worked!

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/component: s3-proxy
    app.kubernetes.io/name: argocd-s3-proxy
    app.kubernetes.io/part-of: argocd
  namespace: argocd
  name: argocd-s3-proxy
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: argocd-s3-proxy
  template:
    metadata:
      labels:
        app.kubernetes.io/name: argocd-s3-proxy
    spec:
      containers:
      - name: s3-proxy
        image: pottava/s3-proxy:2.0
        env:
          - name: AWS_ACCESS_KEY_ID
            valueFrom:
              secretKeyRef:
                name: argocd-aws-user
                key: AWS_ACCESS_KEY_ID
          - name: AWS_SECRET_ACCESS_KEY
            valueFrom:
              secretKeyRef:
                name: argocd-aws-user
                key: AWS_SECRET_ACCESS_KEY
          - name: AWS_REGION
            value: eu-west-3
          - name: AWS_S3_BUCKET
            value: qonto-helm-charts
        ports:
        - containerPort: 80
        resources:
          requests:
            memory: 50Mi
            cpu: 10m
          limits:
            memory: 100Mi
            cpu: 50m
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/component: s3-proxy
    app.kubernetes.io/name: argocd-s3-proxy
    app.kubernetes.io/part-of: argocd
  name: argocd-s3-proxy
  namespace: argocd
spec:
  ports:
  - name: server
    port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app.kubernetes.io/name: argocd-s3-proxy

Now in the UI, I can add the repository with http://argocd-s3-proxy. However, I still get unsupported protocol s3:// while ArgoCD runs helm2 dependency build (because the index.yaml still point to S3).
I dont understand why, because when I sh manually into the container and cd to the tmp directory it works!

argocd@argocd-repo-server-5c5ff5c684-wmlsr$ cd /tmp/[TEMPORARY_GIT_PROJECT
argocd@argocd-repo-server-5c5ff5c684-wmlsr:/tmp/https:XXXX/applications/pdf$ helm init --client-only
argocd@argocd-repo-server-5c5ff5c684-wmlsr:/tmp/https:XXXX/applications/pdf$ helm2 dependency build
Hang tight while we grab the latest from your chart repositories...
...Unable to get an update from the "local" chart repository (http://127.0.0.1:8879/charts):
    Get http://127.0.0.1:8879/charts/index.yaml: dial tcp 127.0.0.1:8879: connect: connection refused
...Successfully got an update from the "stable" chart repository
Update Complete.
Saving 1 charts
Downloading XXX from repo http://XXXX
Deleting outdated charts

EDIT:

Using Helm3 fixed the issue all together. The S3 proxy still apply.