Aws-cdk: [aws-eks] Support S3 private bucket repository from helm deployment

Created on 27 Jul 2020 · 6Comments · Source: aws/aws-cdk

Use Case

Right now helm deployment only support public github repository. We need to read the repository from s3 private repo

Proposed Solution

I would like to help to develop the feature but would like to hear your opinion on the approach

Create a s3 presign url and use the presign url to deploy the chart
new HelmChart(this, 'cortexHelm', { cluster: props.cluster, chart: props.presignUrl, namespace: props.environment.cluster.namespace, values: overrides, release: 'cortex' });
install S3 plugin at lambda layer: https://github.com/aws-samples/aws-lambda-layer-kubectl
Use the S3 plugin to read from S3 private repo while deploy helm chart:
Plugin link: https://github.com/hypnoglow/helm-s3

Other

[x] :wave: I may be able to implement this feature request
[ ] :warning: This feature might incur a breaking change

This is a :rocket: Feature Request

@aws-cdaws-eks efformedium feature-request p2

Source

qinxx108

👍2

Most helpful comment

Filling in some details here:
Let's assume you are a CDK user which has some privately developed Helm chart which you want to deploy via eks.HelmChart construct. For simplicity you use CDK already to upload your Helm chart to S3.

You need a way to have the eks.HelmChart construct download and use these private Helm charts. Today this doesn't work because the custom cloudformation resource which executes helm doesn't have permissions to access this object on S3, and as a CDK user there is no way to attach the right permissions because the handler role is private.

We can make this work by introducing a helper method on eks.HelmChart, fromAsset, which grants the kubectl handler role the permissions to access the object on S3. Because permissions are created as part of the eks.HelmChart.fromAsset method we have to ensure these permissions are narrow.

In code, this would look something like this:

new eks.HelmChart(this, 'private-helm-chart', {
  chart: eks.HelmChart.fromAsset('path/to/asset.tgz')
});

Under the hood, fromAsset will create a CDK Asset and attach s3:GetObject and kms:Decrypt to the kubectl handler role. We need to change the helm handler in python slightly, like this:

     # "log in" to the cluster
    subprocess.check_call([ 'aws', 'eks', 'update-kubeconfig',
        '--role-arn', role_arn,
        '--name', cluster_name,
        '--kubeconfig', kubeconfig
    ])

    # download helm chart if hosted on s3
    if (chart is not None) and (chart.startswith('s3://')):
        bucket_name_key = chart.split('//')[1]
        bucket_name = bucket_name_key.split('/')[0]
        object_key = '/'.join(bucket_name_key.split('/')[1:])
        local_path = '/'.join(('', 'tmp', os.path.basename(object_key)))
        s3 = boto3.client('s3')
        with open(local_path, 'wb') as f:
            s3.download_fileobj(bucket_name, object_key, f)
        chart = local_path

It's important to use the CDK Asset construct here. First we want the helm chart file to be uploaded automatically when cdk deploy is executed. Next the uploaded Helm chart must be immutable, or we risk breaking deployments/ rollbacks.
Assets help here because they generate names, containing the hash of the source file.

I think it's a relatively small change which feels idiomatic as CDK has other classes which also provide fromAsset methods.
When downloading the file inside the existing code we also don't need custom lambda layers to begin with, and we don't generate a pre-signed url either.

What are your thoughts here, @eladb ? Will this provide a good user experience while ensuring that CDK internally generates narrow & secure IAM policies?

nicolai86 on 21 Aug 2020

❤2 👍2 🚀1

All 6 comments

Hello, i want to submit a PR for this implementation but i would like to hear your opinion about which one that your team prefer. Please take a look at the description

Thanks!
yijie

qinxx108 on 7 Aug 2020

@qinxx108 Could you please give some more detail on the first approach of using presign URL's? In general, an approach that doesn't require a custom lambda layer is definitely preferable - but i'd like to understand exactly what it entails and what will be the flow from a user perspective

iliapolo on 18 Aug 2020

@qinxx108 and me chatted today about a potential approach. @qinxx108 will update this issue with details.

eladb on 19 Aug 2020

👍1

In code, this would look something like this:

new eks.HelmChart(this, 'private-helm-chart', {
  chart: eks.HelmChart.fromAsset('path/to/asset.tgz')
});

Under the hood, fromAsset will create a CDK Asset and attach s3:GetObject and kms:Decrypt to the kubectl handler role. We need to change the helm handler in python slightly, like this:

     # "log in" to the cluster
    subprocess.check_call([ 'aws', 'eks', 'update-kubeconfig',
        '--role-arn', role_arn,
        '--name', cluster_name,
        '--kubeconfig', kubeconfig
    ])

    # download helm chart if hosted on s3
    if (chart is not None) and (chart.startswith('s3://')):
        bucket_name_key = chart.split('//')[1]
        bucket_name = bucket_name_key.split('/')[0]
        object_key = '/'.join(bucket_name_key.split('/')[1:])
        local_path = '/'.join(('', 'tmp', os.path.basename(object_key)))
        s3 = boto3.client('s3')
        with open(local_path, 'wb') as f:
            s3.download_fileobj(bucket_name, object_key, f)
        chart = local_path

What are your thoughts here, @eladb ? Will this provide a good user experience while ensuring that CDK internally generates narrow & secure IAM policies?

nicolai86 on 21 Aug 2020

❤2 👍2 🚀1

What @nicolai86 described is exactly what I was looking for. It would be really cool to be able to define a helm chart and Docker image via CDK assets and deploy them into an existing cluster.