Containers-roadmap: [ECR]: Allow for alternate mediaTypes

[jdolitsky]

Tell us about your request

Hello. I'd like to request that ECR allow for arbitrary mediatypes specified in image manifests (config mediaType and layer mediaType).

Alternatively, AWS can keep a public whitelist of mediaTypes allowed in ECR and engage the community to request support for new types.

Which service(s) is this request for?

ECR

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?

The OCI distribution spec is currently being stretched to support more than just container images (Helm charts, OPA bundles, etc). In order for us to reason about these things, the mediaType field of manifests is being replaced with new custom values.

When running an out-of-the-box instance of docker distribution (2.7+), these new mediaTypes are allowed to be pushed.

However, when pushing these things to ECR, the user is met with a 404 Not Found. Please see https://github.com/deislabs/oras/issues/105 where this issue was first reported.

To give more context, a manifest pushed using docker CLI looks like this:

{
    "schemaVersion": 2,
    "config": {
        "mediaType": "application/vnd.docker.container.image.v1+json", // <-----
        "size": 7023,
        "digest": "sha256:b5b2b2c507a0944348e0303114d8d93aaaa081732b86451d9bce1f432a537bc7"
    },
    "layers": [
...

Whereas a manifest pushed by Helm 3 looks like this:

{
    "schemaVersion": 2,
    "config": {
        "mediaType": "application/vnd.cncf.helm.chart.config.v1+json", // <-----
        "size": 2,
        "digest": "sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a"
    },
    "layers": [
...

It appears that ECR is whitelisting mediaTypes allowed in the registry.

Are you currently working around this issue?

No. The only current solution (while staying on AWS) is to run a custom instance of docker/distribution on an EC2 or in a container.

Additional context

Interest is growing in this concept and we would love to be able to point users to use the registry tied to their cloud provider (AWS) vs. running a custom registry.

Additionally, Azure Container Registry (ACR) allows for this, and maintains a public list of mediaTypes with first-class support.

Attachments

Here are a few links that have some more info on the topic:

• "Cloud Native Artifact Registries evolve from Docker Container Registries" blog post
• ORAS: CLI/Go lib for pushing/pulling alternate artifacts
• bundle.bar: A free service that hosts new types of lightweight, cloud-native artifacts

---

I appreciate your time, thank you!

[jdolitsky]

There is also another report in Helm slack seeing a 403 instead:

[...] status="403 Forbidden" url="https://xxx.dkr.ecr.us-east-2.amazonaws.com/v2/testpush/blobs/sha256:22568708069eb08fd477d4785eeb2c60ac1f09c40e1f9cd674250fbf1c8284f6"
Error: unexpected response: 403 Forbidden
helm.go:56: [debug] unexpected response: 403 Forbidden

[jtoberon]

Thanks for the feedback. We're aware of the OCI distribution spec proposal, and we attend the weekly OCI calls to track its progress. We will continue to participate to help drive the distribution spec in a direction that we believe is useful to our customers, and if it addresses their needs we will be happy to adopt it over time.

[eonicle]

+1. Would really like to see both charts and container images in one place.

[dmytroleonenko]

Almost a year now. Is it so hard to implement?

[apenney]

I'm definitely disappointed I can't publish helm charts into ECR at this point. We're going to have to build and maintain additional infrastructure as a result. I hope that AWS can prioritize this (as it doesn't look like a ton of development and is massively beneficial to people that use helm).

[omieomye]

We're starting to look at this with a view to support, and would love to hear if there use cases for more than Helm charts that should publish to ECR, and what problem that would help solve.

[danbeaulieu]

@omieomye This is a good starting point to see the direction OCI is going: https://github.com/containers/image/pulls?q=is%3Apr+mediaType+

CNAB mediaType: application/vnd.docker.distribution.manifest.v2+json, application/vnd.cnab.config.v1+json

Another good source is containerd: https://github.com/containerd/containerd/blob/master/images/mediatypes.go

The criu/checkpoint/restore ones would be extremely useful.

I've run in to a few efforts of using wasm as extensions (envoy proxy filters), with those bundles being stored in oci registries. Example "application/vnd.wasm.content.layer.v1+wasm" from https://radu-matei.com/blog/wasm-to-oci/

[RichiCoder1]

Shipping Helm & CNAB bundles through the registry, plus future support for WASM packages also are motivating factors for us wanting this too.

[gerred]

Shipping kudo.dev bundles as artifacts is also on our roadmap.

[crsantini]

This addition will be very appreciated. We have clients with several compliance cases where ECR ticks all the boxes and wouldn't require us to look to validate other tools like ChartMuseum

[vchan2002]

We're also looking into using Porter.sh and it would be really great if ECR supports CNAB!

https://porter.sh/compatible-registries/

[omieomye]

We've been working on this, and looking to release OCI artifact support soon.

[mhausenblas]

Alright, following up here on what Bob mentioned in his Container Day keynote earlier today. From now on, you can give this feature a try in ap-southeast-1.

For now, focus on Helm charts using something like the following:

# preparation:
CHART_TAG=helm-ehw-0
REPO_NAME=artifact-test
REPO_URI=$(aws ecr create-repository --repository-name $REPO_NAME | jq -r .repository.repositoryUri)
REPO_CHART=$REPO_URI:$CHART_TAG
REGISTRY=$(echo $REPO_CHART | sed s%\/$REPO_NAME:$CHART_TAG%%)

# enable feature:
export HELM_EXPERIMENTAL_OCI=1

# prep Helm chart locally:
helm create example-hello-world
helm chart save example-hello-world $REPO_CHART

# push to ECR:
aws ecr get-login-password | helm registry login -u AWS --password-stdin $REGISTRY
helm chart push $REPO_CHART

# verify:
$ aws ecr batch-get-image \
           --repository-name $REPO_NAME \
           --image-ids imageTag=$CHART_TAG \
           --query 'images[].imageManifest' --output text
{
  "schemaVersion": 2,
  "config": {
    "mediaType": "application/vnd.cncf.helm.config.v1+json",
    "digest": "sha256:d10882dfb2c97a6a8ff886f42a174a475822b3aa1a6bc48c67e4ddc6183ef2a6",
    "size": 153
  },
  "layers": [
    {
      "mediaType": "application/tar+gzip",
      "digest": "sha256:5f0823edaeed5d40a1dffeb5029454426ab2e4626c39c999ceba9661f035d6ce",
      "size": 3591
    }
  ]
}

We will follow up with more details here and the complete set of supported OCI artifacts in Amazon ECR within a week (give or take), so keep an eye on this issue here to learn more.

[bit-cloner]

Would be great to see artifacts like npm, maven and binaries are also supported.
GCP's artifact registry just started supporting npm and maven
https://cloud.google.com/artifact-registry/docs/supported-formats

[fmedery]

Would be great to see artifacts like npm, maven and binaries are also supported.
GCP's artifact registry just started supporting npm and maven
https://cloud.google.com/artifact-registry/docs/supported-formats

Hello,
AWS CodeArtifact works with commonly used package managers and build tools like Maven and Gradle (Java), npm and yarn (JavaScript), or pip and twine (Python).

[mhausenblas]

We shipped the feature now:

• docs
• launch blog post

[omieomye]

Closing.