Gitpod: Using jsii/superchain as the base image causes workspace creation to fail

[skinny85]

Hello,

I'm trying to setup GitPods (awesome product, by the way!) for the AWS CDK project. For that, I'm using our standard Docker image jsii/superchain - source code here.

However, when using the following .gitpod.yml file in my fork of the CDK repository, the workspace fails to build with a rather non-informative error:

sha256:3b0cf1bf33b08034faf88235a1160b2c5a5f61d920923021aa49755ce4ad9b2e: Pulling from jsii/superchain
a3f8e652bdc4: Pulling fs layer
b6715c9d2005: Pulling fs layer
9b891725542b: Pulling fs layer
37ba41153458: Pulling fs layer
19e6f9fe2d74: Pulling fs layer
eb1536d59fd4: Pulling fs layer
7d2f5ba1db46: Pulling fs layer
36070bae538c: Pulling fs layer
e12f2b5805d1: Pulling fs layer
7ca7a90d8803: Pulling fs layer
c49231461ab1: Pulling fs layer
c97059dcf6c5: Pulling fs layer
cb7ff8f194da: Pulling fs layer
54d63734acaa: Pulling fs layer
5ff01f363a52: Pulling fs layer
2fac4c8c4545: Pulling fs layer
bbc80785142d: Pulling fs layer
194703366131: Pulling fs layer
37ba41153458: Waiting
19e6f9fe2d74: Waiting
c49231461ab1: Waiting
c97059dcf6c5: Waiting
eb1536d59fd4: Waiting
7d2f5ba1db46: Waiting
36070bae538c: Waiting
e12f2b5805d1: Waiting
7ca7a90d8803: Waiting
cb7ff8f194da: Waiting
54d63734acaa: Waiting
5ff01f363a52: Waiting
2fac4c8c4545: Waiting
bbc80785142d: Waiting
194703366131: Waiting
9b891725542b: Verifying Checksum
9b891725542b: Download complete
a3f8e652bdc4: Verifying Checksum
a3f8e652bdc4: Download complete
b6715c9d2005: Verifying Checksum
b6715c9d2005: Download complete
19e6f9fe2d74: Verifying Checksum
19e6f9fe2d74: Download complete
eb1536d59fd4: Verifying Checksum
eb1536d59fd4: Download complete
36070bae538c: Verifying Checksum
36070bae538c: Download complete
7d2f5ba1db46: Verifying Checksum
7d2f5ba1db46: Download complete
e12f2b5805d1: Verifying Checksum
e12f2b5805d1: Download complete
c49231461ab1: Verifying Checksum
c49231461ab1: Download complete
37ba41153458: Download complete
7ca7a90d8803: Download complete
c97059dcf6c5: Download complete
54d63734acaa: Verifying Checksum
54d63734acaa: Download complete
cb7ff8f194da: Verifying Checksum
cb7ff8f194da: Download complete
5ff01f363a52: Verifying Checksum
5ff01f363a52: Download complete
2fac4c8c4545: Download complete
194703366131: Verifying Checksum
194703366131: Download complete
bbc80785142d: Verifying Checksum
bbc80785142d: Download complete
a3f8e652bdc4: Pull complete
b6715c9d2005: Pull complete
9b891725542b: Pull complete
37ba41153458: Pull complete
19e6f9fe2d74: Pull complete
eb1536d59fd4: Pull complete
7d2f5ba1db46: Pull complete
36070bae538c: Pull complete
e12f2b5805d1: Pull complete
7ca7a90d8803: Pull complete
c49231461ab1: Pull complete
c97059dcf6c5: Pull complete
cb7ff8f194da: Pull complete
54d63734acaa: Pull complete
5ff01f363a52: Pull complete
2fac4c8c4545: Pull complete
bbc80785142d: Pull complete
194703366131: Pull complete
Digest: sha256:3b0cf1bf33b08034faf88235a1160b2c5a5f61d920923021aa49755ce4ad9b2e
Status: Downloaded newer image for jsii/superchain@sha256:3b0cf1bf33b08034faf88235a1160b2c5a5f61d920923021aa49755ce4ad9b2e
adding Gitpod layer
..
  build failed: cannot build workspace image: container exited with non-zero exit code: 1.
                                                                                          Error: build failed: cannot buildworkspace image: container exited with non-zero exit code: 1

I've shared the workspace, its ID is fb8da305-df77-447d-9fdd-8d1c1ab99818 .

Thanks in advance for your help!

Adam

[Kreyren]

You are tripping a security layer let me see if i can contribute a hotfix untill https://github.com/gitpod-io/workspace-images/pull/204 is processed

[Kreyren]

Created https://github.com/jankeromnes/gitpod-layers/pull/4 waiting for review.

[skinny85]

Hi @Kreyren ,

thanks for the response! I see the PR you linked to has been moved to https://github.com/Kreyren/workspace-images/pull/1 , which is still a draft, and I don't have access to https://github.com/jankeromnes/gitpod-layers/pull/4 .

Can you confirm 2 things for me?

1. This is an issue with GitPods, and not something with the way we build the jsii/superchain image (or something we can fix by some configuration, in the .gitpods.yml file for example).
2. Can you give a rough ETA on when those issues you linked to will be resolved, and we'll be able to use jsii/superchain with GitPods?

Thanks,
Adam

[Kreyren]

thanks for the response! I see the PR you linked to has been moved to Kreyren/workspace-images#1 ,which is still a draft, @skinny85

That's different MR (very old version before it was moved to private), the MR in question is kept in private repository since it's rewriting workspace-images to accept security layers and i don't have permission to make it public yet.

This is an issue with GitPods, and not something with the way we build the jsii/superchain image (or something we can fix by some configuration, in the .gitpods.yml file for example). @skinny85

I've diagnosed it as gitpod security layer issue and since this layer is kept as confidential i can't provide more informations about it, because i don't want to be sued. :p

Can you give a rough ETA on when those issues you linked to will be resolved, and we'll be able to use jsii/superchain with GitPods?

I keep being blamed for making a huge contributions that are hard to process so i make this one to be very small so @jankeromnes should be able to process it fast without being too annoyed that i am taking his time off ^-^ I leave the ETA to him since from my side i believe it to be processed.

disclaimer: i am contributor not gitpod staff

To speed the process up i recommend printing this picture on a printer and praying to it and/or setting it as desktop wallpaper.

Example usage being:

[Kreyren]

@skinny85 status update: my merge request has been submitted for review in gitpod's production, ETA currently unknown

[Kreyren]

@skinny85 status update: the merge request works in private staging, currently waiting for deployment

EDIT: Better wording

[skinny85]

Thanks for the updates @Kreyren ! Any ETA on the production deployment? (I just tried it again, same error)

[jankeromnes]

Hey @skinny85,

I'm trying to setup GitPods (awesome product, by the way!) for the AWS CDK project.

That's awesome! We'd love to support the AWS CDK project in Gitpod.

Also, many thanks to @Kreyren for taking this on! You've identified and fixed the problem extremely fast.

Can you confirm 2 things for me?

1. This is an issue with GitPods, and not something with the way we build the jsii/superchain image (or something we can fix by some configuration, in the .gitpods.yml file for example).

Yes, your configuration looks good, and this is indeed an issue with Gitpod:

• Until now, Gitpod only supported Docker images based on Ubuntu/Debian or on Alpine
• jsii/superchain is based on Amazon Linux, for which @Kreyren has now implemented support in Gitpod

As shown in @Kreyren's screenshot, your fork now works in a private staging deployment of Gitpod without any further changes.

1. Can you give a rough ETA on when those issues you linked to will be resolved, and we'll be able to use jsii/superchain with GitPods?

@Kreyren's Amazon Linux PR works well and was approved, so barring any last-minute problem, it looks like it can be included in tomorrow's (Wednesday) regular production deployment of gitpod.io.

This would enable any project based on jsii/superchain image to work in Gitpod.

If for some reason the PR can't be deployed tomorrow, the next regular production deployment of gitpod.io happens on Wednesday in two weeks. However, if this is something time-critical for you, we could consider an interim hotfix deployment to get jsii/superchain support into production as soon as possible.

We'll keep you posted on tomorrow's deployment.

[skinny85]

That's fantastic! Thank you so much for the detailed update @jankeromnes.

[Kreyren]

@jankeromnes justify closing? afaik it's not in production yet

[jankeromnes]

Ah sorry, I didn't mean to close this yet. PR is merged and should be included in tomorrow's regular deployment (ETA early morning CEST).

[Kreyren]

@jankeromnes status update?

[jankeromnes]

@Kreyren Sorry, we needed more time for testing other things, gitpod.io production deployment was moved to tomorrow Thursday 14 May 05:30 UTC.

[Kreyren]

@jankeromnes Noted, ping me if there are any issues with my MR.

[jankeromnes]

@skinny85 Update: The fix was successfully deployed this morning, and your fork's Gitpod setup now seems to work:

https://gitpod.io/#https://github.com/skinny85/aws-cdk/tree/feat/gitpods

Furthermore, all Amazon Linux based images (like jsii/superchain) should now also be supported by Gitpod.

Thanks again @Kreyren for fixing this so fast! 🙏

[Kreyren]

@skinny85 let me know if something doesn't work as expected or could be done better.

(Btw. I am currently working on a contribution hoping to make the relevant source code open-source)

[skinny85]

@Kreyren @jankeromnes I just tried it, and it works perfectly. Thank you so much for fixing this so quickly, you guys rock! ❤️