Cloud-builders: gcr.io/cloud-builders/mvn:latest missing python - sdk install fails

[tpolekhin]

Affected builder image

gcr.io/cloud-builders/mvn

Expected Behavior

mvn appengine:deploy command success

Actual Behavior

Step #1 - "AppEngine Deploy": Sep 09, 2020 2:54:57 PM com.google.cloud.tools.managedcloudsdk.install.Downloader download
Step #1 - "AppEngine Deploy": INFO: Downloading https://dl.google.com/dl/cloudsdk/channels/rapid/google-cloud-sdk.tar.gz to /root/.cache/google-cloud-tools-java/managed-cloud-sdk/downloads/google-cloud-sdk.tar.gz
Step #1 - "AppEngine Deploy": Welcome to the Google Cloud SDK!
Step #1 - "AppEngine Deploy": WARNING: You appear to be running this script as root. This may cause 
Step #1 - "AppEngine Deploy": the installation to be inaccessible to users other than the root user.
Step #1 - "AppEngine Deploy": /root/.cache/google-cloud-tools-java/managed-cloud-sdk/LATEST/google-cloud-sdk/install.sh: line 180: python: command not found
Step #1 - "AppEngine Deploy": [INFO] ------------------------------------------------------------------------
Step #1 - "AppEngine Deploy": [INFO] BUILD FAILURE
Step #1 - "AppEngine Deploy": [INFO] ------------------------------------------------------------------------
Step #1 - "AppEngine Deploy": [INFO] Total time:  17.552 s
Step #1 - "AppEngine Deploy": [INFO] Finished at: 2020-09-09T14:55:01Z
Step #1 - "AppEngine Deploy": [INFO] ------------------------------------------------------------------------
Step #1 - "AppEngine Deploy": [ERROR] Failed to execute goal com.google.cloud.tools:appengine-maven-plugin:2.2.0:deploy (default-cli) on project REDACTED: Execution default-cli of goal com.google.cloud.tools:appengine-maven-plugin:2.2.0:deploy failed: com.google.cloud.tools.managedcloudsdk.command.CommandExitException: Process failed with exit code: 127 -> [Help 1]

Steps to Reproduce the Problem

steps:
  - id: "AppEngine Deploy"
    name: "gcr.io/cloud-builders/mvn"
    entrypoint: "bash"
    args:
      - -c
      - mvn appengine:deploy -P cloud-gcp

Additional Info

Worked as of yesterday Sep 8 2020, today isn't working anymore Sep 9 2020

Found a local image that has python installed.
Tested cloudbuild with pinned image sha256 and now it works

  - id: "AppEngine Deploy"
    name: "gcr.io/cloud-builders/mvn@sha256:87877ba911c20b9cf622142e990739ecf6c25f3b380970e672d19913238b57a4"
    entrypoint: "bash"
    args:
      - -c
      - mvn appengine:deploy -P cloud-gcp

Step #1 - "AppEngine Deploy": Sep 09, 2020 3:56:13 PM com.google.cloud.tools.managedcloudsdk.install.Downloader download
Step #1 - "AppEngine Deploy": INFO: Downloading https://dl.google.com/dl/cloudsdk/channels/rapid/google-cloud-sdk.tar.gz to /root/.cache/google-cloud-tools-java/managed-cloud-sdk/downloads/google-cloud-sdk.tar.gz
Step #1 - "AppEngine Deploy": Welcome to the Google Cloud SDK!
Step #1 - "AppEngine Deploy": WARNING: You appear to be running this script as root. This may cause 
Step #1 - "AppEngine Deploy": the installation to be inaccessible to users other than the root user.
Step #1 - "AppEngine Deploy": 
Step #1 - "AppEngine Deploy": 
Step #1 - "AppEngine Deploy": Your current Cloud SDK version is: 309.0.0
Step #1 - "AppEngine Deploy": Installing components from version: 309.0.0
Step #1 - "AppEngine Deploy": 
Step #1 - "AppEngine Deploy": +-----------------------------------------------------------------------------+
Step #1 - "AppEngine Deploy": |                     These components will be installed.                     |
Step #1 - "AppEngine Deploy": +-----------------------------------------------------+------------+----------+
Step #1 - "AppEngine Deploy": |                         Name                        |  Version   |   Size   |
Step #1 - "AppEngine Deploy": +-----------------------------------------------------+------------+----------+
Step #1 - "AppEngine Deploy": | BigQuery Command Line Tool                          |     2.0.60 |  < 1 MiB |
Step #1 - "AppEngine Deploy": | BigQuery Command Line Tool (Platform Specific)      |     2.0.58 |  < 1 MiB |
Step #1 - "AppEngine Deploy": | Cloud SDK Core Libraries (Platform Specific)        | 2020.07.10 |  < 1 MiB |
Step #1 - "AppEngine Deploy": | Cloud Storage Command Line Tool                     |       4.53 |  3.5 MiB |
Step #1 - "AppEngine Deploy": | Cloud Storage Command Line Tool (Platform Specific) |       4.51 |  < 1 MiB |
Step #1 - "AppEngine Deploy": | Default set of gcloud commands                      |            |          |
Step #1 - "AppEngine Deploy": | anthoscli                                           |      0.2.7 | 51.7 MiB |
Step #1 - "AppEngine Deploy": | anthoscli                                           |      0.2.7 |          |
Step #1 - "AppEngine Deploy": | gcloud cli dependencies                             | 2020.06.12 |  < 1 MiB |
Step #1 - "AppEngine Deploy": +-----------------------------------------------------+------------+----------+
Step #1 - "AppEngine Deploy": 
Step #1 - "AppEngine Deploy": For the latest full release notes, please visit:
Step #1 - "AppEngine Deploy":   https://cloud.google.com/sdk/release_notes
Step #1 - "AppEngine Deploy": 
Step #1 - "AppEngine Deploy": #============================================================#
Step #1 - "AppEngine Deploy": #= Creating update staging area                             =#
Step #1 - "AppEngine Deploy": #============================================================#
Step #1 - "AppEngine Deploy": #= Installing: BigQuery Command Line Tool                   =#
Step #1 - "AppEngine Deploy": #============================================================#
Step #1 - "AppEngine Deploy": #= Installing: BigQuery Command Line Tool (Platform Spec... =#
Step #1 - "AppEngine Deploy": #============================================================#
Step #1 - "AppEngine Deploy": #= Installing: Cloud SDK Core Libraries (Platform Specific) =#
Step #1 - "AppEngine Deploy": #============================================================#
Step #1 - "AppEngine Deploy": #= Installing: Cloud Storage Command Line Tool              =#
Step #1 - "AppEngine Deploy": #============================================================#
Step #1 - "AppEngine Deploy": #= Installing: Cloud Storage Command Line Tool (Platform... =#
Step #1 - "AppEngine Deploy": #============================================================#
Step #1 - "AppEngine Deploy": #= Installing: Default set of gcloud commands               =#
Step #1 - "AppEngine Deploy": #============================================================#
Step #1 - "AppEngine Deploy": #= Installing: anthoscli                                    =#
Step #1 - "AppEngine Deploy": #============================================================#
Step #1 - "AppEngine Deploy": #= Installing: anthoscli                                    =#
Step #1 - "AppEngine Deploy": #============================================================#
Step #1 - "AppEngine Deploy": #= Installing: gcloud cli dependencies                      =#
Step #1 - "AppEngine Deploy": #============================================================#
Step #1 - "AppEngine Deploy": #= Creating backup and activating new installation          =#
Step #1 - "AppEngine Deploy": #============================================================#
Step #1 - "AppEngine Deploy": 
Step #1 - "AppEngine Deploy": Performing post processing steps...
Step #1 - "AppEngine Deploy": ...........................................................done.

[bendory]

It looks like the older version of the image that you referenced by sha (gcr.io/cloud-builders/mvn@sha256:87877ba911c20b9cf622142e990739ecf6c25f3b380970e672d19913238b57a4) has an installation of EOL obsolete Python 2.7.5.

Per https://github.com/carlossg/docker-maven/issues/164, Python 2.x went EOL as of January 1, 2020; it looks like it was removed from maven's upstream base image.

Our gcr.io/cloud-builders/mvn image is based on maven:latest; assuming I am walking upstream correctly, that is based on https://github.com/docker-library/openjdk/blob/master/14/jdk/oraclelinux8/Dockerfile which in turn is based on oraclelinux:8-slim https://github.com/oracle/container-images/blob/ebe69d3cb52596f308ab2a07e667b2532c8b297d/8-slim/Dockerfile.

It looks like what all that means is that python 2.x support is dropped and that using python will thus require installing a python 3 library first.

Ugh, this is not a great story to tell. I'm going to poke around a bit in search of someone with domain expertise who can help with this; in the interim, I suggest that you continue to pull by sha to keep your builds working.

[tpolekhin]

@bendory thank you for a great issue breakdown!
I agree that this is a weird situation, when it's no-one's fault or responsibility to include python in the image, but I think this is a common use case to use this image to build and deploy java application on App Engine, so I really hope this could be resolved somehow.
Thanks again and looking forward to any updates!

[nourabbas96]

Any update guys I have the same error

[bendory]

Update: the Google App Engine team are having discussions regarding how to best address this issue; note that it is significantly "upstream" from Google (the above-noted removal of python from the base JDK image), and the problem is not specific to Cloud Build or the gcr.io/cloud-builders/mvn image.

In general, a user ought to be able to build for App Engine using the standard maven image in Dockerhub, and that is currently broken (which is the next stop upstream in diagnosing the issue as reported here). The App Engine team is discussing addressing that larger issue, not this narrower issue of the GCB image.

In the interim, I recommend using the workaround in @tpolekhin's original issue report of specifying image by sha.

[miztaka]

+1

[xansrnitu]

+1

[tharris-terratrue]

+1

[pierluigi-montagna]

+1

[bendory]

I apologize that this is taking time; sometimes big companies move slowly. :-(

In the interim, to ease your work-around, I have tagged the last image with python support as gcr.io/cloud-builders/mvn:python -- that will be easier to use as a work-around rather than pulling by sha. (It's the identical image as previously identified in this issue.)

[PJozeph]

+1

[PJozeph]

I am not able to upload the image to App Engine with gcr.io/cloud-builders/mvn:python, I have time out response

[bendory]

I am not able to upload the image to App Engine with gcr.io/cloud-builders/mvn:python, I have time out response

@PJozeph that sounds like a separate problem from this issue; IDK how you set timeouts in maven or with the App Engine plug-in, but I suggest exploring that path. If that doesn't prove fruitful, please contact your Cloud Support specialist or use the App Engine Slack channel for help.

[bendory]

Update: we are looking at what would be required to put together a GCB + App Engine + cloud SDK compatible image for use as a Cloud Builder as a resolution to this issue. I'm sorry that I can't provide a commitment or timeframe at this time.

[mohan-ganesh]

+1
gcr.io/cloud-builders/mvn:python did work.

[loosebazooka]

so actually if we're going to push a new image for appengine specifically I would suggest we have a Dockerfile that looks like:

ARG MAVEN_VERSION=latest
FROM maven:${MAVEN_VERSION}

# install python
RUN microdnf install python3

# install cloud sdk into appengine-maven-plugin cache
RUN  set -eux; \
  mkdir -p ~/.cache/google-cloud-tools-java/managed-cloud-sdk/LATEST; \
  cd ~/.cache/google-cloud-tools-java/managed-cloud-sdk/LATEST; \
  curl https://dl.google.com/dl/cloudsdk/channels/rapid/google-cloud-sdk.tar.gz --output google-cloud-sdk.tar.gz; \
  tar -xzvf google-cloud-sdk.tar.gz; \
  ./google-cloud-sdk/bin/gcloud components install app-engine-java; \
  rm google-cloud-sdk.tar.gz; \
  cd ~

COPY notice.sh /usr/bin
ENTRYPOINT ["/usr/bin/notice.sh"]

Not only will this install python, but will pre-install the cloud sdk (with appengine-java components) in the default appengine-maven-plugin caching location. This should reduce the build cost of download the cloud sdk on each build.

not sure if that final cd ~ is necessary.

[loosebazooka]

Another note here, it seems like only latest is affected. This latest uses maven 3.6.3 and jdk 14, but the bigger change is that only that base image oraclelinux:8-slim doesn't contain python or the apt package manager (hence the use of microdnf to install python3)

[bendory]

Update: https://github.com/GoogleCloudPlatform/cloud-builders/commit/7e9600b97e476c3f213c5e56de7fc6cdb92d7759 may address this; we are testing it out now.

If you would like to try out this experimental image, it is available at gcr.io/cloud-builders/mvn:appengine.

[bendory]

Update: merged PR #723 provides a fix that will be released going forward as gcr.io/cloud-builders/mvn:appengine. A pre-production image is currently available with that tag.