Cloud-builders: mvn for App Engine deployment

[iamacarpet]

We can't get the mvn builder deploying to App Engine Standard.

steps:
- name: 'gcr.io/cloud-builders/mvn'
  args: ['appengine:deploy','-Dpromote=False','-Dversion=test']

      <!-- [START cloudplugin] -->
      <plugin>
        <groupId>com.google.cloud.tools</groupId>
        <artifactId>appengine-maven-plugin</artifactId>
        <version>1.3.1</version>
      </plugin>
      <!-- [END cloudplugin] -->

Gives us the error message:

[ERROR] Failed to execute goal com.google.cloud.tools:appengine-maven-plugin:1.3.1:deploy (default-cli) on project pdf-api: Execution default-cli of goal com.google.cloud.tools:appengine-maven-plugin:1.3.1:deploy failed: The Google Cloud SDK could not be found in the customary locations and no path was provided. -> [Help 1]

Is there a workaround and if not, can you support it by including the "gcloud" tools in this image?

Regards,
iamacarpet

[akerekes]

Check out the documentation for how to configure the plugin to find your gcloud installation:
https://github.com/GoogleCloudPlatform/app-maven-plugin/blob/master/USER_GUIDE_v1.md#cloud-sdk-configuration

If you don't have gcloud installed, follow the instructions here: https://cloud.google.com/sdk/install

[imjasonh]

The mvn builder isn't FROM gcr.io/cloud-builders/gcloud so gcloud isn't installed in the image. If we decide that it should be we can update its FROM, but that will increase the image's size by ~1GB.

[ludoch]

There is no Maven artifact for the Cloud SDK, as opposed to the original GAE Standard Java SDK which
could be taken from maven using this maven dep:
<plugin> <groupId>com.google.appengine</groupId> <artifactId>appengine-maven-plugin</artifactId> <version>1.9.64</version> </plugin>

• some logging step the first time or some flags setup for oauth...
  and then appengine:update.

Now, should we create a new gcloud-mvn image containing mvn and the Cloud SDK? Curious about why it would be 1GB...

[iamacarpet]

We tried the com.google.appengine appengine-maven-plugin plugin with appengine:update and it didn't work out of the box.

That being said, none of us are Java developers (we just have an App Engine service cobbled together in Java, as it's the only language in which we could find a required library), so the Maven tooling is a little alien compared to the gloud app deploy we are used to in other languages.

The error was about no auth being present, so can the old SDK be made to use the service account credentials provided by Cloud Build?

[bendory]

@ludoch gcr.io/cloud-builders/gcloud is ~1G in size; gcr.io/cloud-builders/gcloud-slim is ~500M, and gcr.io/cloud-builders/mvn is ~400M.

The gcloud images are based on launcher.gcr.io/google/ubuntu16_04 which is ~50M, so adding everything needed for gcloud-slim/gcloud adds ~450M/950M respectively. There is no doubt some overlap with what's in the mvn image (which is based on gcr.io/cloud-builders/javac --> openjdk), but definitely expect at least an additional few 100M to be added to the image size.

[denyska]

@iamacarpet as quick fix you can try to combine both maven and gcloud, i.e.

steps:
- id: 'Build fat jar'
  name: 'gcr.io/cloud-builders/mvn'
  args: ['package']
- id: "Prepare staging directory"
  name: 'ubuntu'
  args: ['mkdir', 'deploy-build']
- name: 'ubuntu'
  args: ['cp', 'src/main/appengine/app.yaml', 'deploy-build/']
- name: 'ubuntu'
  args: ['cp', 'target/demo-0.0.1-SNAPSHOT.jar', 'deploy-build/']
- id: "Deploy to AppEngine"
  name: 'gcr.io/cloud-builders/gcloud'
  args: ['app', 'deploy', 'deploy-build/']
timeout: '1600s'

[hienle-hps]

Based on the chain:
cloud-builders/javac -> cloud-builders/gradle

was hoping it'd be possible to tweak gradle/Dockerfile:

• change BASE_IMAGE to cloud-builders/gcloud
• add apt-get install statements for javac

but cloud-builders/javac doesn't seem to actually install javac and gets it from the BASE_IMAGE! @aslo , why does cloud-builders/javac just install Docker?

[hienle-hps]

Cobbled together an image gcloud-gradle that seems to invoke Gradle AppEngine tasks. The process should be similar to hack up a gcloud-mvn image but this doesn't produce the cleanly layered images seen in the other builders.

The images didn't seem chain-able since cloud-builders/gcloud is Ubuntu-based and cloud-builders/gradle is Debian-based. Decided to use cloud-builders/javac as a base image to avoid duplicating all the complexity from openjdk-runtime. Manually combined cloud-builders/gcloud-slim, cloud-builders/gcloud, and cloud-builders/gradle into new Dockerfile:

• removed (Ubuntu) PPA statements
• replaced (or removed) Ubuntu packages with Debian equivalents

[daninge98]

With the new version of the maven app engine plugin (2.0.0+), the plugin will automatically install the gcloud tools if they aren't available. This will allow you to stage the app for deployment from the maven cloud build image but not actually deploy it (as the gcloud tools won't be initialized with your project credentials).

You can combine the maven image with the gcloud image to perform a full build and deploy as shown below:

steps:
- id: 'Stage app using mvn appengine plugin on mvn cloud build image'
  name: 'gcr.io/cloud-builders/mvn'
  args: ['package', 'appengine:stage']
- id: "Deploy to app engine using gcloud image"
  name: 'gcr.io/cloud-builders/gcloud'
  args: ['app', 'deploy', 'target/appengine-staging/app.yaml']
timeout: '1600s'

@denyska 's hack didn't work for me as the fat jar produced was greater than the 40MB app engine limit, and so the deploy would fail. This method gets around that (but requires maven app engine plugin v2.0.0+)

[ghost]

@iamacarpet as quick fix you can try to combine both maven and gcloud, i.e.

steps:
- id: 'Build fat jar'
  name: 'gcr.io/cloud-builders/mvn'
  args: ['package']
- id: "Prepare staging directory"
  name: 'ubuntu'
  args: ['mkdir', 'deploy-build']
- name: 'ubuntu'
  args: ['cp', 'src/main/appengine/app.yaml', 'deploy-build/']
- name: 'ubuntu'
  args: ['cp', 'target/demo-0.0.1-SNAPSHOT.jar', 'deploy-build/']
- id: "Deploy to AppEngine"
  name: 'gcr.io/cloud-builders/gcloud'
  args: ['app', 'deploy', 'deploy-build/']
timeout: '1600s'

Your contribution is very good! The first time I tried with .jar was not successful, but when I switched to .war everything was better.

[NicolaSpreafico]

Hi,
I'm having the same problem, deploying a Maven project with Cloud Build.

This is my current configuration:

steps:

- id: 'Stage app using mvn appengine plugin on mvn cloud build image'
  name: 'gcr.io/cloud-builders/mvn'
  args: [ 'appengine:stage', '-Pdevelopment']
  dir: 'project'

- id: "Deploy to app engine using gcloud image"
  name: 'gcr.io/cloud-builders/gcloud'
  args: ["app", "deploy", "project/target/appengine-staging/app.yaml"]

timeout: "1600s"

And here is the log

Step #0 - "Stage app using mvn appengine plugin on mvn cloud build image": [INFO] Staging the application to: /workspace/project/target/development/appengine-staging
Step #0 - "Stage app using mvn appengine plugin on mvn cloud build image": [INFO] Detected App Engine standard environment application.
Step #0 - "Stage app using mvn appengine plugin on mvn cloud build image": [INFO] ------------------------------------------------------------------------
Step #0 - "Stage app using mvn appengine plugin on mvn cloud build image": [INFO] BUILD FAILURE
Step #0 - "Stage app using mvn appengine plugin on mvn cloud build image": [INFO] ------------------------------------------------------------------------
Step #0 - "Stage app using mvn appengine plugin on mvn cloud build image": [INFO] Total time:  01:31 min
Step #0 - "Stage app using mvn appengine plugin on mvn cloud build image": [INFO] Finished at: 2019-05-10T16:01:09Z
Step #0 - "Stage app using mvn appengine plugin on mvn cloud build image": [INFO] ------------------------------------------------------------------------
Step #0 - "Stage app using mvn appengine plugin on mvn cloud build image": [ERROR] Failed to execute goal com.google.cloud.tools:appengine-maven-plugin:1.3.2:stage (default-cli) on project XXXXXX: Execution default-cli of goal com.google.cloud.tools:appengine-maven-plugin:1.3.2:stage failed: The Google Cloud SDK could not be found in the customary locations and no path was provided. -> [Help 1]
Step #0 - "Stage app using mvn appengine plugin on mvn cloud build image": [ERROR] 
Step #0 - "Stage app using mvn appengine plugin on mvn cloud build image": [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
Step #0 - "Stage app using mvn appengine plugin on mvn cloud build image": [ERROR] Re-run Maven using the -X switch to enable full debug logging.
Step #0 - "Stage app using mvn appengine plugin on mvn cloud build image": [ERROR] 
Step #0 - "Stage app using mvn appengine plugin on mvn cloud build image": [ERROR] For more information about the errors and possible solutions, please read the following articles:
Step #0 - "Stage app using mvn appengine plugin on mvn cloud build image": [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException

I haven't found a solution yet

EDIT: Solved using plugin version 2.0.0

[pierluigi-montagna]

I'm facing the same issue although the error I'm receiving is different, it seems to be related to the missing python installation.

This impacts also the command appengine:stage making impossible to deploy with cloud build at the moment.

Step #3: Welcome to the Google Cloud SDK!
Step #3: /root/.cache/google-cloud-tools-java/managed-cloud-sdk/LATEST/google-cloud-sdk/install.sh: 209: /root/.cache/google-cloud-tools-java/managed-cloud-sdk/LATEST/google-cloud-sdk/install.sh: python: not found
Step #3: [INFO] ------------------------------------------------------------------------
Step #3: [INFO] BUILD FAILURE
Step #3: [INFO] ------------------------------------------------------------------------
Step #3: [INFO] Total time:  27.552 s
Step #3: WARNING: You appear to be running this script as root. This may cause 
Step #3: the installation to be inaccessible to users other than the root user.
Step #3: [INFO] Finished at: 2020-03-02T09:58:03Z
Step #3: [INFO] ------------------------------------------------------------------------
Step #3: [ERROR] Failed to execute goal com.google.cloud.tools:appengine-maven-plugin:2.2.0:stage (default-cli) on project testp: Execution default-cli of goal com.google.cloud.tools:appengine-maven-plugin:2.2.0:stage failed: com.google.cloud.tools.managedcloudsdk.command.CommandExitException: Process failed with exit code: 127 -> [Help 1]
Step #3: org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal com.google.cloud.tools:appengine-maven-plugin:2.2.0:stage (default-cli) on project testp: Execution default-cli of goal com.google.cloud.tools:appengine-maven-plugin:2.2.0:stage failed: com.google.cloud.tools.managedcloudsdk.command.CommandExitException: Process failed with exit code: 127
Step #3:     at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:215)
Step #3:     at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
Step #3:     at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
Step #3:     at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
Step #3:     at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
Step #3:     at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
Step #3:     at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
Step #3:     at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
Step #3:     at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
Step #3:     at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
Step #3:     at org.apache.maven.cli.MavenCli.execute (MavenCli.java:957)
Step #3:     at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:289)
Step #3:     at org.apache.maven.cli.MavenCli.main (MavenCli.java:193)
Step #3:     at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
Step #3:     at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
Step #3:     at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
Step #3:     at java.lang.reflect.Method.invoke (Method.java:566)
Step #3:     at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
Step #3:     at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
Step #3:     at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
Step #3:     at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347) 

[loosebazooka]

@pierluigi-montagna can you share your cloudbuild.yaml? I wonder if we need to change our recommendations based on what's available on these base images.

[bendory]

Per https://github.com/GoogleCloudPlatform/cloud-builders/issues/665, the maven cloud-builder is being deprecated and turned down.

I suggest migrating further discussion of this topic to the appengine-maven repo at https://github.com/GoogleCloudPlatform/appengine-maven-plugin