Jib: kokoro-macos/continuous builds failing

[TadCordle]

It seems there's an issue on macOS where consecutive builds introduce a slowdown, which causes the test to fail.

[ERROR] testExecute_empty(com.google.cloud.tools.jib.maven.BuildImageMojoIntegrationTest)  Time elapsed: 20.664 s  <<< FAILURE!
java.lang.AssertionError: First build time (8728749621) is not greater than second build time (9143508245)
    at com.google.cloud.tools.jib.maven.BuildImageMojoIntegrationTest.buildAndRun(BuildImageMojoIntegrationTest.java:67)
    at com.google.cloud.tools.jib.maven.BuildImageMojoIntegrationTest.testExecute_empty(BuildImageMojoIntegrationTest.java:129)

[TadCordle]

Now it's failing due to bad certificates.

[ERROR] testExecute_empty(com.google.cloud.tools.jib.maven.BuildImageMojoIntegrationTest)  Time elapsed: 8.916 s  <<< ERROR!
java.lang.RuntimeException: 
Command 'docker pull gcr.io/distroless/java:latest' failed: Error response from daemon: Get https://gcr.io/v2/: x509: certificate has expired or is not yet valid

    at com.google.cloud.tools.jib.maven.BuildImageMojoIntegrationTest.setup(BuildImageMojoIntegrationTest.java:160)

[ERROR] testExecute_complex_sameFromAndToRegistry(com.google.cloud.tools.jib.maven.BuildImageMojoIntegrationTest)  Time elapsed: 5.884 s  <<< ERROR!
java.lang.RuntimeException: 
Command 'docker pull gcr.io/distroless/java:latest' failed: Error response from daemon: Get https://gcr.io/v2/: x509: certificate has expired or is not yet valid

    at com.google.cloud.tools.jib.maven.BuildImageMojoIntegrationTest.setup(BuildImageMojoIntegrationTest.java:160)

[ERROR] testExecute_defaultTarget(com.google.cloud.tools.jib.maven.BuildImageMojoIntegrationTest)  Time elapsed: 5.891 s  <<< ERROR!
java.lang.RuntimeException: 
Command 'docker pull gcr.io/distroless/java:latest' failed: Error response from daemon: Get https://gcr.io/v2/: x509: certificate has expired or is not yet valid

    at com.google.cloud.tools.jib.maven.BuildImageMojoIntegrationTest.setup(BuildImageMojoIntegrationTest.java:160)

[ERROR] testExecute_complex(com.google.cloud.tools.jib.maven.BuildImageMojoIntegrationTest)  Time elapsed: 5.931 s  <<< ERROR!
java.lang.RuntimeException: 
Command 'docker pull gcr.io/distroless/java:latest' failed: Error response from daemon: Get https://gcr.io/v2/: x509: certificate has expired or is not yet valid

    at com.google.cloud.tools.jib.maven.BuildImageMojoIntegrationTest.setup(BuildImageMojoIntegrationTest.java:160)

[ERROR] testExecute_simple(com.google.cloud.tools.jib.maven.BuildImageMojoIntegrationTest)  Time elapsed: 5.877 s  <<< ERROR!
java.lang.RuntimeException: 
Command 'docker pull gcr.io/distroless/java:latest' failed: Error response from daemon: Get https://gcr.io/v2/: x509: certificate has expired or is not yet valid

    at com.google.cloud.tools.jib.maven.BuildImageMojoIntegrationTest.setup(BuildImageMojoIntegrationTest.java:160)

[briandealwis]

What JDK is it using?

[TadCordle]

It's using 1.8.0_152, but that's probably unrelated to the certificate issue considering it's caused by docker pull. I wonder if the java version could be the source of the build time problem, though.

[briandealwis]

Not sure how I missed that it was docker pull failing! But
https://github.com/moby/moby/issues/11534#issuecomment-89405874 suggests that this problem can occur if there is clock drift.

[briandealwis]

And that explain the original build error. (I thought Mac boxes were configured to use NTP by default?)

[coollog]

Btw, I filed an issue with the Kokoro team to get their help with this issue.

[chanseokoh]

From my testing, seems like we can work around the certificate error by following the simple trick documented in the bug (just restarting the Docker daemon). It may take a long time for the Kokoro team to apply and roll out a permanent fix, so let's just restart Docker on our side as a temporary fix. I'll work on this.

[TadCordle]

Can we close this, or do you want to leave it open for the otherwise flaky tests?

[coollog]

Let's leave this open for a few more commits to master and see how flaky the tests still are.

[TadCordle]

If they are still flaky, we do have this issue open https://github.com/GoogleContainerTools/jib/issues/940

[coollog]

Let's see if the first vs. second build time check still fails regularly.

[TadCordle]

Yeah, looks like it's still not fixed. Latest error:

[ERROR] testExecute_simple(com.google.cloud.tools.jib.maven.BuildImageMojoIntegrationTest)  Time elapsed: 34.124 s  <<< FAILURE!
java.lang.AssertionError: First build time (8945.196) is not greater than second build time (9264.685)
    at com.google.cloud.tools.jib.maven.BuildImageMojoIntegrationTest.buildAndRun(BuildImageMojoIntegrationTest.java:90)
    at com.google.cloud.tools.jib.maven.BuildImageMojoIntegrationTest.testExecute_simple(BuildImageMojoIntegrationTest.java:218)

[coollog]

This is happening with much less frequency now. Moving this to a later milestone to revisit if needed.

[coollog]

Closing as this is basically not happening anymore.