odo push gets succeded even if Devfile build command fails
/kind bug
What versions of software are you using?
Operating System:
Output of odo version:
How did you run odo exactly?
$ odo create java-spring-boot
..........
$ odo push -f
• Push devfile component java-spring-boot ...
Applying URL changes
✓ URL java-spring-boot-8080 already exists
✓ Waiting for component to start [1ms]
✓ Syncing files to the component [135ms]
✓ Executing devbuild command "/artifacts/bin/build-container-full.sh" [1s]
✓ Executing devrun command "/artifacts/bin/start-server.sh" [132ms]
✓ Push devfile component java-spring-boot [1s]
✓ Changes successfully pushed to component
Actual behavior
build command failed inside the container, still odo shows it as successfull
1000540000@java-spring-boot-648c9ffd6b-l7hjm:/artifacts/bin$ ./build-container-full.sh
Wed Apr 15 12:05:41 UTC 2020
Started - Full build using container folders
listing project src
total 8
drwxr-xr-x. 2 root root 60 Apr 9 22:38 .
drwxr-xr-x. 3 root root 17 Apr 9 22:38 ..
-rwxr-xr-x. 1 root root 527 Apr 9 22:38 build-container-full.sh
-rwxr-xr-x. 1 root root 593 Apr 9 21:12 start-server.sh
Wed Apr 15 12:05:41 UTC 2020
running full maven build
POM file ./pom.xml specified with the -f/--file command line argument does not exist
[INFO] Scanning for projects...
[ERROR] [ERROR] Some problems were encountered while processing the POMs:
[FATAL] Non-readable POM /artifacts/bin/./pom.xml: /artifacts/bin/./pom.xml (No such file or directory) @
@
[ERROR] The build could not read 1 project -> [Help 1]
[ERROR]
[ERROR] The project (/artifacts/bin/./pom.xml) has 1 error
[ERROR] Non-readable POM /artifacts/bin/./pom.xml: /artifacts/bin/./pom.xml (No such file or directory)
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
Expected behavior
odo push, should show some info about failed devbuild, devrun command.
Any logs, error output, etc?
adisky
All 9 comments
@adisky To run the script when inside the container, you need to run it from the directory of the project source in the container, basically cd /projects/springbootproject && /artifacts/bin/build-container-full.sh
That being said, you did find a bug in the devfile exec code. It seems that's it not catching build failures.
To test this, I caused maven to fail in two ways:
1) Syntax error in a .java file -> maven failed, but not detected by supervisord or odo
2) OOM error in maven -> maven failed, but also not detected by supervisord or odo
Additionally, it seems like failures in the devRun commands may not be getting detected either
Johns-MacBook-Pro-3:springboot johncollier$ echo 'sdfsdfsdfdsfdsf' > src/main/java/application/rest/v1/test.java
Johns-MacBook-Pro-3:springboot johncollier$ odo push
• Push devfile component java-spring-boot ...
✓ Waiting for component to start [22ms]
✓ Checking file changes for pushing [7ms]
✓ Syncing files to the component [1s]
✓ Executing devbuild command "/artifacts/bin/build-container-full.sh" [15s]
✓ Executing devrun command "/artifacts/bin/start-server.sh" [2s]
✓ Push devfile component java-spring-boot [20s]
✓ Changes successfully pushed to component
Setting priority high for this, since odo needs to be able to catch build and run failures
/priority high
/area devfile
CC @maysunfaisal
johnmcollier
on 17 Apr 2020
@adisky To run the script when inside the container, you need to run it from the directory of the project source in the container, basically
cd /projects/springbootproject && /artifacts/bin/build-container-full.shThat being said, you did find a bug in the devfile exec code. It seems that's it not catching build failures.
To test this, I caused maven to fail in two ways:
- Syntax error in a .java file -> maven failed, but not detected by supervisord or odo
- OOM error in maven -> maven failed, but also not detected by supervisord or odo
Additionally, it seems like failures in the devRun commands may not be getting detected either
Johns-MacBook-Pro-3:springboot johncollier$ echo 'sdfsdfsdfdsfdsf' > src/main/java/application/rest/v1/test.java Johns-MacBook-Pro-3:springboot johncollier$ odo push • Push devfile component java-spring-boot ... ✓ Waiting for component to start [22ms] ✓ Checking file changes for pushing [7ms] ✓ Syncing files to the component [1s] ✓ Executing devbuild command "/artifacts/bin/build-container-full.sh" [15s] ✓ Executing devrun command "/artifacts/bin/start-server.sh" [2s] ✓ Push devfile component java-spring-boot [20s] ✓ Changes successfully pushed to component
i got why build is failing inside the container, just wanted to point it is not detected by odo.
adisky
on 17 Apr 2020
It looks like this may be an issue with the spring boot stack? Its build and run scripts aren't detecting failures and aren't exiting properly, so the odo exec code can't catch that.
If I force a failure in the script (like an exit 1 at the end of the script), odo catches that properly:
Johns-MacBook-Pro-3:springboot johncollier$ odo push
• Push devfile component java-spring-boot ...
✓ Waiting for component to start [20ms]
Applying URL changes
✓ URL java-spring-boot-8080 already exists
✓ Checking file changes for pushing [1ms]
✓ Syncing files to the component [756ms]
◐ Executing devbuild command "/artifacts/bin/build-container-full.sh" ✗
Unable to exec command [/bin/sh -c cd /projects/springbootproject && /artifacts/bin/build-container-full.sh]:
Fri Apr 17 14:29:39 UTC 2020
Started - Full build using container folders
listing project src
total 40
drwxr-xr-x 4 root root 4096 Apr 17 06:04 .
drwxrwxrwx 3 root root 4096 Apr 17 06:04 ..
-rw-r--r-- 1 501 staff 213 Apr 8 16:12 .cw-settings
-rw-r--r-- 1 501 staff 65 Apr 8 16:16 .gitignore
drwxr-xr-x 3 root root 4096 Apr 17 06:04 .odo
-rw-r--r-- 1 501 staff 97 Apr 8 16:12 Jenkinsfile
-rw-r--r-- 1 501 staff 1814 Apr 8 16:12 README.md
-rw-r--r-- 1 501 staff 1213 Apr 17 06:04 devfile.yaml
-rw-r--r-- 1 501 staff 3011 Apr 8 16:12 pom.xml
drwxr-xr-x 4 root root 4096 Apr 17 06:04 src
Fri Apr 17 14:29:39 UTC 2020
running full maven build
[INFO] Scanning for projects...
[INFO]
[INFO] -------------------------< projects:sprtest1 >--------------------------
[INFO] Building sprtest1 1.0-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
Downloading from spring-milestones: https://repo.spring.io/milestone/org/jboss/shrinkwrap/resolver/shrinkwrap-resolver-bom/2.2.0/shrinkwrap-resolver-bom-2.2.0.pom
Killed
ls: cannot access 'target/*.jar': No such file or directory
Fri Apr 17 14:29:52 UTC 2020
copying target jar to output dir
cp: missing destination file operand after '/data/output/'
Try 'cp --help' for more information.
Fri Apr 17 14:29:52 UTC 2020
listing /data/output
total 8
drwxr-sr-x 2 root root 4096 Apr 17 06:05 .
drwxrwsrwx 4 root root 4096 Apr 17 06:05 ..
Fri Apr 17 14:29:52 UTC 2020
Finished - Full build using container folders
✗ Executing devbuild command "/artifacts/bin/build-container-full.sh" [14s]
✗ Failed to start component with name java-spring-boot.
Error: Failed to create the component: error while streaming command: command terminated with exit code 1
Removing the high priority since it's restricted to just springboot stack and it's an easy fix according to Maysun
johnmcollier
on 17 Apr 2020
@johnmcollier: Those labels are not set on the issue: priority/high
In response to this:
It looks like this may be an issue with the spring boot stack? Its build and run scripts aren't detecting failures and aren't exiting properly, so the odo exec code can't catch that.
If I force a failure in the script (like an
exit 1at the end of the script), odo catches that properly:Johns-MacBook-Pro-3:springboot johncollier$ odo push • Push devfile component java-spring-boot ... ✓ Waiting for component to start [20ms] Applying URL changes ✓ URL java-spring-boot-8080 already exists ✓ Checking file changes for pushing [1ms] ✓ Syncing files to the component [756ms] ◐ Executing devbuild command "/artifacts/bin/build-container-full.sh" ✗ Unable to exec command [/bin/sh -c cd /projects/springbootproject && /artifacts/bin/build-container-full.sh]: Fri Apr 17 14:29:39 UTC 2020 Started - Full build using container folders listing project src total 40 drwxr-xr-x 4 root root 4096 Apr 17 06:04 . drwxrwxrwx 3 root root 4096 Apr 17 06:04 .. -rw-r--r-- 1 501 staff 213 Apr 8 16:12 .cw-settings -rw-r--r-- 1 501 staff 65 Apr 8 16:16 .gitignore drwxr-xr-x 3 root root 4096 Apr 17 06:04 .odo -rw-r--r-- 1 501 staff 97 Apr 8 16:12 Jenkinsfile -rw-r--r-- 1 501 staff 1814 Apr 8 16:12 README.md -rw-r--r-- 1 501 staff 1213 Apr 17 06:04 devfile.yaml -rw-r--r-- 1 501 staff 3011 Apr 8 16:12 pom.xml drwxr-xr-x 4 root root 4096 Apr 17 06:04 src Fri Apr 17 14:29:39 UTC 2020 running full maven build [INFO] Scanning for projects... [INFO] [INFO] -------------------------< projects:sprtest1 >-------------------------- [INFO] Building sprtest1 1.0-SNAPSHOT [INFO] --------------------------------[ jar ]--------------------------------- Downloading from spring-milestones: https://repo.spring.io/milestone/org/jboss/shrinkwrap/resolver/shrinkwrap-resolver-bom/2.2.0/shrinkwrap-resolver-bom-2.2.0.pom Killed ls: cannot access 'target/*.jar': No such file or directory Fri Apr 17 14:29:52 UTC 2020 copying target jar to output dir cp: missing destination file operand after '/data/output/' Try 'cp --help' for more information. Fri Apr 17 14:29:52 UTC 2020 listing /data/output total 8 drwxr-sr-x 2 root root 4096 Apr 17 06:05 . drwxrwsrwx 4 root root 4096 Apr 17 06:05 .. Fri Apr 17 14:29:52 UTC 2020 Finished - Full build using container folders ✗ Executing devbuild command "/artifacts/bin/build-container-full.sh" [14s] ✗ Failed to start component with name java-spring-boot. Error: Failed to create the component: error while streaming command: command terminated with exit code 1Removing the high priority since it's restricted to just springboot stack and it's an easy fix according to Maysun
/remove-priority high
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
openshift-ci-robot
on 17 Apr 2020
Unassign since this one is related to the spring stack which the runtime team will be providing.
elsony
on 28 Jul 2020
I agree with @elsony, this is not an odo issue but rather an issue with the specified devfile which doesn't properly report an error code when running its commands. That should be addressed at the devfile level, not at the odo one. /cc @kadel @girishramnani
metacosm
on 7 Aug 2020
There is a PR for this which might resolve this https://github.com/openshift/odo/pull/3799
girishramnani
on 24 Aug 2020
We are hitting build failure with successful push on periodic jobs for java-springboot.
Downloaded from central: https://repo.maven.apache.org/maven2/junit/junit/3.8.2/junit-3.8.2.jar (121 kB at 266 kB/s)
[odo] Downloaded from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-classworlds/2.2.2/plexus-classworlds-2.2.2.jar (46 kB at 100 kB/s)
[odo] [INFO] ------------------------------------------------------------------------
[odo] [INFO] BUILD FAILURE
[odo] [INFO] ------------------------------------------------------------------------
[odo] [INFO] Total time: 20.060 s
[odo] [INFO] Finished at: 2020-09-07T02:15:09Z
[odo] [INFO] ------------------------------------------------------------------------
[odo] [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project sprtest1: Execution default-compile of goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile failed: Plugin org.apache.maven.plugins:maven-compiler-plugin:3.1 or one of its dependencies could not be resolved: The following artifacts could not be resolved: log4j:log4j:jar:1.2.12, com.google.collections:google-collections:jar:1.0: Could not transfer artifact log4j:log4j:jar:1.2.12 from/to central (https://repo.maven.apache.org/maven2): Transfer failed for https://repo.maven.apache.org/maven2/log4j/log4j/1.2.12/log4j-1.2.12.jar: Connection reset -> [Help 1]
[odo] [ERROR]
[odo] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[odo] [ERROR] Re-run Maven using the -X switch to enable full debug logging.
[odo] [ERROR]
[odo] [ERROR] For more information about the errors and possible solutions, please read the following articles:
[odo] [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginResolutionException
[odo] ls: cannot access 'target/*.jar': No such file or directory
[odo] Mon Sep 7 02:15:09 UTC 2020
[odo] copying target jar to output dir
[odo] Mon Sep 7 02:15:09 UTC 2020
[odo] listing /data/output
[odo] cp: missing destination file operand after '/data/output/'
[odo] Try 'cp --help' for more information.
[odo] total 8
[odo] drwxr-sr-x. 2 1004130000 1004130000 4096 Sep 7 02:15 .
[odo] drwxrwsr-x. 5 root 1004130000 4096 Sep 7 02:15 ..
[odo] Mon Sep 7 02:15:09 UTC 2020
[odo] Finished - Full build using container folders
[odo]
✓ Executing defaultbuild command "/artifacts/bin/build-container-full.sh" [23s]
[odo] I0907 02:15:09.507770 13445 exec.go:27] Executing command [/opt/odo/bin/supervisord -c /opt/odo/conf/devfile-supervisor.conf -d] for pod: rrjcmh-7798b598d7-vthzk in container: runtime
[odo] • Executing defaultrun command "/artifacts/bin/start-server.sh" ...
[odo] I0907 02:15:09.810194 13445 exec.go:27] Executing command [/opt/odo/bin/supervisord ctl start devrun] for pod: rrjcmh-7798b598d7-vthzk in container: runtime
[odo] devrun: started
[odo]
✓ Executing defaultrun command "/artifacts/bin/start-server.sh" [1s]
[odo]
[odo] Pushing devfile component rrjcmh
[odo] ✓ Changes successfully pushed to component
More details - https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-odo-master-v4.3-integration-e2e-periodic/1302758652611923968#1:build-log.txt%3A1941
prietyc123
on 7 Sep 2020
with the new springboot devfile it is fixed
$ odo push -f
Validation
✓ Validating the devfile [26984ns]
Creating Kubernetes resources for component java-springboot
✓ Waiting for component to start [1ms]
Applying URL changes
✓ URLs are synced with the cluster, no changes are required.
Syncing to component java-springboot
✓ Checking file changes for pushing [507196ns]
✓ Syncing files to the component [117ms]
Executing devfile commands for component java-springboot
✗ Executing build command "mvn -Dmaven.repo.local=/home/user/.m2/repository package -Dmaven.test.skip=true" [1s]
✗ Failed to start component with name java-springboot. Error: Failed to create the component: command execution failed: unable to execute the run command: unable to exec command [/bin/sh -c mvn -Dmaven.repo.local=/home/user/.m2/repository package -Dmaven.test.skip=true]:
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 0.059 s
[INFO] Finished at: 2020-09-14T09:17:41Z
[INFO] ------------------------------------------------------------------------
[ERROR] The goal you specified requires a project to execute but there is no POM in this directory (/projects). Please verify you invoked Maven from the correct directory. -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MissingProjectException
: error while streaming command: command terminated with exit code 1
Related issues
amitkrout
·
3Comments
prietyc123
·
3Comments
valaparthvi
·
3Comments
surajnarwade
·
5Comments
anandrkskd
·
5Comments
Most helpful comment
i got why build is failing inside the container, just wanted to point it is not detected by odo.