Odo: component is left in broken state after redeployment of the pod

Created on 26 Aug 2020 · 7Comments · Source: openshift/odo

/kind bug

What versions of software are you using?

Operating System:
macOS

Output of odo version:
▶ odo version
odo v1.2.5 (552aad7ea)

How did you run odo exactly?

▶ odo create  java-maven
Experimental mode is enabled, use at your own risk

Validation
 ✓  Checking devfile existence [39413ns]
 ✓  Checking devfile compatibility [84734ns]
 ✓  Creating a devfile component from registry: DefaultDevfileRegistry [89449ns]
 ✓  Validating devfile component [247861ns]

Please use `odo push` command to create the component with source deployed

▶ odo push

Validation
 ✓  Validating the devfile [188307ns]

Creating Kubernetes resources for component java-maven
 ✓  Waiting for component to start [2m]

Applying URL changes
 ✓  URL http-8080: http://http-8080-java-maven-tkral-test.apps.testocp4x.psiodo.net/ created

Syncing to component java-maven
 ✓  Checking files for pushing [5ms]
 ✓  Syncing files to the component [25s]

Executing devfile commands for component java-maven
 ✓  Executing mvn-package command "mvn package", if not running [1m]
 ✓  Executing run command "java -jar target/*.jar", if not running [2s]

Pushing devfile component java-maven
 ✓  Changes successfully pushed to component

▶ curl http://http-8080-java-maven-tkral-test.apps.testocp4x.psiodo.net/
Greetings from Spring Boot!%

▶ odo storage create mystorage --path=/data --size=1Gi
 ✓  Added storage mystorage to java-maven

Please use `odo push` command to make the storage accessible to the component

▶ odo push

Validation
 ✓  Validating the devfile [124117ns]

Creating Kubernetes resources for component java-maven
 ✓  Waiting for component to start [4m]

Applying URL changes
 ✓  URLs are synced with the cluster, no changes are required.

Syncing to component java-maven
 ✓  Checking file changes for pushing [3ms]
 ✓  Syncing files to the component [2s]

Executing devfile commands for component java-maven
 ✗  Executing mvn-package command "mvn package", if not running [3s]
 ✗  Failed to start component with name java-maven. Error: Failed to create the component: command execution failed: unable to execute the run command: unable to exec command [/bin/sh -c mvn package]:
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  0.089 s
[INFO] Finished at: 2020-08-26T09:42:09Z
[INFO] ------------------------------------------------------------------------
[ERROR] The goal you specified requires a project to execute but there is no POM in this directory (/projects). Please verify you invoked Maven from the correct directory. -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MissingProjectException
: error while streaming command: command terminated with exit code 1

Expected behavior

source code should be present in container and my application should be running after odo push

Any logs, error output, etc?

build command while running odo push fails because the directory where the application source is supposed to be is empty.

▶ oc exec java-maven-748556d67-kmk7x  -- ls /projects
?
devfile.yaml

kinbug prioritHigh

Source

kadel

Most helpful comment

I'm hitting this issue too, without using odo storage.

It seems syncing is entirely broken whenever a component is restarted on a push (with or without the -f flag). If the component restarts, it only ever syncs the delta, based on what's in odo-file-index.json.

It seems odo is no longer determining when it needs to bypass the odo-file-index.json file (such as when a component is redeployed). I can also confirm that odo v1.2.5 works fine.

I think the reason why this is happening is because we're no longer resetting the index file on a force push:
https://github.com/openshift/odo/commit/4a67c9adaf5dc349134f61f81f910b5b0f15e336#diff-474c230eecd0c5084d941b6d880143b1L111

In my opinion, this should be high priority given how visible it is, and it's a regression from v1.2.5 functionality

/priority high

johnmcollier on 27 Aug 2020

👍2

All 7 comments

The problem is in the sync code. After the storage is mounted, the pod is restarted and source code files are lost. Any command which causes the pod to restart will break the component.

mik-dass on 26 Aug 2020

is it related with this https://github.com/openshift/odo/pull/3409??

adisky on 26 Aug 2020

is it related with this #3409??

I don't think so, that bug was causing the redeployment. Here the issue is about the syncing after a redeployment not being handled by the sync adapter.

mik-dass on 26 Aug 2020

I'm hitting this issue too, without using odo storage.

It seems odo is no longer determining when it needs to bypass the odo-file-index.json file (such as when a component is redeployed). I can also confirm that odo v1.2.5 works fine.

In my opinion, this should be high priority given how visible it is, and it's a regression from v1.2.5 functionality

/priority high

johnmcollier on 27 Aug 2020

👍2

Confirmed this is due to https://github.com/openshift/odo/pull/3619 ... removing the index-removal code was intentional, but it looks like it regresses the 'sync after new pod' scenario as seen on this issue.

In light of this issue, the correct behaviour for this logic should be:

If force push is specified, and a new pod depoyed:

Delete the index, so everything is re-transferred from local, to the new pod

If force push is specified, and existing pod reused:

Don't delete the index, because we need to use the existing index in order to find files that were deleted/renamed since the last push.
Incorrectly handling this causes the bug originally reported here: https://github.com/openshift/odo/issues/3460

If force push is not specified, and new pod deployed:

Delete the index, so everything is re-transferred from local, to the new pod

If force push is not specified, and existing pod is reused:

Don't delete the index: the existing state in the remote pod should be how we last left it, and the index file should be a reliable representation of that.

New PR to do those things, plus an integration test to verify the behaviour: https://github.com/openshift/odo/pull/3859