odo push trigger redeploy even if there are no changes in devfile

Created on 2 Mar 2020  ·  16Comments  ·  Source: openshift/odo

/kind bug
/area devfile
/priority medium

If there are no changes to Devfile.yaml that would resolve in changes in Deploment the Deployment should not be triggered.

Current behaviour

export ODO_EXPERIMENTAL=true
# running push for the first time
▶ odo push ; kubectl get pods
 •  Push devfile component odo-examples  ...
 ✓  Waiting for component to start [9s]
 ✓  Push devfile component odo-examples [10s]
NAME                           READY   STATUS    RESTARTS   AGE
odo-examples-8cb546c7b-sbmxd   1/1     Running   0          11s


# just triggering push again without any changes

▶ odo push ; kubectl get pods
 •  Push devfile component odo-examples  ...
 ✓  Waiting for component to start [174ms]
 ✓  Push devfile component odo-examples [1s]
NAME                           READY   STATUS        RESTARTS   AGE
odo-examples-8cb546c7b-sbmxd   1/1     Terminating   0          71s
aredevfile kinbug lifecyclrotten prioritMedium triagneeds-information
Source
picture kadel

Most helpful comment

@yangcao77 That's correct, I was able to see the order of the volumes in the pod spec change sometimes and that causes the redeploy to occur. Another way to check that it's happening is to run kubectl get rs -w to watch for any ReplicaSet changes. If a redeploy occurs you will see new events during the push.

odoUpdateProblem

@cdrage @kadel Since I was the only one who was able to reproduce it, I investigated a bit further and found the problem and have a fix I'll submit a PR for. The problem was related to iterating over maps in a couple places and iteration order of maps is not guaranteed so I changed it to use splices of structs instead. Here are the changes for anyone that's interested: https://github.com/maysunfaisal/odo-fork/compare/2654-1...rajivnathan:2654-1?expand=1

With this change I can no longer reproduce. Since this is a relatively minor problem I'll put up a PR after https://github.com/openshift/odo/pull/2735 is merged.

picture rajivnathan on 3 Apr 2020
❤4 👍3

All 16 comments

@cdrage I was wondering why I couldn't reproduce it but others could. We figured out that it has to do with ordering of things in the deployment spec when there is more than a single item in an array. In my case I had a devfile with only a single volume but others were reproducing the problem with devfiles that had two volumes.

For example, I cannot reproduce this problem with this devfile:

apiVersion: 1.0.0
metadata:
  generateName: openLiberty
attributes:
  odo.autorestart: "false"
projects:
- name: openLiberty
  source:
    type: git
    location: https://github.com/rajivnathan/openLiberty.git
components:
- type: chePlugin
  id: redhat/java/latest
  memoryLimit: 1512Mi
- alias: devruntime
  mountSources: true
  type: dockerimage
  image: jeevandroid/open-liberty-dev:latest
  command: ['tail']
  args: ['-f', '/dev/null']
  memoryLimit: 768Mi
  env:
  - name: test15
    value: test4
  volumes:
  - name: m2
    containerPath: /home/user/.m2
  endpoints:
  - name: 9090/tcp
    port: 9090
commands:
- actions:
  - command: /artifacts/bin/build.sh
    component: devruntime
    type: exec
    workdir: /projects/openLiberty
  name: DevBuild
- actions:
  - command: /artifacts/bin/run.sh
    component: devruntime
    type: exec
    workdir: /projects/openLiberty
  name: DevRun
- actions:
  - command: /artifacts/bin/debug.sh
    component: devruntime
    type: exec
    workdir: /projects/openLiberty
  name: DevDebug

But if you just add another volume to the devruntime component you'll most likely see the problem. It doesn't seem to happen if you have multiple environment variables so I'm wondering if it's something to do with how we're parsing the volumes specifically.

rajivnathan picture rajivnathan on 18 Mar 2020

@rajivnathan I can't seem to replicate it even with an extra volume with one of the components..

See my below devfile.yaml:

---
apiVersion: 1.0.0
metadata:
  generateName: openLiberty
projects:
  -
    name: openLiberty
    source:
      type: git
      location: "https://github.com/rajivnathan/openLiberty.git"
components:
  -
    type: chePlugin
    id: redhat/java/latest
    memoryLimit: 1512Mi
  -
    type: dockerimage
    image: jeevandroid/open-liberty-dev:latest
    alias: runtime
    memoryLimit: 768Mi
    command: ['tail']
    args: [ '-f', '/dev/null']
    env:
      - name: "MODE"
        value: "TEST"
      - name: "myprop"
        value: "myval"
    endpoints:
      - name: '9090/tcp'
        port: 9090
    mountSources: true
    volumes:
      - name: m2
        containerPath: /home/user/.m2
      - name: m3
        containerPath: /home/user/.m3
  -
    type: dockerimage
    image: jeevandroid/open-liberty-dev:latest
    alias: build
    memoryLimit: 768Mi
    command: ['tail']
    args: [ '-f', '/dev/null']
    env:
      - name: "MODE2"
        value: "TEST2"
      - name: "myprop2"
        value: "myval2"
    endpoints:
      - name: '9191/tcp'
        port: 9191
    mountSources: true
    volumes:
      - name: m2
        containerPath: /home/user/.m2
commands:
  -
    name: Build
    actions:
      -
        type: exec
        component: runtime
        command: "TEST_ENV=true /artifacts/bin/build.sh"
        workdir: /projects/openLiberty
  -
    name: Run
    actions:
      -
        type: exec
        component: runtime
        command: "/artifacts/bin/run.sh"
        workdir: /projects/openLiberty

And when I run it:

▶ odo push
 •  Push devfile component devfile  ...
 ✓  Checking files for pushing [241560ns]
 ✓  Waiting for component to start [56s]
 ✓  Syncing files to the component [1s]
 ✓  Push devfile component devfile [58s]
 ✓  Changes successfully pushed to component

~/openshift/devfile                                                                                                                                                                                                                                                                                                                                                         
▶ odo push
 •  Push devfile component devfile  ...
 ✓  Checking file changes for pushing [171035ns]
 ✓  No file changes detected, skipping build. Use the '-f' flag to force the build.
 ✓  Push devfile component devfile [526ms]
 ✓  Changes successfully pushed to component

This is against the latest master.

cdrage picture cdrage on 24 Mar 2020

@cdrage I just tried with the latest from master and I can still reproduce it. It's easy to miss and you may not be able to reproduce it consistently but here's how you can verify you're hitting it:

  1. Run push with more verbose logs: odo push -v 4
  2. If you see the following output then you're hitting the problem:
    I0325 13:00:56.132616 95345 deployments.go:81] Waiting for deployment spec update to be observed...

I didn't see it the first 2-3 times so you may need to try it a few times.

Logs without the problem:

I0325 12:58:17.631253   94341 preference.go:165] The path for preference file is /Users/rajiv/.odo/preference.yaml
I0325 12:58:17.631469   94341 preference.go:165] The path for preference file is /Users/rajiv/.odo/preference.yaml
I0325 12:58:17.631576   94341 preference.go:165] The path for preference file is /Users/rajiv/.odo/preference.yaml
I0325 12:58:17.631701   94341 context.go:48] absolute devfile path: '/Users/rajiv/git/myprojects/openLibertyDevfile/devfile.yaml'
I0325 12:58:17.632087   94341 content.go:32] converted devfile YAML to JSON
I0325 12:58:17.632137   94341 apiVersion.go:35] devfile apiVersion: '1.0.0'
I0325 12:58:17.632145   94341 context.go:64] devfile apiVersion '1.0.0' is supported in odo
I0325 12:58:17.633243   94341 schema.go:47] validated devfile schema
 •  Push devfile component openlibertydevfile  ...
I0325 12:58:18.185417   94341 utils.go:17] Found component dockerimage with alias devruntime
I0325 12:58:18.185506   94341 utils.go:17] Found component dockerimage with alias devruntime
I0325 12:58:18.185524   94341 adapter.go:159] Generating PVC name for m2
I0325 12:58:18.185553   94341 utils.go:98] Checking PVC for volume m2 and label component=openlibertydevfile,storage-name=m2
I0325 12:58:18.322705   94341 utils.go:105] Found an existing PVC for volume m2 and label component=openlibertydevfile,storage-name=m2
I0325 12:58:18.322734   94341 adapter.go:171] Found an existing PVC for m2, PVC m2-openlibertydevfile-rbrt will be re-used
I0325 12:58:18.322745   94341 adapter.go:159] Generating PVC name for m3
I0325 12:58:18.322777   94341 utils.go:98] Checking PVC for volume m3 and label component=openlibertydevfile,storage-name=m3
I0325 12:58:18.455312   94341 utils.go:105] Found an existing PVC for volume m3 and label component=openlibertydevfile,storage-name=m3
I0325 12:58:18.455343   94341 adapter.go:171] Found an existing PVC for m3, PVC m3-openlibertydevfile-naud will be re-used
I0325 12:58:18.455354   94341 adapter.go:159] Generating PVC name for m4
I0325 12:58:18.455388   94341 utils.go:98] Checking PVC for volume m4 and label component=openlibertydevfile,storage-name=m4
I0325 12:58:18.591481   94341 utils.go:105] Found an existing PVC for volume m4 and label component=openlibertydevfile,storage-name=m4
I0325 12:58:18.591516   94341 adapter.go:171] Found an existing PVC for m4, PVC m4-openlibertydevfile-oqes will be re-used
I0325 12:58:18.591577   94341 adapter.go:201] Creating deployment openlibertydevfile
I0325 12:58:18.591590   94341 adapter.go:202] The component name is openlibertydevfile
I0325 12:58:18.720470   94341 adapter.go:205] The component already exists, attempting to update it
I0325 12:58:18.863233   94341 adapter.go:210] Successfully updated component openlibertydevfile
I0325 12:58:19.133733   94341 adapter.go:224] Successfully update Service for component openlibertydevfile
I0325 12:58:19.133777   94341 utils.go:98] Checking PVC for volume m2 and label component=openlibertydevfile,storage-name=m2
I0325 12:58:19.263397   94341 utils.go:105] Found an existing PVC for volume m2 and label component=openlibertydevfile,storage-name=m2
I0325 12:58:19.263437   94341 utils.go:98] Checking PVC for volume m3 and label component=openlibertydevfile,storage-name=m3
I0325 12:58:19.397770   94341 utils.go:105] Found an existing PVC for volume m3 and label component=openlibertydevfile,storage-name=m3
I0325 12:58:19.397793   94341 utils.go:98] Checking PVC for volume m4 and label component=openlibertydevfile,storage-name=m4
I0325 12:58:19.529344   94341 utils.go:105] Found an existing PVC for volume m4 and label component=openlibertydevfile,storage-name=m4
I0325 12:58:19.532510   94341 file_indexer.go:204] .odo or .git directory detected, skipping it
I0325 12:58:19.533981   94341 file_indexer.go:204] .odo or .git directory detected, skipping it
 ✓  Checking file changes for pushing [8ms]
I0325 12:58:19.538045   94341 adapter.go:99] List of files to be deleted: +[]
 ✓  No file changes detected, skipping build. Use the '-f' flag to force the build.
 ✓  Push devfile component openlibertydevfile [2s]
 ✓  Changes successfully pushed to component
I0325 12:58:19.538096   94341 odo.go:80] Could not get the latest release information in time. Never mind, exiting gracefully :)

Logs with the problem:

I0325 12:59:39.806384   95003 preference.go:165] The path for preference file is /Users/rajiv/.odo/preference.yaml
I0325 12:59:39.806603   95003 preference.go:165] The path for preference file is /Users/rajiv/.odo/preference.yaml
I0325 12:59:39.806731   95003 preference.go:165] The path for preference file is /Users/rajiv/.odo/preference.yaml
I0325 12:59:39.806904   95003 context.go:48] absolute devfile path: '/Users/rajiv/git/myprojects/openLibertyDevfile/devfile.yaml'
I0325 12:59:39.807837   95003 content.go:32] converted devfile YAML to JSON
I0325 12:59:39.807901   95003 apiVersion.go:35] devfile apiVersion: '1.0.0'
I0325 12:59:39.807910   95003 context.go:64] devfile apiVersion '1.0.0' is supported in odo
I0325 12:59:39.810278   95003 schema.go:47] validated devfile schema
 •  Push devfile component openlibertydevfile  ...
I0325 12:59:40.444710   95003 utils.go:17] Found component dockerimage with alias devruntime
I0325 12:59:40.444794   95003 utils.go:17] Found component dockerimage with alias devruntime
I0325 12:59:40.444810   95003 adapter.go:164] Generating PVC name for m2
I0325 12:59:40.444838   95003 utils.go:98] Checking PVC for volume m2 and label component=openlibertydevfile,storage-name=m2
I0325 12:59:40.576924   95003 utils.go:105] Found an existing PVC for volume m2 and label component=openlibertydevfile,storage-name=m2
I0325 12:59:40.576946   95003 adapter.go:176] Found an existing PVC for m2, PVC m2-openlibertydevfile-rbrt will be re-used
I0325 12:59:40.576953   95003 adapter.go:164] Generating PVC name for m3
I0325 12:59:40.576974   95003 utils.go:98] Checking PVC for volume m3 and label component=openlibertydevfile,storage-name=m3
I0325 12:59:40.708774   95003 utils.go:105] Found an existing PVC for volume m3 and label component=openlibertydevfile,storage-name=m3
I0325 12:59:40.708806   95003 adapter.go:176] Found an existing PVC for m3, PVC m3-openlibertydevfile-naud will be re-used
I0325 12:59:40.708821   95003 adapter.go:164] Generating PVC name for m4
I0325 12:59:40.708859   95003 utils.go:98] Checking PVC for volume m4 and label component=openlibertydevfile,storage-name=m4
I0325 12:59:40.970176   95003 utils.go:105] Found an existing PVC for volume m4 and label component=openlibertydevfile,storage-name=m4
I0325 12:59:40.970206   95003 adapter.go:176] Found an existing PVC for m4, PVC m4-openlibertydevfile-oqes will be re-used
I0325 12:59:40.970266   95003 adapter.go:206] Creating deployment openlibertydevfile
I0325 12:59:40.970281   95003 adapter.go:207] The component name is openlibertydevfile
I0325 12:59:41.106118   95003 adapter.go:210] The component already exists, attempting to update it
I0325 12:59:41.245317   95003 adapter.go:215] Successfully updated component openlibertydevfile
I0325 12:59:41.523499   95003 adapter.go:229] Successfully update Service for component openlibertydevfile
I0325 12:59:41.523531   95003 utils.go:98] Checking PVC for volume m2 and label component=openlibertydevfile,storage-name=m2
I0325 12:59:41.655775   95003 utils.go:105] Found an existing PVC for volume m2 and label component=openlibertydevfile,storage-name=m2
I0325 12:59:41.655803   95003 utils.go:98] Checking PVC for volume m3 and label component=openlibertydevfile,storage-name=m3
I0325 12:59:41.851365   95003 utils.go:105] Found an existing PVC for volume m3 and label component=openlibertydevfile,storage-name=m3
I0325 12:59:41.851386   95003 utils.go:98] Checking PVC for volume m4 and label component=openlibertydevfile,storage-name=m4
I0325 12:59:41.989198   95003 utils.go:105] Found an existing PVC for volume m4 and label component=openlibertydevfile,storage-name=m4
I0325 12:59:41.989227   95003 deployments.go:43] Waiting for openlibertydevfile deployment roll out
I0325 12:59:42.127735   95003 deployments.go:71] Waiting for deployment "openlibertydevfile" rollout to finish: 0 out of 1 new replicas have been updated...
I0325 12:59:42.127762   95003 deployments.go:81] Waiting for deployment spec update to be observed...
I0325 12:59:50.673572   95003 deployments.go:71] Waiting for deployment "openlibertydevfile" rollout to finish: 0 out of 1 new replicas have been updated...
I0325 12:59:50.673588   95003 deployments.go:81] Waiting for deployment spec update to be observed...
I0325 12:59:50.688827   95003 deployments.go:71] Waiting for deployment "openlibertydevfile" rollout to finish: 0 out of 1 new replicas have been updated...
I0325 12:59:50.688854   95003 deployments.go:81] Waiting for deployment spec update to be observed...
I0325 12:59:50.726590   95003 deployments.go:75] Waiting for deployment "openlibertydevfile" rollout to finish: 0 of 1 updated replicas are available...
I0325 12:59:50.726605   95003 deployments.go:81] Waiting for deployment spec update to be observed...
I0325 12:59:54.770653   95003 deployments.go:77] Deployment "openlibertydevfile" successfully rolled out
I0325 12:59:54.770677   95003 deployments.go:81] Waiting for deployment spec update to be observed...
I0325 12:59:54.774020   95003 file_indexer.go:204] .odo or .git directory detected, skipping it
I0325 12:59:54.774271   95003 file_indexer.go:204] .odo or .git directory detected, skipping it
 ✓  Checking file changes for pushing [7ms]
I0325 12:59:54.777777   95003 adapter.go:104] List of files to be deleted: +[]
 ✓  No file changes detected, skipping build. Use the '-f' flag to force the build.
 ✓  Push devfile component openlibertydevfile [15s]
 ✓  Changes successfully pushed to component
I0325 12:59:54.777819   95003 odo.go:80] Could not get the latest release information in time. Never mind, exiting gracefully :)
rajivnathan picture rajivnathan on 25 Mar 2020

Hey @rajivnathan both @kadel and I are still unable to reproduce it. Are you sure you're trying this against the latest master?

cdrage picture cdrage on 30 Mar 2020

Since @rajivnathan is no longer working on this, can anyone else confirm that you've had the same issue?

Ping: @GeekArthur @yangcao77

cdrage picture cdrage on 31 Mar 2020

@cdrage I can give a try to see if I can reproduce this issue on my environment.

GeekArthur picture GeekArthur on 31 Mar 2020

@cdrage FYI, I couldn't reproduce this issue, it works for me 👍

GeekArthur picture GeekArthur on 31 Mar 2020

I couldn't reproduce this neither:

$ ./odo push
 •  Push devfile component test1  ...

Applying URL changes
 ✓  URL testurl: https://testurl.1.2.3.4.nip.io created
 ✓  Checking files for pushing [8ms]
 ✓  Waiting for component to start [122ms]
 ✓  Syncing files to the component [3m]
 ✓  Push devfile component test1 [4m]
 ✓  Changes successfully pushed to component
$ ./odo push
 •  Push devfile component test1  ...

Applying URL changes
 ✓  URL testurl already exists
 ✓  Checking file changes for pushing [2ms]
 ✓  No file changes detected, skipping build. Use the '-f' flag to force the build.
 ✓  Push devfile component test1 [2s]
 ✓  Changes successfully pushed to component
yangcao77 picture yangcao77 on 1 Apr 2020

Is this the issue that the volumes order change causes rebuild?
ping @maysunfaisal and @johnmcollier

yangcao77 picture yangcao77 on 1 Apr 2020

@yangcao77 That's correct, I was able to see the order of the volumes in the pod spec change sometimes and that causes the redeploy to occur. Another way to check that it's happening is to run kubectl get rs -w to watch for any ReplicaSet changes. If a redeploy occurs you will see new events during the push.

odoUpdateProblem

@cdrage @kadel Since I was the only one who was able to reproduce it, I investigated a bit further and found the problem and have a fix I'll submit a PR for. The problem was related to iterating over maps in a couple places and iteration order of maps is not guaranteed so I changed it to use splices of structs instead. Here are the changes for anyone that's interested: https://github.com/maysunfaisal/odo-fork/compare/2654-1...rajivnathan:2654-1?expand=1

With this change I can no longer reproduce. Since this is a relatively minor problem I'll put up a PR after https://github.com/openshift/odo/pull/2735 is merged.

rajivnathan picture rajivnathan on 3 Apr 2020
❤4 👍3

Thank you @rajivnathan ! https://github.com/openshift/odo/pull/2735 is merged now :)

cdrage picture cdrage on 6 Apr 2020

I haven't had this issue in a while.
Was this fixed?

kadel picture kadel on 26 Aug 2020

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

openshift-bot picture openshift-bot on 25 Nov 2020

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale

openshift-bot picture openshift-bot on 25 Dec 2020

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

openshift-bot picture openshift-bot on 24 Jan 2021

@openshift-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci-robot picture openshift-ci-robot on 24 Jan 2021
Was this page helpful?
0 / 5 - 0 ratings

Related issues

johnmcollier picture johnmcollier  ·  7Comments
dharmit picture dharmit  ·  6Comments
kadel picture kadel  ·  8Comments
amitkrout picture amitkrout  ·  3Comments
valaparthvi picture valaparthvi  ·  3Comments