Origin: DeploymentConfig behaviour change from 3.6 to 3.7

Created on 2 Feb 2018  路  19Comments  路  Source: openshift/origin

In 3.6, using F-M-P which updates the both the container image reference and the image trigger reference, there was no automatic re-deployment when trigger is set to automatic false.

In 3.7, it seems like an update to the container image reference automatically re-deploys.

This breaks our workflow.

Version

oc v3.7.0+7ed6862
kubernetes v1.7.6+a08f5eeb62
features: Basic-Auth

openshift v3.7.0+7ed6862
kubernetes v1.7.6+a08f5eeb62

Steps To Reproduce

3.7:

  1. Create a DC and deploy it using the rollout command.
  2. Update the image container reference and redeploy.

3.6:

  1. Create a DC and deploy it using the rollout command.
  2. Update the image container reference and redeploy.
Current Result

3.7:
It will trigger a deployment.
3.6:
Nothing will happen.

Expected Result

3.7:
Nothing should happen (as in 3.6) until a manual rollout command has been executed.

@soltysh @mfojtik @bparees

componenapps kinbug prioritP1 simaster
Source
picture mojsha

Most helpful comment

Thanks for the info I think I now now what's wrong. The trigger controller won't inject an image because it's paused, but since you set it "manually" the DC controller thinks there has been an image injected and considers that an image change.

I think that should be fixable by the DC controller also checking if the trigger isn't paused.

There should be an easy way to around it in the mean time: If you are using ImageChangeTrigger always set container's image to " ". Actually that applies in general - if there is a configChangeTrigger never set image to anything else than " ". That way even if you need to do something like PUT or oc apply won't mess it up.

picture tnozicka on 5 Feb 2018
👍4

All 19 comments

starting as a p1 because it's a regression/change in behavior (possibly intentional and justifiable)

bparees picture bparees on 2 Feb 2018

(if anything it sounds like the 3.6 behavior was broken, so perhaps this was a bug fix in 3.7)

bparees picture bparees on 2 Feb 2018

@bparees From my perspective, a change in the DC - any change - should _only_ be triggered if there is a ConfigChange trigger. That is, after all, why we have the ConfigChange trigger.

If anything else in the DC is changed, it does not re-deploy. So if this is intentional behaviour, it would effectively amount to an implicit/hidden trigger on the container image reference being introduced.

@LarsMilland

mojsha picture mojsha on 5 Feb 2018

Ok, after further analysis I can see that F-M-P is doing a PUT which triggers the re-deploy.

If I do a PATCH with the same content and change the container image reference, it does not seem to trigger a re-deploy.

mojsha picture mojsha on 5 Feb 2018

What is FMP?

kargakis picture kargakis on 5 Feb 2018

@kargakis Sorry, I thought it was a known abbreviation/acronym... it stands for Fabric8 Maven Plugin.

mojsha picture mojsha on 5 Feb 2018

Can you post the DC before and after doing a PUT and before and after doing a PATCH?

cc @tnozicka

kargakis picture kargakis on 5 Feb 2018

That what I was going to ask, please always provide the DC's YAML and exact steps to reproduce (ideally commands) - it will save us time when trying to understand and resolve the issue.

tnozicka picture tnozicka on 5 Feb 2018

Ok, I think I was wrong about the PUT vs PATCH thing. The behaviour is the same regardless.

But I will try to provide all the steps with a higher level of detail:
I use a disabled (automatic = false) ImageTrigger in order to be able to refer to images using the ImageStream tags, so the "before" is going to be the state after which it has been rolled out and the container image resolved.

Before PUT / PATCH:

apiVersion: v1
kind: DeploymentConfig
metadata:
  annotations:
    fabric8.io/git-branch: master
    fabric8.io/git-commit: 321aca8e89fa7c173def57e078e24c433b413c3a
    fabric8.io/scm-tag: HEAD
    fabric8.io/scm-url: https://github.com/spring-projects/spring-boot/spring-boot-starter-parent/demo
  creationTimestamp: null
  generation: 1
  name: demo
spec:
  replicas: 1
  revisionHistoryLimit: 2
  selector:
    group: test.eu
    project: demo
    provider: fabric8
  strategy:
    activeDeadlineSeconds: 21600
    resources: {}
    rollingParams:
      intervalSeconds: 1
      maxSurge: 25%
      maxUnavailable: 25%
      timeoutSeconds: 3600
      updatePeriodSeconds: 1
    type: Rolling
  template:
    metadata:
      annotations:
        fabric8.io/git-branch: master
        fabric8.io/git-commit: 321aca8e89fa7c173def57e078e24c433b413c3a
        fabric8.io/scm-tag: HEAD
        fabric8.io/scm-url: https://github.com/spring-projects/spring-boot/spring-boot-starter-parent/demo
      creationTimestamp: null
      labels:
        group: test.eu
        project: demo
        provider: fabric8
    spec:
      containers:
      - args:
        - /usr/local/s2i/run
        image: 172.30.251.31:5000/fabric8test/demo@sha256:b49793e3c286cbd491a4790bca06c4ea8a805568df1e75b8bd075cd3e298fe5c
        imagePullPolicy: Always
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /health
            port: 8081
            scheme: HTTP
          initialDelaySeconds: 180
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        name: demo
        ports:
        - containerPort: 8443
          name: https
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /health
            port: 8081
            scheme: HTTP
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /sample
          name: sample
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      volumes:
      - emptyDir: {}
        name: sample
  test: false
  triggers:
  - imageChangeParams:
      containerNames:
      - demo
      from:
        kind: ImageStreamTag
        name: demo:1.0.37-Stable-Release
        namespace: fabric8test
    type: ImageChange
status:
  availableReplicas: 0
  latestVersion: 0
  observedGeneration: 0
  replicas: 0
  unavailableReplicas: 0
  updatedReplicas: 0

After PUT (using F-M-P)

apiVersion: v1
kind: DeploymentConfig
metadata:
  annotations:
    fabric8.io/git-branch: master
    fabric8.io/git-commit: 321aca8e89fa7c173def57e078e24c433b413c3a
    fabric8.io/scm-tag: HEAD
    fabric8.io/scm-url: https://github.com/spring-projects/spring-boot/spring-boot-starter-parent/demo
  creationTimestamp: null
  generation: 1
  name: demo
spec:
  replicas: 1
  revisionHistoryLimit: 2
  selector:
    group: test.eu
    project: demo
    provider: fabric8
  strategy:
    activeDeadlineSeconds: 21600
    resources: {}
    rollingParams:
      intervalSeconds: 1
      maxSurge: 25%
      maxUnavailable: 25%
      timeoutSeconds: 3600
      updatePeriodSeconds: 1
    type: Rolling
  template:
    metadata:
      annotations:
        fabric8.io/git-branch: master
        fabric8.io/git-commit: 321aca8e89fa7c173def57e078e24c433b413c3a
        fabric8.io/scm-tag: HEAD
        fabric8.io/scm-url: https://github.com/spring-projects/spring-boot/spring-boot-starter-parent/demo
      creationTimestamp: null
      labels:
        group: test.eu
        project: demo
        provider: fabric8
    spec:
      containers:
      - args:
        - /usr/local/s2i/run
        image: fabric8test/demo:1.0.38-Stable-Release
        imagePullPolicy: Always
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /health
            port: 8081
            scheme: HTTP
          initialDelaySeconds: 180
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        name: demo
        ports:
        - containerPort: 8443
          name: https
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /health
            port: 8081
            scheme: HTTP
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /sample
          name: sample
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      volumes:
      - emptyDir: {}
        name: sample
  test: false
  triggers:
  - imageChangeParams:
      containerNames:
      - demo
      from:
        kind: ImageStreamTag
        name: demo:1.0.38-Stable-Release
        namespace: fabric8test
    type: ImageChange
status:
  availableReplicas: 0
  latestVersion: 0
  observedGeneration: 0
  replicas: 0
  unavailableReplicas: 0
  updatedReplicas: 0

After PATCH

apiVersion: v1
kind: DeploymentConfig
metadata:
  annotations:
    fabric8.io/git-branch: master
    fabric8.io/git-commit: 321aca8e89fa7c173def57e078e24c433b413c3a
    fabric8.io/scm-tag: HEAD
    fabric8.io/scm-url: https://github.com/spring-projects/spring-boot/spring-boot-starter-parent/demo
  creationTimestamp: null
  generation: 1
  name: demo
spec:
  replicas: 1
  revisionHistoryLimit: 2
  selector:
    group: test.eu
    project: demo
    provider: fabric8
  strategy:
    activeDeadlineSeconds: 21600
    resources: {}
    rollingParams:
      intervalSeconds: 1
      maxSurge: 25%
      maxUnavailable: 25%
      timeoutSeconds: 3600
      updatePeriodSeconds: 1
    type: Rolling
  template:
    metadata:
      annotations:
        fabric8.io/git-branch: master
        fabric8.io/git-commit: 321aca8e89fa7c173def57e078e24c433b413c3a
        fabric8.io/scm-tag: HEAD
        fabric8.io/scm-url: https://github.com/spring-projects/spring-boot/spring-boot-starter-parent/demo
      creationTimestamp: null
      labels:
        group: test.eu
        project: demo
        provider: fabric8
    spec:
      containers:
      - args:
        - /usr/local/s2i/run
        image: fabric8test/demo:1.0.38-Stable-Release
        imagePullPolicy: Always
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /health
            port: 8081
            scheme: HTTP
          initialDelaySeconds: 180
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        name: demo
        ports:
        - containerPort: 8443
          name: https
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /health
            port: 8081
            scheme: HTTP
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /sample
          name: sample
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      volumes:
      - emptyDir: {}
        name: sample
  test: false
  triggers:
  - imageChangeParams:
      containerNames:
      - demo
      from:
        kind: ImageStreamTag
        name: demo:1.0.38-Stable-Release
        namespace: fabric8test
    type: ImageChange
status:
  availableReplicas: 0
  latestVersion: 0
  observedGeneration: 0
  replicas: 0
  unavailableReplicas: 0
  updatedReplicas: 0
mojsha picture mojsha on 5 Feb 2018

And finally, the reason I was wrong about the difference between PUT / PATCH:

When you PUT / PATCH an _undeployed_ (i.e. not yet had 'oc rollout' executed against it) application, it seems to work fine, i.e. it does not trigger a deployment. I had executed the PATCH against an undeployed application.

Here is the DC before and after an undeployed app has been patched:

Before

apiVersion: v1
kind: DeploymentConfig
metadata:
  annotations:
    fabric8.io/git-branch: master
    fabric8.io/git-commit: 321aca8e89fa7c173def57e078e24c433b413c3a
    fabric8.io/scm-tag: HEAD
    fabric8.io/scm-url: https://github.com/spring-projects/spring-boot/spring-boot-starter-parent/demo
  creationTimestamp: null
  generation: 1
  name: demo
spec:
  replicas: 1
  revisionHistoryLimit: 2
  selector:
    group: test.eu
    project: demo
    provider: fabric8
  strategy:
    activeDeadlineSeconds: 21600
    resources: {}
    rollingParams:
      intervalSeconds: 1
      maxSurge: 25%
      maxUnavailable: 25%
      timeoutSeconds: 3600
      updatePeriodSeconds: 1
    type: Rolling
  template:
    metadata:
      annotations:
        fabric8.io/git-branch: master
        fabric8.io/git-commit: 321aca8e89fa7c173def57e078e24c433b413c3a
        fabric8.io/scm-tag: HEAD
        fabric8.io/scm-url: https://github.com/spring-projects/spring-boot/spring-boot-starter-parent/demo
      creationTimestamp: null
      labels:
        group: test.eu
        project: demo
        provider: fabric8
    spec:
      containers:
      - args:
        - /usr/local/s2i/run
        image: fabric8test/demo:1.0.38-Stable-Release
        imagePullPolicy: Always
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /health
            port: 8081
            scheme: HTTP
          initialDelaySeconds: 180
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        name: demo
        ports:
        - containerPort: 8443
          name: https
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /health
            port: 8081
            scheme: HTTP
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /sample
          name: sample
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      volumes:
      - emptyDir: {}
        name: sample
  test: false
  triggers:
  - imageChangeParams:
      containerNames:
      - demo
      from:
        kind: ImageStreamTag
        name: demo:1.0.38-Stable-Release
        namespace: fabric8test
    type: ImageChange
status:
  availableReplicas: 0
  latestVersion: 0
  observedGeneration: 0
  replicas: 0
  unavailableReplicas: 0
  updatedReplicas: 0

After

apiVersion: v1
kind: DeploymentConfig
metadata:
  annotations:
    fabric8.io/git-branch: master
    fabric8.io/git-commit: 321aca8e89fa7c173def57e078e24c433b413c3a
    fabric8.io/scm-tag: HEAD
    fabric8.io/scm-url: https://github.com/spring-projects/spring-boot/spring-boot-starter-parent/demo
  creationTimestamp: null
  generation: 1
  name: demo
spec:
  replicas: 1
  revisionHistoryLimit: 2
  selector:
    group: test.eu
    project: demo
    provider: fabric8
  strategy:
    activeDeadlineSeconds: 21600
    resources: {}
    rollingParams:
      intervalSeconds: 1
      maxSurge: 25%
      maxUnavailable: 25%
      timeoutSeconds: 3600
      updatePeriodSeconds: 1
    type: Rolling
  template:
    metadata:
      annotations:
        fabric8.io/git-branch: master
        fabric8.io/git-commit: 321aca8e89fa7c173def57e078e24c433b413c3a
        fabric8.io/scm-tag: HEAD
        fabric8.io/scm-url: https://github.com/spring-projects/spring-boot/spring-boot-starter-parent/demo
      creationTimestamp: null
      labels:
        group: test.eu
        project: demo
        provider: fabric8
    spec:
      containers:
      - args:
        - /usr/local/s2i/run
        image: fabric8test/demo:1.0.40-Stable-Release
        imagePullPolicy: Always
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /health
            port: 8081
            scheme: HTTP
          initialDelaySeconds: 180
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        name: demo
        ports:
        - containerPort: 8443
          name: https
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /health
            port: 8081
            scheme: HTTP
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /sample
          name: sample
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      volumes:
      - emptyDir: {}
        name: sample
  test: false
  triggers:
  - imageChangeParams:
      containerNames:
      - demo
      from:
        kind: ImageStreamTag
        name: demo:1.0.38-Stable-Release
        namespace: fabric8test
    type: ImageChange
status:
  availableReplicas: 0
  latestVersion: 0
  observedGeneration: 0
  replicas: 0
  unavailableReplicas: 0
  updatedReplicas: 0
mojsha picture mojsha on 5 Feb 2018

Thanks for the info I think I now now what's wrong. The trigger controller won't inject an image because it's paused, but since you set it "manually" the DC controller thinks there has been an image injected and considers that an image change.

I think that should be fixable by the DC controller also checking if the trigger isn't paused.

There should be an easy way to around it in the mean time: If you are using ImageChangeTrigger always set container's image to " ". Actually that applies in general - if there is a configChangeTrigger never set image to anything else than " ". That way even if you need to do something like PUT or oc apply won't mess it up.

tnozicka picture tnozicka on 5 Feb 2018
👍4

@tnozicka Thank you for this suggestion - it worked!

EDIT: @tnozicka My earlier experiment was unfortunately not entire correctly placed. It does work - _the first time_. Any subsequent rollout leads to a problem with the RC complaining about a missing image reference and seems to be an issue you are well aware of: https://github.com/openshift/origin/issues/16728.

Can you tell me how I should proceed? This is a huge blocker for my organization - we cannot proceed with upgrade to 3.7+ due to this, as all our projects are dependent on this build and deployment mechanism.

Can you backport the fix to a 3.7.x-version, or provide an alternative way short of waiting for 3.9 to be released? Much appreciated.

mojsha picture mojsha on 6 Feb 2018

I'll check and see how much conflict there will be in release-3.7.

tnozicka picture tnozicka on 7 Feb 2018

Not sure whether there will be another release of 3.7.x though.

tnozicka picture tnozicka on 7 Feb 2018

backport to 3.7 https://github.com/openshift/origin/pull/18524

tnozicka picture tnozicka on 8 Feb 2018
🎉1 👍1

Hello @tnozicka , I try to apply your workaround from https://github.com/openshift/origin/issues/18406#issuecomment-363067144 which is to set the DC's container.image to " " (string containing a space).

Using OSE v3.7.0, I am unable to deploy the DC with container.image set to " " and automatic imageChangeTrigger, the following messages appears in Events "Couldn't deploy version 5: ReplicationController "xxxx-5" is invalid: spec.template.spec.containers[0].image: Required value".

jperville picture jperville on 8 Feb 2018

Hi @jperville, that backport is not merged yet (hitting unrelated flake)

Using OSE v3.7.0, I am unable to deploy the DC with container.image set to " " and automatic imageChangeTrigger, the following messages appears in Events "Couldn't deploy version 5: ReplicationController "xxxx-5" is invalid: spec.template.spec.containers[0].image: Required value".

OSE backport will follow once that PR makes it through the queue in Origin ;)

I am unable to deploy the DC with container.image set to " "

That is strange as this is just short time failure as the image in DC will get eventually rewritten by image trigger causing a new rollout with valid image.

tnozicka picture tnozicka on 12 Feb 2018

maybe it doesn't get synced; that part is being fixed in that backport as well

tnozicka picture tnozicka on 12 Feb 2018

Both backports are merged now.

tnozicka picture tnozicka on 25 Apr 2018
Was this page helpful?
0 / 5 - 0 ratings

Related issues

enj picture enj  路  4Comments
slmzig picture slmzig  路  3Comments
edseymour picture edseymour  路  3Comments
rkrmishra picture rkrmishra  路  4Comments
smugcloud picture smugcloud  路  5Comments