Openshift-ansible: CI job for 3.7 branch does screw the spec file Version

[DanyC97]

@jupierce fyi it seems the CI job which cuts a new release for 3.7 branch no longer update the spec file and that has an impact on us - PaaS when we create the rpms.

any chance you can correct it please?

[DanyC97]

@sdodson fyi

[jupierce]

@DanyC97 @markllama should be able to help here. For future build related work, @tbielawa is the new technical lead for the team.

[sdodson]

/assign @markllama @tbielawa

[tbielawa]

Only 3.7?

[DanyC97]

@jupierce thank you for the info, i ack the change.

@tbielawa yes, only 3.7, 3.9 looks okay so far.

[DanyC97]

and note @tbielawa that up until 3.7.52 it was all good so recently something went south.
thank you for looking at btw

[DanyC97]

@tbielawa i take back what i said, i just built a 3.10 rpm and happens the same

https://github.com/openshift/openshift-ansible/blob/release-3.10/openshift-ansible.spec#L12-L13

[markllama]

@jupierce can you give me a spec for what should be happening compared to what is?

[tbielawa]

This should be filed on the CI jobs page. I opened an issue there

[markllama]

@DanyC97 I'm not sure what was being done to the spec file prior to a recent change to standardize package builds. What change are you expecting and what are you seeing? It's possible that you were depending on a side effect that should not have gotten downstream to you. If so we'll have to talk about how to resolve it.

[markllama]

@DanyC97 can you point me at where/how you're seeing a failure? What condition is failing that depends on the spec file updating for each build?

[DanyC97]

@markllama so here is the story.

In PaaS what we do for openshift-ansbile is to checkout the repo and then we build the src rpm using tito. Once that is done then CBS is building the rpms.

Now up until now we have not changed the spec file to accommodate every tag release we were planning to build the rpm for since it was already done by you guys.

Because of the recent change (would appreciate if you can point me to _recent change to standardize package builds_ ) the behavior is seen in https://ci.centos.org/view/PaaS-SIG/job/paas-ci-cd/1756/console where

21:07:19 TASK [cbs : Build srpm with tito for openshift-ansible] ************************
21:07:19 task path: /home/sig-paas/workspace/paas-ci-cd/paas-ci/playbooks/openshift/roles/cbs/tasks/cbs_build_srpms.yml:3
21:07:19 Friday 22 June 2018  21:07:19 +0000 (0:00:03.274)       0:00:18.257 *********** 

21:07:26 changed: [n18.gusty] => {"changed": true, "cmd": "tito build --test --srpm --rpmbuild-options=\"--define 'dist .el7'\" --output=/root/openshift-ansible | grep 'src.rpm' | awk '{print $NF}'", "delta": "0:00:05.295975", "end": "2018-06-22 22:07:25.371620", "failed": false, "rc": 0, "start": "2018-06-22 22:07:20.075645", "stderr": "", "stderr_lines": [], "stdout": "/root/openshift-ansible/openshift-ansible-3.7.52-1.git.14.8a32cd1.el7.src.rpm", "stdout_lines": ["/root/openshift-ansible/openshift-ansible-3.7.52-1.git.14.8a32cd1.el7.src.rpm"]}

look at the rpm version although that should have been

21:06:57 TASK [bfs : Checkout openshift-ansible version - openshift-ansible-3.7.55-1] ***
21:06:57 task path: /home/sig-paas/workspace/paas-ci-cd/paas-ci/playbooks/openshift/roles/bfs/tasks/build_openshift-ansible_srpm.yml:25
21:06:57 Friday 22 June 2018  21:06:56 +0000 (0:00:00.134)       0:02:12.441 *********** 

21:06:58 changed: [n18.gusty] => {"changed": true, "cmd": ["git", "checkout", "openshift-ansible-3.7.55-1"], "delta": "0:00:00.533361", "end": "2018-06-22 22:06:57.816246", "failed": false, "rc": 0, "start": "2018-06-22 22:06:57.282885", "stderr": "Note: checking out 'openshift-ansible-3.7.55-1'.\n\nYou are in 'detached HEAD' state. You can look around, make experimental\nchanges and commit them, and you can discard any commits you make in this\nstate without impacting any branches by performing another checkout.\n\nIf you want to create a new branch to retain commits you create, you may\ndo so (now or later) by using -b with the checkout command again. Example:\n\n  git checkout -b new_branch_name\n\nHEAD is now at 3698b3e... Merge pull request #8778 from fbac/patch-1", "stderr_lines": ["Note: checking out 'openshift-ansible-3.7.55-1'.", "", "You are in 'detached HEAD' state. You can look around, make experimental", "changes and commit them, and you can discard any commits you make in this", "state without impacting any branches by performing another checkout.", "", "If you want to create a new branch to retain commits you create, you may", "do so (now or later) by using -b with the checkout command again. Example:", "", "  git checkout -b new_branch_name", "", "HEAD is now at 3698b3e... Merge pull request #8778 from fbac/patch-1"], "stdout": "", "stdout_lines": []}

hope that makes sense ...

[DanyC97]

now of course i can modify the spec file if i know what the plan is moving fwd - ie you guys are stopping updating the version/ release with each tag release.

Frankly to me it makes sense to have the version in the spec file since you are cutting the tag release but i guess i don't have the full picture on why that is a bad thing to do

[DanyC97]

@markllama do you need any other info from me ? if possible i'd like to resolve this issue soon so any help much appreciated

[markllama]

@DanyC97: I have what I need for now. I'm waiting for @smarterclayton to decide on the best course of action.

[smarterclayton]

At this point it would be better if you modify the spec file. For common components, we pass variables to the spec file rather than updating them and committing them.

[DanyC97]

okay @smarterclayton i have no problem with that.

The only thing i'd have preferred to see was a notification (an email dropped to dev mailer or something ) to say that you guys are planning to make the changes on the CI job for consistency rather than just discover when things break.

Hopefully that can be taken into consideration next time ;)

PS - i'd still love @markllama if you can point me to the commit which change the behavior for ansible CI job

[DanyC97]

i should also add that with the change you guys made we lost the changelog which came with each release.

• release with changelog
• release w/o changelog

What are we going to do about it @smarterclayton @markllama ?

[markllama]

@DanyC97 DanyC97: Changelog is back: PR https://github.com/openshift/aos-cd-jobs/pull/1493

[markllama]

@DanyC97 The RPM change for openshift-ansible has a long and twisty history.

The motivation for the change is that the build process has a standard mechanism for building OCP related RPMs. openshift-ansible build code was custom snowflake and was written before the standard mechanism existed. "make it standard" I was told. OK.

These two PRs moved the openshift-ansible build for versions > 3.5 to use the "standard" method.

https://github.com/openshift/aos-cd-jobs/pull/1404
https://github.com/openshift/enterprise-images/pull/298

That's when we lost the changelog for openshift-ansible (since returned, see openshift/aos-cd-jobs#1493).

When I was making the change, I was unaware that our "standard" build process isn't so standard. It ignores the version in the spec file and forces the package version to be in lockstep with the openshift version it is being built with. Because we change the version, we can't legitimately push it back to the original sources (not my call).

This change uncovered a number of undocumented side effects of both the custom and "standard" RPM build processes for OpenShift RPMs. This is the last one (I'm aware of) that needs to be addressed.

The normal behavior of tito is to update the spec file with the build version and push it back. This isn't happening now. I'm still in discussions about the proper change to address this. I'm still pressing for consistent resolution.

[tbielawa]

@DanyC97 that second link to enterprise-images may not load for you. It's just a lot of YAML formatted metadata updates.

[DanyC97]

thanks a bunch for the info provided @markllama !

regarding

The normal behavior of tito is to update the spec file with the build version and push it back.

let me know / ping me when you find out as i'm curious and maybe i cna do the same in the PaaS code base.

@tbielawa indeed it doesn't ;) thx for heads up

[DanyC97]

@markllama fyi the latest 3.9 or 3.7 openshift-ansible releases still don't have the changelog, am i missing s'thing here ?

[markllama]

@DanyC97 I reverted the openshift-ansible RPM build method, so the tagging and changelog of future builds should return to pre-change states. As they come in, let me know.

My apologies.

[DanyC97]

issue being solved, i'm closing it. Thank you all for your help