Origin: OS_RELEASE=n hack/build-images.sh not working

[coreydaley]

[16:23] $ OS_RELEASE=n hack/build-images.sh
[FATAL] No release RPMs have been built! RPMs are necessary to build container images.
[FATAL] Build them with:
[FATAL] $ OS_BUILD_ENV_PRESERVE=_output/local/bin:_output/local/releases hack/env make build-rpms

Looks like these checks were added recently:
https://github.com/openshift/origin/blob/master/hack/build-images.sh#L12-L16

/cc @bparees

[bparees]

we definitely need a path to build product images for rapid test/iteration that does not require building an entire release first.

[kargakis]

You will always need to build the binaries beforehand, now you need to build the RPMs: hack/env make release or OS_BUILD_ENV_PRESERVE=_output/local hack/env make build-rpms; hack/build-images.sh if you want to avoid running extract-release.sh. I should have mentioned that on aos-devel yesterday, will drop an e-mail tomorrow. @stevekuznetsov not sure if it's possible to avoid rebuilding the RPMs every time you need to build an image, probably can't happen?

[kargakis]

we definitely need a path to build product images for rapid test/iteration that does not require building an entire release first.

Heard the same complaints today by @legionus @mfojtik - @stevekuznetsov @jupierce @smarterclayton we need to make a call. Can we have two sets of images, one that is using plain binaries and one that installs RPMs? The former used by devs, the latter used by our CI?

[smarterclayton]

One option is to do a quick makefile target that, given docker and a
built Linux binary, results in the image being tagged. The
alternative is using Cesar's proxy. And making it easier to run
alternate images

What do people need? Code -> real image? Code -> inside running oc
cluster up? Code -> inside our satellite images? People using the
registry?

[stevekuznetsov]

if it's possible to avoid rebuilding the RPMs every time you need to build an image, probably can't happen?

@kargakis there's probably some rpmbuild trickery we could come up with to update a specific file in the RPM only.

One option is to do a quick makefile target that, given docker and a built Linux binary, results in the image being tagged.

I think this is ultimately the goal. We should bring focus to it and try to determine the best approach. Currently we have a couple of options, right?

Can we have two sets of images, one that is using plain binaries and one that installs RPMs? The former used by devs, the latter used by our CI?

This would quickly become a headache as people don't understand why their CI isn't working when it worked locally.

[bparees]

What do people need? Code -> real image? Code -> inside running oc cluster up? Code -> inside our satellite images? People using the registry?

for me i have two use cases:

1) code running inside cluster up (need a new origin master image)
2) all the build logic that runs inside an image, whenever we are working with that we need to update the builder images.

i'm guessing @mfojtik has a similar need for (2) w/ respect to deployment images.

[legionus]

@kargakis @stevekuznetsov It doesn't matter anymore I made my own build scripts to speedup my work and now I do not use hack/build-images.sh at all. I use my own Dockerfile and so on.

[bparees]

@kargakis @stevekuznetsov It doesn't matter anybore I made my own build scripts to speedup my work and now I do not use hack/build-images.sh at all. I use my own Dockerfile and so on.

"i fixed it for myself and everyone else can go find their own solution" is not helpful to the organization.

[stevekuznetsov]

@legionus @dmage I agree with @bparees on this one ... that does not seem like the most productive process for the team.

My utmost concern its that we minimize the differences between what a developer has on their local machine and what we use in CI and then what we have in the product. The further that developers drift from what we push out to the world for a product and what we have in our development environments, the greater the chance is that your efforts in a development environment is not actually valid.

[dmage]

@stevekuznetsov the change-build-run cycle should take seconds, not minutes. I'm not going to make the release build on every save, because it will slow down debugging dramatically. I can re-test my changes using the "right way" build when I finish, but we have the CI, don't we?

[stevekuznetsov]

the change-build-run cycle should take seconds, not minutes

No argument there.

I can re-test my changes using the "right way" build when I finish, but we have the CI, don't we?

Are people generally happy to spend a development cycle doing work that ultimately is not correct because they were testing with an incorrect configuration, meaning that they end up doing more work to finish it out?

[legionus]

"i fixed it for myself and everyone else can go find their own solution" is not helpful to the organization.

It's even more unhelpful for the organization if I wait 10-15 minutes to see my debugging messages during development.

I agree to wait for rebuilding to a maximum of 3 minutes. While our build process takes more time, I will not use it during development. If I am developing dockerregistry I don't want to wait much longer than the hack/build-go.sh cmd/dockerregistry.

My utmost concern its that we minimize the differences between what a developer has on their local machine and what we use in CI and then what we have in the product. The further that developers drift from what we push out to the world for a product and what we have in our development environments, the greater the chance is that your efforts in a development environment is not actually valid.

The current build process is not friendly to the developer.

FYI I know that my colleagues have own scripts like mine. So basically everybody uses own solutions already.

[bparees]

It's even more unhelpful for the organization if I wait 10-15 minutes to see my debugging messages during development.

no disagreement, that's why this issue was created. So we can create a solution that everyone can benefit from, rather than:

FYI I know that my colleagues have own scripts like mine. So basically everybody uses own solutions already.

That means every new developer who joins the team has to figure out on their own, "oh, this situation isn't optimal. I wonder how i can fix it." If they're lucky, they'll ask someone who has their own scripts and copy them, eventually. If they're unlucky, they'll waste time writing yet another set of custom scripts.

My point was your response of "i don't care about this anymore because i solved it for myself" is not helpful. Helpful is: "Yes this is a real problem, here are the scripts I've created, perhaps we can vet them and put them in the repo so everyone can use them, if they are deemed to be a good solution?"

[bparees]

And while we're on the subject of speeding up building the images, something i'd like to see in the build-images script is the capability to build specific images rather than all of them. I don't need to build 90% of the images, i just need the base and the build images.

[legionus]

Helpful is: "Yes this is a real problem, here are the scripts I've created, perhaps we can vet them and put them in the repo so everyone can use them, if they are deemed to be a good solution?"

@bparees This approach assumes that I have time to develop a "right solution" that everyone will like. But it is not so. I have quite different tasks in trello that need to be done on time. I solved my problem to work effectively. I'm surprised that having your own build scripts for someone is a surprise. The only thing that I did and that was unusual, I sounded well-known fact.

Even more, we have a hack in the repository to speed up the dockerregistry development in some cases: https://github.com/openshift/origin/commit/5935b5a4e5e6d6e082bc9b5f017f13a9620c4d50 The presence of such hacks indicates that the current build system is not very useful.

[stevekuznetsov]

something i'd like to see in the build-images script is the capability to build specific images rather than all of them

@bparees work was started on a Makefile approach to this in #13444 by @smarterclayton

This approach assumes that I have time to develop a "right solution" that everyone will like.

@legionus while I understand where you're coming from, it just isn't ultimately a productive approach for the organization. I'll try to get an aos-devel thread open for this so we can try to bring some other tools into the Origin repository. In the meanwhile, can you open a PR with your scripts, or a gist?

[bparees]

I feel like maybe my message isn't clear. I'm not surprised you created your own scripts. and I don't expect you go evangelize them or develop a solution the organization can use (though it would be great if you did have the time to do so, but i understand the constraints we all work under). My sole criticism was your response to this thread of "this isn't a problem anymore because i have my own scripts".

This thread is to solve this problem for the organization because it's a problem we all have. The fact that you had to solve it on your own is proof it is a problem and not a solved problem.

[stevekuznetsov]

This thread is to solve this problem for the organization because it's a problem we all have. The fact that you had to solve it on your own is proof it is a problem and not a solved problem.

Also to be clear I'm asking that you share them not because I want you to maintain them for everyone, but just so we can have a more broad set of options available to us to choose from when we are looking to solve this problem for everyone/

[dmage]

@stevekuznetsov here is mine: https://github.com/dmage/dotfiles/blob/base/bin/ocdev

[stevekuznetsov]

@dmage so you have approached this with the mount approach. Cool. @smarterclayton what were your concerns with pushing for the bind mount proxy for being the defacto dev solution?

[smarterclayton]

I'm ok with a simple set of stuff that makes bind mount easy. If it
spreads into lots of other code that's probably not good.

[stevekuznetsov]

@csrwng It was mentioned that the proxy would not work for non-binary files in the images -- is it a simple thing to extend it?

[csrwng]

@stevekuznetsov you can mount whatever files you want with the proxy. It uses a json config telling it what to mount into what image (specified through an env var):
https://github.com/csrwng/bindmountproxy/blob/master/cmd/bindmountproxy/main.go#L23

Here's an example config:
https://gist.github.com/csrwng/33ff19a06b30989951ec39dd95f3ccd4