Linuxbrew-core: Merge upstream to branch other than master
Right now when a maintainer merges upstream changes, it's merged to master branch. After that, there are no bottles available for upgraded formulas yet and when a user wants to install some package which formula were updated, at this time, he needs to build it from source and there is no guarantee that it will build successfully because a PR testing this might just be created.
I think it's a bad user experience and I propose to merge upstream changes to branch other than master, let's call it dev for now.
Would it be feasible? @iMichka @issyl0
dawidd6
All 8 comments
These questions are better asked on https://discourse.brew.sh.
As the response is quite long, I'll try to write something down in the next days :)
iMichka
on 26 Sep 2019
There a multiple reasons why we proceed like this. Let me try to explain them. Most of them are historical.
In the upstream Mac repo, they build each formula separately. Once the CI is green, the change gets merged. If some reverse deps need to be rebuilt, they will bump all the revisions where needed, until the CI is green). This makes sure that at any time, the formula tree is working and consistent.
In short: upstream is building most of the stuff in parallel. (It鈥檚 mostly parallel: if two pull request for two packages modify the same part of the dependency tree, one has to go in first, and the second PR has to be rebased on top of the first one once it has been merged).
On our side, we merge the upstream project into linuxbrew daily. Our process is more 芦聽serial聽禄 as upstream. Though we do build most stuff in parallel, we face the same issue as upstream and must be careful of the build and merge order for packages that depend on each other. We can not ignore an upstream change and just skip version updates they made: we try to be as much as possible in sync with upstream.
There are two strategies for the merge.
Open the merge pull request and wait until the 5-6 formulae have been built. Then we pull and merge the change. This is the best solution. Right now it is broken due to some various bugs in the pull command, when used with a merge commit. I need to work on a fix for this.
But sometimes I know that the build will last 2 hours or more (4:30 hours for llvm). As long as the merge request is open, we can not merge anything else into this repo.
As I have only one or two hours (or less) per day for linuxbrew, I often do it differently: I merge the pull request with the merge commit without waiting for the CI to finish. And then I open a separate pull request for each formula that needs a bottle rebuild. This allows me to catch up on all the upstream changes at once. It also allows to build multiple bottles in parallel.
The second strategy has a drawback: if a formula is broken, we ship a broken formula. And the bottles are not published directly.
The first solution may also have an issue: one of the formulae may fail to build. Still, I need to make that merge because I can not hold up the next merges. Especially when I have only a timeframe of one or two hours. So often I just do the merge, and open pull requests to build bottles, which will sit there waiting until I (or somebody else) finds time to fix it.
Back then when we were using circle-ci, it was the only way to follow the insane upstream rhythm. The CI was way slower. I had to open pull requests during breakfast time, let it build the whole day, and would pull them when coming back from work. The same happened in the evening and during the night. I was also mostly alone to work on the merges, and it was sort of the only way to move forward.
We now have more maintainers and a faster CI. We may even automatise the merge process even more.
We could do our merges to a second branch, and merge that once per day (or more) when it is in a stable state.
But nothing can go in master while unneeded stuff is on that branch: we can not push a quick fix for something else, like CVE fixes (though I never had to make that kind of change in a hurry).
We would need some changes to our infrastructure for all of this.
Of course it does not solve the issue for a package that would be completely broken.
The real solution would be to build the Mac and Linux bottles together in a single repo. We have not considered this until now because the Linux team was too small and it would slow down the upstream Mac repo. Also the dependency tree is not the same on Linux and on Mac: I think it would only be possible if they stopped using libraries provided by the system, and everything was built using brewed libraries.
Changing our way of working is possible but we need to think the different scenarios through ...
I think the first step will be to fix the building and pulling of merge commits. This is sort of a cheap version of building on a separate branch.
iMichka
on 30 Sep 2019
I think the first step will be to fix the building and pulling of merge commits. This is sort of a cheap version of building on a separate branch.
Building and pulling of merge commits was working at one point with CircleCI, and we can get back to that state with Azure. This workflow has its own challenges, namely that it's very serial, but I think it's worth pursuing.
sjackman
on 2 Oct 2019
This is an ongoing discussion, so I'm going to re-open this. Since the removal of the close bot, it shouldn't auto-close again? :pray:
issyl0
on 2 Oct 2019
Building and pulling of merge commits was working at one point with CircleCI, and we can get back to that state with Azure. This workflow has its own challenges, namely that it's very serial, but I think it's worth pursuing.
Agreed. We need to try to fix this.
The biggest issue with that workflow is that while the merge PR is open, nothing can be merged to master. A long time ago I would announce that homebrew-core was "frozen" in Slack, but that was too much a hassle. And it happened a few times in the last years where something was merged into master before the PR was merged, which was annoying (everybody hates losing time).
Ideally pushing to master should be disabled while a merge PR is open ...
iMichka
on 6 Oct 2019
If there were a change to master, depending on the change, it would be possible to merge (with a merge commit) the head of the PR into the new master, rather than requiring a fast-forward merge. I think that's the way to go, rather than locking (freezing) master.
sjackman
on 6 Oct 2019
https://github.com/Homebrew/linuxbrew-core/pull/16726
In this PR the bottle was built and uploaded successfully, but there were issues with brew pull.
dawidd6
on 27 Oct 2019
Closing here for the moment. We can still continue to discuss this but it looks like the status-quo is working right now. We are not really working on a change in the workflow for the moment. Of course I would also like to see something more robust being implemented regarding the way we are pushing changes into the master branch.
iMichka
on 30 Dec 2019
Related issues
iMichka
路
6Comments
baranyildirim
路
6Comments
vkyt
路
4Comments
tkaehn
路
5Comments
tsela
路
5Comments
Most helpful comment
There a multiple reasons why we proceed like this. Let me try to explain them. Most of them are historical.
In the upstream Mac repo, they build each formula separately. Once the CI is green, the change gets merged. If some reverse deps need to be rebuilt, they will bump all the revisions where needed, until the CI is green). This makes sure that at any time, the formula tree is working and consistent.
In short: upstream is building most of the stuff in parallel. (It鈥檚 mostly parallel: if two pull request for two packages modify the same part of the dependency tree, one has to go in first, and the second PR has to be rebased on top of the first one once it has been merged).
On our side, we merge the upstream project into linuxbrew daily. Our process is more 芦聽serial聽禄 as upstream. Though we do build most stuff in parallel, we face the same issue as upstream and must be careful of the build and merge order for packages that depend on each other. We can not ignore an upstream change and just skip version updates they made: we try to be as much as possible in sync with upstream.
There are two strategies for the merge.
Open the merge pull request and wait until the 5-6 formulae have been built. Then we pull and merge the change. This is the best solution. Right now it is broken due to some various bugs in the pull command, when used with a merge commit. I need to work on a fix for this.
But sometimes I know that the build will last 2 hours or more (4:30 hours for llvm). As long as the merge request is open, we can not merge anything else into this repo.
As I have only one or two hours (or less) per day for linuxbrew, I often do it differently: I merge the pull request with the merge commit without waiting for the CI to finish. And then I open a separate pull request for each formula that needs a bottle rebuild. This allows me to catch up on all the upstream changes at once. It also allows to build multiple bottles in parallel.
The second strategy has a drawback: if a formula is broken, we ship a broken formula. And the bottles are not published directly.
The first solution may also have an issue: one of the formulae may fail to build. Still, I need to make that merge because I can not hold up the next merges. Especially when I have only a timeframe of one or two hours. So often I just do the merge, and open pull requests to build bottles, which will sit there waiting until I (or somebody else) finds time to fix it.
Back then when we were using circle-ci, it was the only way to follow the insane upstream rhythm. The CI was way slower. I had to open pull requests during breakfast time, let it build the whole day, and would pull them when coming back from work. The same happened in the evening and during the night. I was also mostly alone to work on the merges, and it was sort of the only way to move forward.
We now have more maintainers and a faster CI. We may even automatise the merge process even more.
We could do our merges to a second branch, and merge that once per day (or more) when it is in a stable state.
But nothing can go in master while unneeded stuff is on that branch: we can not push a quick fix for something else, like CVE fixes (though I never had to make that kind of change in a hurry).
We would need some changes to our infrastructure for all of this.
Of course it does not solve the issue for a package that would be completely broken.
The real solution would be to build the Mac and Linux bottles together in a single repo. We have not considered this until now because the Linux team was too small and it would slow down the upstream Mac repo. Also the dependency tree is not the same on Linux and on Mac: I think it would only be possible if they stopped using libraries provided by the system, and everything was built using brewed libraries.
Changing our way of working is possible but we need to think the different scenarios through ...
I think the first step will be to fix the building and pulling of merge commits. This is sort of a cheap version of building on a separate branch.