Yay: Install Algorithm Progress

[Morganamilo]

This issue is to centralise all problems and progress on the new install algorithm. The prior pull requests can be seen here #105 #107 and should be read as some of the know problems are outlined there in detail.

Formatting and printing to the user should be discussed separately in #109 .

If there is a major bug in the install or you wish to discuss a problem in detail a separate issue should be opened. This issue is for general discussion and bug reporting.

Testing

If you are willing to test you can either test against the master branch or any open pull requests if there are any. If doing the latter you should comment on the pull request instead of here.

My testing is usually preformed in a clean chroot with only base and base-devel installed as to pull in as many dependencies as possible during each operation. This really helps in spotting missing or incorrect dependency solving.

I will be opening another issue #110 to keep track of difficult to install packages and packages which currently do not install correctly.

## Status

• [x] Reliable solver of complex dependencies
• Nothing has been found yet so I'm amusing this is working
• As far as i know the following issues are fixed but more testing it needed to tick this box.
• This is mostly the case, we still miss some dependencies when dealing with packages that are pulled using its provides instead of its package name.

• This can be seen when installing discord and the dependency tree tries to pull libjpeg provided by libjpeg-turbo

• [x] Split packages

• Split packages are currently all or nothing. Yay will always install every package generated by a split package. If two packages foo and bar from the same pkgbase base are specified, Yay in an attempt to build foo will build base, install foo and bar then in an attempt to install bar, will rebuild base and reinstall foo and bar.

• This can be seen when installing discord and the dependency tree pulls libc++abi and libc++.

• [x] Circular dependencies

• After realising packages wont even build without their dependencies installed I'm not supporting circular dependencies and marking this as solved.

• Don't seem to be a problem for repo packages, I am currently unaware of any AUR packages with circular dependencies.

• [x] Versioned dependancies
  - Checked by goalpm for repo packages. Currently ignored for AUR packages.
  - I am unsure whether yay should handle this, goalpm should handle this or gopkgbuild should handle this.

• [ ] IgnorePkg/IgnoreGroup

• Checked during sysupgrade, not during normal install.

• [ ] Respect all Pacman options (--root --dbpath --config...)

• Multiple CacheDirs is not supported
• Passing an option twice but mixing long and short format e.g. pacman -s -i --info will not be respected by yay but will be respected when passed to pacman.
• Pretty much every option should be passed to pacman now but yay does not respect all of them.
• Siglevel is not respected in go-alpm but we always pass to pacman for installs so it shouldn't effect anything.
• HoldPkg is only used on remove and we always pass to pacman for that so we dont need to handle it.
• We respect most Pacman options via alpm. Some options such as HoldPkg are Pacman specific and not part of alpm, Yay does not support these.

• Installing packages from the AUR causes Pacman options to go unrespected due to makepkg handling this via makepkg -i and makepkg -s

• [ ] Make sure there are as few install transactions as possible, one call to pacman -U is preferable to multiple makepkg -i calls.

• One pacman -U is now used once per pkgBase

[maximbaz]

This is implied by the last point, but I just want to explicitly add it to the list:

• [ ] Make sure there are as few install transactions as possible, one call to pacman -U is preferable to multiple makepkg -i calls.

[Morganamilo]

Added to the list. I don't plan to implement this myself though. My plan is one call to pacman -U per pkgbase. If i ever get around to this it will probably be last.

[Morganamilo]

So I have come to a bit of an impasse with split packages, namely python-virtualfish/python2-virtualfish.

As far as I know it's impossible to query a pkgbase directly through rpc.
Now the virtualfish pkgbase has the depends depends=('python-virtualenv' 'python2-virtualenv' 'python-setuptools' 'python2-setuptools')
But the python-virtualfish package only has depends=('python-virtualenv' 'python-setuptools')
So when installing python-virtualfish yay will not detect the extra python2 dependencies. causing the build to fail.

pacaur gets around this by calling -s on makepkg so the packages are installed automatically when running makepkg. This works for pacaur but just like Yay the extra python2 dependencies are missed out during the initial listing of dependencies. I'm also assuming this would fail if these extra python2 dependencies were on the aur because makepkg -s would not be able to find them.

Because it is impossible to get the pkgbase dependancies through rpc the only option is to parse the srcinfo which would require us to download the pkgbuild. Normally we list all the packages to be installed before downloading pkgbuilds. But this would require us to download pkgbuilds then list out more packages to be installed which seems clunky to me.

I am very unsure how to go about this.

[maximbaz]

I think it's safe to assume that in _most cases_ people will not abort installation after the see the list of dependencies, they will proceed with installation, and that's why I think it's totally fine to let yay download as many PKGBUILDs as it needs, parse them all and then spit out the correct tree of dependencies.

Like this:

download all PKGBUILDs -> list all dependencies

But not like this:

list some dependencies -> download some PKGBUILDs -> list some more dependencies

Would you agree?

[Morganamilo]

That's what I think would be best but downloading pkgbuilds takes time and there's no way to detect if something is a split package before getting the pkgbuild so we would need to download every one. I was very happy with the speed of dependency parsing and would hate to see that go.

I really want a perfect solution although one does not always exist. :(

[maximbaz]

On a thought of a perfect solution, do you know if it is even remotely possible for us to request / implement the missing feature in AUR's RPC interface?

[Morganamilo]

There's web scraping but that's messy
You could send a patch to aurweb but I think the rpc is purposely minimal to minimise load on the aur. So I don't know if what they would be willing to accept.

[maximbaz]

I'm not sure if this point belongs to this issue or should be filed separately, but here we go:

• [ ] Upgrade with yay -Syu should also support "ask-first" approach, right now it:
  
  
  
  • upgrades all repo packages
  
  
  • asks questions about the 1st AUR package
  
  
  • upgrades the 1st AUR package
  
  
  • asks questions about the 2nd AUR package
  
  
  • upgrades the 2nd AUR package
  
  
  • asks questions about all AUR packages
  
  
  • installs packages one by one, which requires constant attention to confirm sudo password to install packages as they are compiled.
  
  
  • ...
  
  

Expected result:

• Ask all questions
• Upgrade all repo packages (ideally all in 1 transaction)
• Upgrade all AUR packages (ideally all in 1 transaction)

[Morganamilo]

Does it not already do this? -Syu should already do this.

[maximbaz]

That's not correct indeed, I remember that I had to get back to terminal running upgrade process over and over again to confirm something, but I didn't express the details correctly, sorry. I just tested more carefully, here's what actually happens:

1. Run yay -Syu and confirm that you want to upgrade everything, repo and AUR packages.
2. Yay starts a long process of upgrading repo packages - I switch away to other activities.
3. I return to the terminal and see that yay upgraded repo packages and waits on the first question about the first AUR package.
   
   
   
   • this should have been asked before step 2.
   
   
4. I answer all question about all AUR packages and switch away.
5. I return to see that yay compiled only the first package and waits for sudo password to be entered. I enter password and it starts compiling the second package - I switch away.
6. After a while I switch back to the terminal and see that step 5 is repeated, yay is waiting on sudo password yet again.

The easy improvement is to move step 3 before step 2, and the part about installing all AUR packages in one transaction has already been mentioned.

[Morganamilo]

The problem is that Yay is not smart enough to handle multiple providers.
If you try to install libgl through pacman it will ask if you want nvidia-340xx-utils or libglvnd. We could do this through Yay but instead of asking it would just assume the first. Also these two packages might have different dependency trees so it would be impossible to generate a full tree before deciding which to install.

Thats why calls to -S will simply pass pacman -S repo targets first so pacman can handle the questions for us, then we generate a full dependency tree from the aur packages and present that to the user with questions.

The sudo problem you will just have to increase the time it allows you to sudo without a password.

[maximbaz]

But we could also ask questions about AUR packages first, _then_ run pacman -S repo targets and then upgrade AUR packages, right?

Increasing sudo timeout is a possibility, but it's a hack, I know it's not a priority right now but I'll keep dreaming about decreasing the number of pacman transactions and one call to pacman -U 😄

[Morganamilo]

Well we could, but if we were installing libgl yay would assume you want libglvnd and generate the deptree for that. Then if we do pacman -S repodeptree it will install and you would never have known you had a choice. The same goes for groups actually. If we did it like pacman -S repo targets then pacman would still ask the right questions but because we have already generated the deptree before hand if the user chooses anything non default we will have a mismatch.

There plenty of ways around the sudo thing although they all feel hacky. The main two options would be:

• fork a background process that calls sudo : every so often to reset the timer
• fork and run as root the entire time and su back the the original user for makepkg calls

when it comes to minimising transactions the more and more I look into it I think that's never going to happen. Previously i though:

• depends = runtime
• makedepends = maketime

But after looking into it I'm pretty sure it's

• depends = maketime and runtime
• makedepends = maketime only

makepkg refuses to build a package unless all of the depends and makedepends are inatalled.
python2-vitualfish fails to build without python2 installed even though its not listed as a makedepend.

[maximbaz]

Interesting thoughts.

Please correct me if I'm wrong, but I think the question about libglvnd comes only at the time of first installation, it will not come again during upgrades. So we could keep the installation process as it is, but tweak the upgrade process, and only during upgrades we could ask AUR-related questions before upgrading repo packages. Maybe not with high priority, but in general I think it makes sense to optimize upgrade process because on average you upgrade packages more often than you install new packages. What do you think?

Of the two sudo hacks, the second one seems less hacky 🙂

Minimizing transactions doesn't necessarily mean one transaction, it only means using as many transactions as needed, but not more than that. If you can install or upgrade everything in one transaction - awesome! If you need to solve dependencies with 2, 3 or more transactions - so be it.

[Morganamilo]

The question will occur whenever you attempt to install (reinstalls don't count) something that doesn't match a package name but there are packages that provide what you want. libgl does not exist as a package but nvidia-340xx-utils and libglvnd provide libgl. Now sure this would mostly occur during and install but it could happen during an upgrade. When a package gains new dependencies or a package replaces another.

Currently in my branch the flow is like so.

1. install repo targets
2. install aur-repo depends
3. install all aur packages one by one

This could be adjusted to preform less transactions but my main focus right now is just getting a working system that's ready for a release. Do you plan on implementing this yourself at all? You've contributed before so I would assume you're able.

[maximbaz]

I see, thanks, it's good to understand all this complexity. I might contribute some of this, yeah, but probably after your improvements are settled in first. And I want to make sure to align my ideas with what you and @Jguer think is the right direction for yay, for example I want to see what @Jguer wants to do with #42, as he described pretty much the same idea with "tiers" in there.

[AladW]

I'll never understand why people cling to the antiquated pacman -U method. There's a huge deal of stuff you have to reimplement - like the mentioned handling of split packages - which is already provided by pacman itself through local repositories. In the end you'll also stumble on issues with versioned dependencies and will be tempted to resort to pacman -Ud, like pacaur and trizen did. I needn't explain why that's not a desirable thing.

[maximbaz]

The wish is simply to minimize user interaction and number of pacman transactions, repo packages are installed/upgraded in bulk with one question to user and in one transaction, makepkg -i asks too many questions and makes many transactions. What would be an alternative to pacman -U that still solves this issue?

[AladW]

Instead of having a bunch of foreign packages from pacman -U, you'd put those packages in a separate repository (a file:// repository, see man pacman.conf for details or /etc/pacman.conf for an example).

In my experience this avoids questions from makepkg/pacman since packages don't have to remain installed at all times, and can be removed when the desired packages are done building. Put differently, the need for complex logic to "predict" pacman questions like in pacaur is reduced significantly.

Similarly you can build all packages in a split pkgbase, add them to the repo, and have the user easily install the part(s) he wants through pacman -S.

[Morganamilo]

How would that fix anything? if AUR package foo is a dependency of bar then to run makepkg on bar we would require foo to be made and installed anyway. And you said avoid questions? Why would we want to avoid them? they are important things that the user should see and be given the option to answer.

[AladW]

Your first sentence assumes that anything you build from AUR automatically has to be or remain installed on the system. That's not the case. For example, a package may have some special makedepend that can be removed after the build has finished.

That's what's meant by _avoiding_ questions - the scenario where the question would be asked simply does not arise. After all, when a package isn't installed it can't cause conflicts.

The issues with versioned dependencies and split packages remain for pacman -U. See for example https://github.com/trizen/trizen/issues/46 or the highly complex logic pacaur implemented to install split packages individually through pacman -U.

Not to mention many other side benefits you get from using actual repositories, like being to easily share packages over the network or enable the use of pacman -F.

[Morganamilo]

I still don't understand, the question of whether or not to remove makedepens is a valid one I would like to be asked this. Even if you build a pacman repo you still need to build the packages at some point.

[AladW]

If a package is a foreign package, the question whether to remove it is indeed quite valid, since the helper may have not stored the built package persistently (e.g. in /tmp) or in some volatile location (in the AUR git repo). If it's built inside a repo, you can always reinstall it after "removal" by issuing a pacman -S command.

But again, you should look at what happens to split packages and versioned dependencies. Handling those efficiently with foreign packages is painful at best.

[Morganamilo]

Again you have not answered the question. You still have to run makepkg at some point so you would still have to ask. How does the repo method avoid the question?

[Morganamilo]

Also sorry if I'm getting all negative, i get that it has it's merits. I just don't get the point with the questions.

[AladW]

Yes, there are always questions to ask in both the pacman -U and the local repo scenario. What I'm trying to expand on are those questions that ask about _package conflicts_.

In the pacman -U scenario, you likely have all AUR packages you've built installed on the system. More packages installed means more chances that they conflict with _other_ packages. That's why pacaur had an elaborate mechanism in place to compute conflicts/provides, so it could ask those questions before doing anything else.

In the local repo scenario, this is not necessarily the case since any dependencies can be easily retrieved via pacman even if the package is removed. So in total you have less packages _installed_, and thus less chances of conflicts on build.

Does that clear up things up a bit?

[Morganamilo]

But say I'm building an AUR package that pulls in a dependency that conflicts with another installed package. You have a conflict during build time.

[AladW]

Going over https://github.com/Jguer/yay/issues/108#issuecomment-360207256 again:

What happens should AUR targets have make/depends that conflict? Let's say you do yay -S foo bar, where

• foo has aaa as makedepend, and
• bar has bbb.

Let's further say aaa and bbb conflict, but foo and bar _can_ coexist. To make things further complicated, let's assume aaa and bbb are both in AUR.

Installing aaa and bbb one-by-one as in your step 3. would be problematic since either AUR package would be "lost" (as an uninstalled foreign package) and you get a conflict question in the middle of the build.

In this - admittedly extremely convoluted - example, when using a local repo, and assuming a total dependency order, each step would simply be covered by makepkg -sr where aaa and bbb are added to the repo and installed/removed where required.

This approach would work even if foo and bar would conflict with eachother (say, when you want to build both a release and a development version and periodically switch between them)

Of course you would still have an "install this" prompt unless you resorted to --noconfirm, but I would argue --noconfirm is acceptable there since no conflicts occur, by construction. If you think "install this package to make the build run?" questions are equally important as conflict questions, then you need a different approach.

Also a note about dependency resolution: not even the web interface lists make/depends for pkgbase, but getting this information is completely valid both by your virtualfish example and the specification in PKGBUILD(5). I would say getting .SRCINFO files is too slow to be viable, and would definitely suggest to file a feature request in flyspray.

[Morganamilo]

Even in that example you would get mid install questions, you talk about resorting to --noconfirm but pacman will default to n on conflicts and I dont think makepkg supports the --ask option. You could set the PACMAN variable to use a wrapper that calls pacman with the --ask option but that seems convoluted. I get using a repo would have a much cleaner install process but this sacrifices the build process.

Also I want an aur helper that supports all pacman options --config --root --dbonly ect. This wont work with makepkg -si unless you wrap PACMAN to respect the values passed to yay.

Maybe i'll give your aurutils a try at some point to see how it works.

[AladW]

Yes, the idea is to avoid conflicts in the first place so an n is not a problem. At least for me that approach has worked quite well so far.

Regarding --ask, even if makepkg supported it, using it like pacaur does is not a great approach in my opinion, considering the computational cost involved to ensure you confirm the "right" conflicts. (of course you're quite welcome to prove me wrong there)

Anyway I'll keep following this thread and see what you guys come up with. All the best.

edit: https://bugs.archlinux.org/task/57230

[maximbaz]

I tried out aurutils to see how this approach with local repo works, and I must say the experience is amazing! There's definitely something to learn from that project.

First of all, it's all nicely separated into stages, kind of what you are trying to do now for yay:

1. First you review all PKGBUILDs in one go (file viewer was already suggested here, and now I fully support this idea)
2. Then you compile all the necessary packages and their dependencies.
   > Important for me: compilation with aursync -sN <packages> doesn't require user interaction and doesn't even issue sudo calls (with this patch), so you can run it and forget, it won't get stuck on waiting for sudo password.
3. And finally, when all the compiled packages are available in the local repo, you install / upgrade them in one go.
   > Important for me: all the packages are installed/upgraded in _one_ pacman transaction, it's a dream, because now I can use pacman hooks and have _one_ system snapshot before the full upgrade, whether repo or AUR packages are involved - so #95 is definitely not needed.

[eli-schwartz]

I just want to point out regarding things like the virtualfish example, this is arguably a flawed PKGBUILD as it is abusing the global depends=() array for makedepends, then overwriting it in package_*()

The RPC will list all makedepends for a split package, as it is expected to do.

...

That being said, the RPC is obviously somewhat inconsistent and it may well be worth fixing it. But that shouldn't feature into your dependency handling. No AUR helper should be expected to fix the mistakes of every badly-written PKGBUILD. If you do your best, but some PKGBUILD does something against the PKGBUILD(5) spec that messes everything up, then you've done your best...

[osleg]

Just one note to the flow, I think when one package fails to install it should be skipped instead of break the entire build.
tbh that's the main reason I tried to stick with yaourt as long as possible, it was building each package independently and if something failed it was removing the dependencies for failed package

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.