Yay: Feature request: Download concurrency

[jaythomas]

It would be pretty great to have packages download in parallel like they do with the powerpill package manager. Having fiber internet and a fairly modern computer I often find my bottleneck during system updates and large package installs is the single-threaded downloads. Not sure if yay can wrap powerpill, but even if it does it seems self-defeating to rely on another package manager.

I'm not a guru with golang but happy to help where I can.

[Morganamilo]

Yay's pkgbuild downloading is already in parallel. Powerpill can be used for repo packages. yay --pacman powerpill -Syu. Not sure what would be self defeating about it.

[jaythomas]

With the following commands I've tested for downloaded concurrency:

yay -S libreoffice-fresh
yay -S --downloadonly libreoffice-fresh

But following the output, the packages only download in sequence. Is there an additional argument or config option I am missing?

For what it's worth, I mean it's self-defeating to depend on powerpill as one of the nice things about yay is it has no additional dependencies. If I had to install powerpill and its python dependencies I'd just use powerpill directly.

[Morganamilo]

But following the output, the packages only download in sequence. Is there an additional argument or config option I am missing?

That's a repo package. Like I said, our concurrency applies to pkgbuilds.

For what it's worth, I mean it's self-defeating to depend on powerpill as one of the nice things about yay is it has no additional dependencies. If I had to install powerpill and its python dependencies I'd just use powerpill directly.

You have a point but powerpill is a long standing and proven tool. There would be no point to make a clone and duplicate all that code.

[joeky888]

If I had to install powerpill and its python dependencies

You can use pacman -Sup to print url, and then use aria2 to download it, just like powerpill.

I wrote a script called apt-fast on Ubuntu/Debian and just ported it to Archlinux, the only dependency is aria2.

Usage on Ubuntu/Debian: apt-fast install vlc -y
Usage on Archlinux: apt-fast -S vlc

Save it to somewhere like /usr/bin/apt-fast

#!/bin/bash

Edit: see below.

[Morganamilo]

But what if I'm on arch and have apt-get installed ;)

[joeky888]

Is that possible? :O

Show me some screenshots.

[Morganamilo]

https://aur.archlinux.org/packages/apt/

[joeky888]

Quick fix, need more tests:

#!/bin/bash

[ "`whoami`" = root ] || exec sudo "$0" "$@"

DOWNLOADARGS='-c -s16 -k1M -x16'
DNS=""

if ping -c 1 -W 2 8.8.8.8 2>/dev/null | grep -q -i "ttl"; then
  DNS="--async-dns-server=8.8.8.8"
fi

if echo "$@" | grep -q "upgrade\|install\|dist-upgrade\|full-upgrade\|build-dep"; then
  APT=$([ $(command -v apt) ] && echo "apt" || echo "apt-get")
  cd /var/cache/apt/archives/

  # Download unfinished jobs
  [ $(command -v aria2c) ] || $APT install -y aria2
  [[ -f /var/cache/apt/archives/apt-fast.list ]] && aria2c $DOWNLOADARGS $DNS -i apt-fast.list

  # Create new jobs
  $APT -y --print-uris $@ | grep -E -o -e "(ht|f)tp[s]?://[^\']+" | tee apt-fast.list && aria2c -c $DOWNLOADARGS $DNS -i apt-fast.list

  echo $APT $@
  $APT $@ && rm -f /var/cache/apt/archives/apt-fast.list
elif echo "$@" | grep -q "update"; then
  APT=$([ $(command -v apt) ] && echo "apt" || echo "apt-get")
  $APT -o 'Acquire::Queue-mode=access; APT::Acquire=Retries 3;' update

elif echo "$@" | grep -q "\-S"; then
  cd /var/cache/pacman/pkg/

  # Download unfinished jobs
  [ $(command -v aria2c) ] || pacman -S --noconfirm --needed aria2
  [[ -f /var/cache/pacman/pkg/apt-fast.list ]] && aria2c $DOWNLOADARGS $DNS -i apt-fast.list

  # Create new jobs
  pacman -Syup --print-format "%l" | grep -E -o -e "(ht|f)tp[s]?://[^\']+" | tee apt-fast.list && aria2c -c $DOWNLOADARGS $DNS -i apt-fast.list
  pacman $@ --print --print-format "%l" | grep -E -o -e "(ht|f)tp[s]?://[^\']+" | tee apt-fast.list && aria2c -c $DOWNLOADARGS $DNS -i apt-fast.list

  echo "pacman $@"
  pacman $@ && rm -f /var/cache/pacman/pkg/apt-fast.list
fi

[joeky888]

Oops, update regex.

The previous one will not download packages which contain a minus(-) character.

apt-fast -S bind-tools # Fails!

[Morganamilo]

I don't want to turn this into a thread about your script but maybe you should read about --print-format.

[joeky888]

Thank you for all of your help above, I wrote this just for funsies. :)

I started it because powerpill is broken in the past few weeks, and I still don't know how to fix it.

read about --print-format

Done.

[jaythomas]

Just tried it out, very nice script. Consider moving it to a gist or repo in case you need to make future revisions. Gists support revisions and would give these discussions an appropriate home.

[joeky888]

Sure, I am very new to Archlinux (actually Manjaro).

Give me some time to test it before I release it on Github.

And if I don't, the code above is Public domain, do whatever you want lol.

[joeky888]

Here it is! https://github.com/joeky888/app-fast

[HaleTom]

My solution was:

Have pacmatic wrap yay wrap powerpill wrap pacman

@joeky888 what advantages does your script have over yay --pacman powerpill?

[joeky888]

Powerpill requires python3 and lots of dependencies, which I don't need in docker environment.

Also it was broken for weeks, I can't get it work so I made another one.

Basically doing the same thing, but in bash/zsh.

[HaleTom]

@joeky888, sorry I missed that. Thanks for reiterating.

I came here because of the same breakage. The powerpill 2009 versioned package now no longer has a python issue, but I do see the advantage of disconnecting from python altogether.

[apoorvaanand1998]

My solution was:

Have pacmatic wrap yay wrap powerpill wrap pacman

@joeky888 what advantages does your script have over yay --pacman powerpill?

@HaleTom, hey sorry if this isn't the place to be asking this but when I trying to do this, I got the following error:

Traceback (most recent call last):
  File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/lib/python3.7/site-packages/Powerpill.py", line 43, in <module>
    import pm2ml
  File "/usr/lib/python3.7/site-packages/pm2ml.py", line 38, in <module>
    import XCPF.PacmanConfig
  File "/usr/lib/python3.7/site-packages/XCPF/PacmanConfig.py", line 28, in <module>
    from pycman.config import pacman_conf_enumerator, _logmask, cb_log, LIST_OPTIONS, BOOLEAN_OPTIONS
ImportError: cannot import name 'pacman_conf_enumerator' from 'pycman.config' (/usr/lib/python3.7/site-packages/pycman/config.py)

This error is probably because of powerpill. Any idea how to fix it?

[joeky888]

Any idea how to fix it?

This is a known issue with powerpill.

See comments here https://aur.archlinux.org/packages/powerpill/

[apoorvaanand1998]

I made that latest comment lol

As I said in the comment there, supposedly a patch has been integrated to fix it, but I still get that problem

What do you suggest I do?

[gtors]

I fix it localy by patching /usr/lib/python3.7/site-packages/XCPF/PacmanConfig.py

Before:

from pycman.config import pacman_conf_enumerator, _logmask, cb_log, LIST_OPTIONS, BOOLEAN_OPTIONS

# ... other code

  def load_from_file(self, filename):
    for section, key, value in pacman_conf_enumerator(filename):

After:

from pycman.config import PacmanConfEnumeratorSession, _logmask, cb_log, LIST_OPTIONS, BOOLEAN_OPTIONS

# ... other code

  def load_from_file(self, filename):
    with PacmanConfEnumeratorSession(filename) as pacman_conf_enumerator:
      for section, key, value in pacman_conf_enumerator():

The main difference is that pacman_conf_enumerator replaced with PacmanConfEnumeratorSession