Nix: Multi-connection download of single substitute

[puffnfresh]

Sometimes when downloading binary substitutes I'll only get downloads of around 80kbps. This is not so bad when I have at least 25 substitutes to download, because Nix includes parallel downloading of substitutes.

It's not very good when I have a single substitute of many megabytes that I need to download. I can be waiting for a long time.

I have taken a substitute URL and downloaded it via aria2 which makes it pretty much saturate my connection, given enough concurrent requests. It's tricky to get that substitute into the store once I download it. I would instead just like to replace the substituter that Nix uses.

Is it possible to override the Nix curl binary substituter with my own?

[edolstra]

No, that's not possible. In fact, the Perl-based substituter is gone, replaced by a C++ implementation that's pretty closely tied to libcurl.

[puffnfresh]

If I use curl to download a 7mb package, it takes a minute:

$ time curl https://cache.nixos.org/nar/1bvgs0rigij7arzscc0hn8nsqm988slvyz94s09lkhlcywlmyl0g.nar.xz > /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 7986k  100 7986k    0     0   131k      0  0:01:00  0:01:00 --:--:--  136k
curl  > /dev/null  0.13s user 0.09s system 0% cpu 1:00.80 total

But switching over to aria2c gets it down to 13 seconds:

$ time aria2c https://cache.nixos.org/nar/1bvgs0rigij7arzscc0hn8nsqm988slvyz94s09lkhlcywlmyl0g.nar.xz

12/24 13:08:09 [NOTICE] Downloading 1 items
[#bb9c9d 6.0MiB/7.7MiB(78%) CN:1 DL:634KiB ETA:2s]                                                                               
12/24 13:08:22 [NOTICE] Download complete: /tmp/1bvgs0rigij7arzscc0hn8nsqm988slvyz94s09lkhlcywlmyl0g.nar.xz

Download Results:
gid   |stat|avg speed  |path/URI
======+====+===========+=======================================================
bb9c9d|OK  |   694KiB/s|/tmp/1bvgs0rigij7arzscc0hn8nsqm988slvyz94s09lkhlcywlmyl0g.nar.xz

Status Legend:
(OK):download completed.
aria2c   0.12s user 0.10s system 1% cpu 13.204 total

That's even without parallel connections. I'm not sure why aria2c is so much faster, but I'd love to be able to use it.

I can find the URL for a package with some SQL:

SELECT url FROM NARs WHERE storePath = '00000000000000000000000000000000-ghc-8.0.2';
-- nar/0000000000000000000000000000000000000000000000000000.nar.xz

Then I can download via aria2c which seems to consistently give a 4.5x speedup.

But if I've downloaded a cached package, how can get it into my store? Doing a nix-store --restore gives me "Read-only file system" - so I'm not sure if it has the right metadata.

[domenkozar]

@puffnfresh double check you're not hitting https://github.com/NixOS/nix/issues/1128

[puffnfresh]

@domenkozar yes, thank you! I am hitting that:

$ time curl --http1.1 https://cache.nixos.org/nar/1bvgs0rigij7arzscc0hn8nsqm988slvyz94s09lkhlcywlmyl0g.nar.xz > /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 7986k  100 7986k    0     0   833k      0  0:00:09  0:00:09 --:--:--  982k
curl --http1.1  > /dev/null  0.06s user 0.07s system 1% cpu 9.585 total

Both of my main development systems are on the 16.09 channel but have this problem. I'll try a nixos-rebuild.

[domenkozar]

Make sure you upgrade to latest channel :)

[puffnfresh]

@domenkozar I have to get off stable? Is this possible to backport at all?

[domenkozar]

No, updating to latest stable should also fix your issues.

[puffnfresh]

I have updated but I still have the same problem:

$ sudo nix-channel --list
nixos https://nixos.org/channels/nixos-16.09
$ sudo nixos-rebuild switch --upgrade
$ reboot

[domenkozar]

It wasn't backported, I was just assuming that.

@puffnfresh backported now: https://github.com/NixOS/nixpkgs/pull/20639#issuecomment-269497956 please give a few days for channel to update

[puffnfresh]

@domenkozar I've got the backport and it's working great. Getting speeds of 3123k! :fireworks:

[nh2]

Mind to reopen this feature request? I have some machines that would significantly benefit from parallel downloading.

[vcunat]

Until someone is actually inclined to work on the feature, I see little use for an open issue.

[nh2]

@vcunat Tracking missing desired features in closed issues is ... a novel and interesting approach :)

[vcunat]

I have experience from the nixpkgs repo. Such issues are typically not picked up by anyone, lying around forever (months, years). The issue counts steadily increase, and the more important ones get more easily drowned in hundreds of old feature requests that noone is interested enough to work on. Some members even close such long-dead issues (unsolved).

[tomberek]

Leaving a note that I've run into this issue with a large ML model. My hack workaround has been to split the file into 52 multi-outputs and reconstitute them into a fifo when loading. This greatly speeds up the download to saturate the network and provides parallel decompression.