Nixpkgs: How to build a mirror of cache.nixos.org or anything else to speed up nixos-install?

[OlingCat]

Issue description

The downloading speed of nix in my country is too slow, about 5~50KB/s. Is there a way to speed it up?
eg. build a mirror, use multi-connection download tools, etc.

[edolstra]

What country?

Here is some info on how to set up a mirror: https://nixos.org/nix-dev/2016-October/022029.html

[OlingCat]

Well, in China, there has a GFW= =||

[vcunat]

@OlingCat: you may try https://cache.nixos.community instead of https://cache.nixos.org . It's only experimental service located in Europe for now, built on WIP around http://github.com/NixIPFS (note: using IPFS for mirroring was dropped for now, as it seems not suitable yet) I hope we will be able to make a better usable community CDN during 2018.

[dramforever]

Is it possible to rsync the two cache repos? The two kind mirror providers (see referenced issues) say that if they could rsync then they might be able to provide mirroring.

Also is it possible to clone the s3 bucket? The second provider listed below says that they would be able to rclone to make a mirror.

1. https://github.com/tuna/issues/issues/323
2. https://github.com/ustclug/mirrorrequest/issues/165

[shankerwangmiao]

@edolstra I'm quite confused with the command nix mentioned in your post about setting up a mirror. After a successful installation of nix using curl https://nixos.org/nix/install | sh, I ended up with several commands beginning with nix- such as nix-build, nix-daemon and there was no such executable in the name nix. I cannot find the command in your docker [image] (https://hub.docker.com/r/nixos/nix/), either. Furthermore, there seems no introduction for the nix command in your [manual] (https://nixos.org/nix/manual/).

I wonder if nix is an internal tool or I am missing something. Where can I find the nix command？

[vcunat]

@shankerwangmiao: the nix command is only present in/since nix-1.12 which doesn't have a stable release yet.

[pallavagarwal07]

@edolstra
We at IIT Kanpur (institute) also want to set up a local mirror. I tried the nix copy command but it seems to download packages sequentially one by one (and thus is extremely slow). Could you advise on what I might be doing wrong?

Note that our mirror would be public, so probably would help other people too :)

[vcunat]

@pallavagarwal07: it uses the job-level parallelism, not distinguishing between pure downloads and other jobs. See build-max-jobs in the manual.

[pallavagarwal07]

@vcunat
I have passed --option max-jobs 40 on the command line.
I think the issue may be that I am using xargs to ensure that I don't pass too many paths on the command line (argument list too long error). Is there a way to read the paths to be copied from a file, so as to pass them all at once?

[vcunat]

@pallavagarwal07: hmm, that should be all OK. You use https://cache.nixos.org right? (https to get http/2 IIRC)

[pallavagarwal07]

I am using HTTP (I had made a docker without SSL certs :P )
Trying to switch to https, why should that create a difference though?

[edolstra]

nix copy is not affected by max-jobs. It does use a thread pool but the number of threads is not configurable.

[pallavagarwal07]

@edolstra
Then is there a preferable (faster) way to clone cache.nixos.org?
If I know the paths from the file store-paths.xz, can I determine which files I need to download (and thus write my own scripts instead of relying on nix copy?)

[pallavagarwal07]

Okay, for future reference, I think I managed to get it to work as follows:

paths="$(curl -L https://d3g5gsiof5omrk.cloudfront.net/nixos/unstable/nixos-18.03pre127302.e860b651d6e/store-paths.xz | xz -d)"                              
echo "$paths" | xargs -L 200 -P 5 nix-store --max-jobs 20 --realise  

nix-store honors max-jobs, so the download is pretty fast now. I just hope serving via nix-serve is fast enough to be used as a mirror. realise doesn't download packages already present, so should be trivial to update the packages.

[dramforever]

@pallavagarwal07 Maybe check out this? https://nixos.org/nix/manual/#binary-cache-format-and-operation

It says how binary caches work. I think you can just download the .narinfo and the corresponding archive and put them together or something...

[pallavagarwal07]

@dramforever Thank you!! That's exactly what I needed to know. I don't know how I missed it, I must've read the manual at least 5 times.
Now I can create a mirror script without using any nix tools (I think).

[vcunat]

BTW, there's some set of mirror scripts already (preliminary but tested) https://github.com/NixIPFS/nixipfs-scripts/tree/master/nixipfs (Don't get confused by "ipfs" in the name; that technology turned out not to be suitable yet.)

[brainrape]

Can we close this? Seems like it has been answered to a sufficient degree, and there has been no activity for 6mo.

[vcunat]

Well, I suppose.

One way is to use an http(s) proxy, though you won't get speed-up on the _first_ download. For example, last month I tried free CloudFlare's, but it's hacky as they assume you will redirect public DNS to their nameservers – still, it worked okay over plain http when I overrode DNS locally.

I expect I can divulge that over the long term you may expect changes around the CDN for cache.nixos.org, and I believe that will result in more consistent performance all over the world.

[vcunat]

Perhaps now you might have much better performance in China (and elsewhere): https://groups.google.com/forum/#!topic/nix-devel/S6kx0d2_b3Y

[Lunaticzy]

唉~

[dramforever]

(TL;DR: We are thinking of mirroring channels and cache.nixos.org again because of renewed interest. Please review this gist on how we are planning it: https://gist.github.com/dramforever/d2ff99318c70f44149db6070a87da5a0)

There has been some renewed interest in a mirror in the Chinese Nix community. (Even since the switch to Fastly, cache.nixos.org is still not ideal (still much slower than other distros), and the improvement seems have occurred only in random regions, probably depending on the ISP.)

We have created a mirroring script and a supporting environment, and currently TUNA and USTCLUG, hosters of two prominent open source software mirror sites in China (TUNA's mirror, USTC's mirror) are interested in running such a mirror for Nix packages. (TUNA is having other issues at the moment and is unable to do so in the short term.)

We are ready for some initial testing. But we would like to request official review from the NixOS team before we do so. Mirroring entire channels is no small task and we hope that communication will help it to go smoothly. Also, we just might be doing something wrong, and we really hope someone knowledgeable can point it out.

Information regarding how we are planning to mirror, a few links, and other things can be found in this gist, and we will keep it up to date. We request a review of what will be doing in order to minimize risk of mistakes and surprises. Also, please tell us if you need additional information about our mirrors before we start.

Thanks for creating the amazing Nix ecosystem, and thanks in advance for your assistance.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/requesting-review-before-starting-nixos-mirror-in-china/5514/1

[edolstra]

Looks good to me. However, you may want to avoid using nix copy because it recompresses NARs, so it will be quite CPU-intensive when you're mirroring entire NixOS releases. The alternative is to mirror .narinfo files and the .nar files they point to directly. You can get the .nar from the .narinfo, e.g.

# nix path-info --store https://cache.nixos.org --json /nix/store/jcv7h31hb0rvz79il6q4lghs7f7qcqb6-coreutils-8.31 | jq -r .[].url
nar/08m23594iql4bla3lklag0f9bhjwn10jps7nrv6b3rcya5kj77ql.nar.xz

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/announcing-tuna-nix-mirror/6144/2