Nix: Get rid of the Perl dependency

[edolstra]

I.e. rewrite nix-build, download-from-binary-cache etc. in C++.

• [x] [scripts/nix-collect-garbage.in](https://github.com/NixOS/nix/pull/522)
• [x] [scripts/find-runtime-roots.pl.in](https://github.com/NixOS/nix/commit/ee3032e4de439b481ce1a4bc7fd8d71ea6ed2c94)
• [x] [scripts/nix-copy-closure.in](https://github.com/NixOS/nix/pull/1023)
• [x] [scripts/nix-prefetch-url.in](https://github.com/NixOS/nix/commit/bec3c3160881fcedf226080f13739aee81adae1a)
• [x] [scripts/nix-install-package.in](https://github.com/NixOS/nix/pull/1021)
• [x] [corepkgs/buildenv.pl](https://github.com/NixOS/nix/pull/1020)
• [x] [scripts/nix-channel.in](https://github.com/NixOS/nix/pull/1026)
• [x] [scripts/build-remote.pl.in](https://github.com/NixOS/nix/pull/981)
• [x] [scripts/nix-build.in](https://github.com/NixOS/nix/pull/1018)
• [x] [scripts/nix-push.in](https://github.com/NixOS/nix/pull/1024)
• [x] [scripts/download-from-binary-cache.pl.in](https://github.com/NixOS/nix/commit/a6eed133c5a3602037bc48675ca783185cca5454)

Darwin:

• [x] [scripts/resolve-system-dependencies.pl.in](https://github.com/NixOS/nix/pull/1030)

[linquize]

:+1: to rewrite in C++

[wmertens]

:+1:!

Just to have an idea of the scope:

$ git grep -l perl scripts/ corepkgs/ | xargs wc -l | sort -n
  17 corepkgs/config.nix.in
  28 corepkgs/buildenv.nix
  51 scripts/nix-generate-patches.in
  65 scripts/nix-collect-garbage.in
  73 scripts/show-duplication.pl
  79 scripts/find-runtime-roots.pl.in
 100 scripts/nix-copy-closure.in
 102 scripts/copy-from-other-stores.pl.in
 102 scripts/nix-pull.in
 132 scripts/nix-prefetch-url.in
 143 scripts/nix-install-package.in
 168 corepkgs/buildenv.pl
 212 scripts/nix-channel.in
 282 scripts/build-remote.pl.in
 288 scripts/nix-build.in
 294 scripts/nix-push.in
 377 scripts/download-using-manifests.pl.in
 612 scripts/download-from-binary-cache.pl.in
3125 total

and

  42 perl//lib/Nix/Config.pm.in
 114 perl//lib/Nix/CopyClosure.pm
  42 perl//lib/Nix/Crypto.pm
 340 perl//lib/Nix/GeneratePatches.pm
 468 perl//lib/Nix/Manifest.pm
 103 perl//lib/Nix/SSH.pm
  92 perl//lib/Nix/Store.pm
 292 perl//lib/Nix/Store.xs
  47 perl//lib/Nix/Utils.pm
  40 perl//local.mk
   7 perl//MANIFEST
1587 total

[domenkozar]

:+1:

[offlinehacker]

+1

[matejc]

:+1:

[hilyjiang]

+1

[domenkozar]

Would also get rid of https://github.com/NixOS/nix/issues/19

[copumpkin]

:+1: obviously. Seems like it wouldn't have to be a monolithic work item. I'm guessing many of the *.pm files are just wrapping functionality that's already in the C++ library, so some of that wouldn't even be necessary anymore. The individual utilities could be replaced one at a time if someone didn't want to take on the entire task at once.

[lethalman]

I'm on nix-collect-garbage :) starting low

[copumpkin]

Yay!

[edolstra]

:beer:

[lethalman]

I'm now on find-runtime-roots.pl

[edolstra]

Rather than replacing it with an external C++ program, please add it directly to gc.cc.

[domenkozar]

I've added a 50$ bounty for this issue at https://www.bountysource.com/issues/4399291-get-rid-of-the-perl-dependency

[bjornfor]

Is there a bounty service that doesn't require sign-up? I'd like to support, but I don't want yet another account on some web service...

[domenkozar]

You can login with github account. But no :)

On Sat, 7 Nov 2015, 10:17 Bjørn Forsman [email protected] wrote:

Is there a bounty service that doesn't require sign-up? I'd like to
support, but I don't want yet another account on some web service...

—
Reply to this email directly or view it on GitHub
https://github.com/NixOS/nix/issues/341#issuecomment-154667256.

[vcunat]

And you can always make promises without any service at all...

[bjornfor]

Ah, I didn't know bountysource was "a promise", I thought it was direct payment.

[vcunat]

Well, maybe, but I think there's an unavoidable tricky point: who decides that the issue is solved well-enough?

[AndersonTorres]

Also, what if multiple people solve it?

[domenkozar]

The one offering the bounty decides how the money should be split.

[vcunat]

Well, nice theoretical discussion, but for all nixos.org stuff I've only seen this single bounty offer of $50 – that probably wouldn't pay more than a single day of work for most people, so overall there seem to be basically no incentive for bounty hunting.

[AndersonTorres]

Well, I will study the sources at Saturday. Maybe I can do something :)

[offlinehacker]

I've added additional 50$ to the bounty, i know the pain involved with cross compiling nix, so i would love to see this solved :)

[garbas]

i've also added additional $50 for this effort ...

[domenkozar]

So here's an idea: let's fund the whole development of this effort. I think it's an important milestone for Nix and I'm willing to invest some of personal finances just to really get it done. It's a tedious job and I really think whoever gets it done deserves a payment.

Proposal: is anyone willing to port Perl code to C++ and being paid for it? Send me an email to [email protected] and let's talk how to make it happen.

[davidak]

@domenkozar great. i also added some money to the bounty. you can pay with bitcoin :)

i also shared it on Diaspora*

[domenkozar]

I'm really excited to announce a (proper!) funding campaign for this issue. Shea Levy stepped up as the implementer, he's familiar with the codebase and currently top 2 contributor.

I'd like to invite everyone that is excited about this work (if not, see the link for why you should be) to donate funds at https://www.gofundme.com/htuafwrg/

[MarcWeber]

I just saw this effort. Let me remind that there are alternatives to C++ such as rust which could be evaluated as well. We all know that Sha is very capable of many things. Thus an interesting idea would be to do ask another question: If there were $3000, would there be other tasks as well? If I had $3000 I'd spend them on a proper open source versioning database from which many .nix derivations could be derived. Something like hackage but for C/C++ and all the other small items. Because if you do that, more devs might join in the long run.

[domenkozar]

@MarcWeber I agree - but funding this was initiated because doing such tedious rewriting work is not exciting, but it's important for Nix to prosper. Scope of porting Perl parts to C++ is smaller compared to introducing Rust, so that's why we're going after the former. Baby steps.

[copumpkin]

Yeah, rewriting the whole thing in rust seems far more expensive (time and money). It might be more appealing for people working on it in their free time, but in the meantime we have a problem that can be solved fairly easily and quickly.

Rust also adds additional compilation requirements on environments we want to bootstrap and compile Nix on, whereas a c++ compiler is pretty much ubiquitous.

I personally dislike c++ but still think this is a pretty valuable thing to do (which is why I overcame my dislike for the language threw money behind it anyway) independent of any other concerns.

[AndersonTorres]

Also, we can just do a parallel working for Rust - even opening a new github page for it. If someone wants to do it, it can be done in an unnofficial way.

[domenkozar]

@offlinehacker @davidak did you donate to the campaign? We'll cancel the bounty :)

[davidak]

@domenkozar like i said on twitter, i can't because only credit cards are accepted.

https://twitter.com/davidak/status/686518703085727745

i would like to pay with Bitcoin or SEPA to have no fees.

[domenkozar]

@davidak that's fine, I'll make sure we also catch your contribution on bountysource :)

[vcunat]

SEPA is only possible in Euros and within EU, AFAIK. I believe that's not useful for Shea in particular. (Maybe a promise of sending bitcoins is enough, but I can't speak for him.)

[davidak]

@vcunat Yes, SEPA is wire transfer within the EU. He could just claim the bounty at Bountysource and withdraw it with "Paypal, Bitcoin, or physical check". If that count official to the needed amount i may donate some more. If @shlevy would accept i can also send Bitcoins directly to lower the fees.

[garbas]

I've also added $50 to bountysource. you can also count on that (apart from the
money i've donated).

[shlevy]

@davidak What's the fee from bountysource to get a check? If it's in the same range as gofundme's, I'm happy to count bountysource toward the total.

[vcunat]

@shlevy: 10% on withdrawal.

[copumpkin]

Wow, that seems pretty brutal. Makes me not want to use them anymore.
On Tue, Jan 12, 2016 at 10:54 Vladimír Čunát [email protected]
wrote:

@shlevy https://github.com/shlevy: 10% on withdrawal
https://www.bountysource.com/fees.

—
Reply to this email directly or view it on GitHub
https://github.com/NixOS/nix/issues/341#issuecomment-170955399.

[copumpkin]

I guess gofundme ends up being pretty expensive too, after WePay fees. I hate the payments industry.

On Jan 12, 2016, at 10:54, Vladimír Čunát [email protected] wrote:

@shlevy: 10% on withdrawal.

—
Reply to this email directly or view it on GitHub.

[shlevy]

Ugh, I thought gofundme was 5% but I didn't see the wepay cut. I'll start the work when the total after all fees is around $3220 (i.e. as if gofundme hit the campaign goal)

[davidak]

@copumpkin i thought the same! SEPA is free and Bitcoin has technical a very low fee. (you can pay nothing but the transfer takes more time) And bitcoin.de as a marketplace has 1% fee divided between the 2 trading partners (->0,5%). Paypal has 3% fee what is also very expensive i think. So i try to use Bitcoin for most payments.

[domenkozar]

We're at $2220 out of $3500 at https://www.gofundme.com/htuafwrg/

[bjornfor]

Thanks for the reminder. (Now it's at $2320.)

[domenkozar]

We're at $3220. Only $280 left! https://www.gofundme.com/htuafwrg/

[shlevy]

We're funded! I'll get started soon, hopefully this weekend!

[copumpkin]

Amazing!

[aristidb]

What's the current status of this?

[shlevy]

Mea culpa, I've seriously let this slip! Working on build-remote now, will keep updated as I go.

[domenkozar]

@shlevy I think it's best to open a PR per script to get the process going asap and to have it split into minimalistic PRs

[shlevy]

Update: Mostly through build-remote, I had a sudden job opportunity starting the end of April and am planning and presenting at a conference at the end of May, so I won't be able to do more on this until June. I will update again June 5, hopefully with a PR or two

[domenkozar]

Awesome :)

[domenkozar]

I'm really excited about this work, I just went through our issues tagging those that are related to install and this should really reduce the number of errors folks are getting! :)

[shlevy]

OK, back on a normal schedule! Should have 1-3 hours a day for the next week, will update again on Wednesday evening with status. Currently have tests and the scaffolding of build-remote done, as well as about 70% of the logic.

[domenkozar]

@shlevy that's a 10% removal, baby steps :) Keep it going!

[domenkozar]

@shlevy also don't forget about https://github.com/NixOS/nix/issues/533

[domenkozar]

@shlevy ping :)

[shlevy]

WIP at https://github.com/shlevy/nix/tree/build-remote-c%2B%2B

[shlevy]

WIP updated and rebased on master

[shlevy]

Updated with implementation of ssh connection.

[shlevy]

PR #981 open

[shlevy]

PR #985 open

[domenkozar]

@shlevy to spare you some time:

scripts/nix-install-package.in is deprecated https://github.com/NixOS/nix/issues/831

[shlevy]

SSH store (to replace nix-copy-closure with nix copy as well as improve build-remote and bring back ssh substitutes) WIP at https://github.com/shlevy/nix/tree/ssh-store

[shlevy]

Update: still working through nix-shell/nix-build conversion.

[shlevy]

nix-build/nix-shell at #1018

[shlevy]

1020 for buildenv

[shlevy]

Nuked nix-install-package in #1021

[domenkozar]

@shlevy I've changed the description of this issue to reflect current status. Feel free to modify, for example I don't think scripts/show-duplication.pl is needed at runtime, scripts/nix-generate-patches.in is part of manifests also to be removed, etc.

[shlevy]

@domenkozar How old is your nix checkout? :tongue: nix-generate-patches was removed in April

[domenkozar]

Ah, I was copying @wmertens output. Removed :)

[domenkozar]

@shlevy wait, only nix-channel and nix-push left?!

EDIT: also nix-copy-closure.

$ wc -l scripts/{nix-push,nix-channel,nix-copy-closure}.in
  296 scripts/nix-push.in
  228 scripts/nix-channel.in
  103 scripts/nix-copy-closure.in
  627 total

[shlevy]

@domenkozar Too fast for me #1022

[edolstra]

We can drop nix-push (since probably very few people use it and nix copy provides largely the same functionality), but we definitely need to keep nix-copy-closure for backwards compatibility.

[shlevy]

@edolstra OK

[shlevy]

OK nix-copy-closure migrated in #1023

[shlevy]

nix-push removed in #1024

[knedlsepp]

@pikajude: Am I correct in assuming that scripts/resolve-system-dependencies.pl.in is only relevant in the case of a chrooted build? It seems those don't work properly anyway: https://github.com/NixOS/nix/issues/759

[domenkozar]

nix-channel port away from perl free Nix? :dancers:

[shlevy]

nix-channel in #1026

[davidak]

@shlevy Thank you! I just added another $50 to the bounty.

https://www.bountysource.com/issues/4399291-get-rid-of-the-perl-dependency

Don't forget to claim it when it is complete.

[shlevy]

I consider the core work finished (if all PRs are merged and a simple final Makefile tweak added on top, nix on Linux has no dependency on perl) and the bounty is linked to this github issue. Any objections to me closing the issue to claim the bounty, and using the perl-to-c++ milestone to track the remaining bits?

[domenkozar]

@shlevy sure, if PRs are merged and Nix can compile without perl, feel free to claim this :)

[vcunat]

I understood it as a suggestion to close/claim _before_ merging the PRs.

[davidak]

@vcunat No, the issue is solved for me when every change is merged in master. (and properly works together)

[shlevy]

OK, thought I'd check :smile:

[shlevy]

1027

[domenkozar]

@shlevy I'd say you have done your part of the work so it's fine if you collect the bounty.

[shlevy]

@domenkozar Do I have to do anything besides just close the issue?

[domenkozar]

I think you have to claim bounty on bountysource manually.

[shlevy]

From https://github.com/bountysource/core/wiki/Frequently-Asked-Questions it looks like the only way is to close the issue, maybe even the only way is to merge a commit that closes the issue. I will try closing this, let me know if you get the confirmation email.

[domenkozar]

I needed to refresh about 6 times before you showed up as a claimer on https://www.bountysource.com/issues/4399291-get-rid-of-the-perl-dependency, but it's accepted from me.

I'll try to write up a summary for all chip-in-ers :)

[shlevy]

Did my summary not show up?

[domenkozar]

Nope