Nixpkgs: [Suggestion] State of the Erlang ecosystem in NixOS

[k32]

Issue description

Background: I professionally develop and operate some large-scale Erlang services and I am investigating NixOS as a potential platform, and here's my thoughts. (Not saying this to brag, but rather to share an outsider's impression)

1) NixOS ships a patched version of rebar3 with "hermetic" patches and a vanilla rebar3 called rebar3-open. This behavior is confusing for anyone with Erlang background who wants to use rebar3 for local development. I would propose renaming rebar3 to rebar3-hermetic and making rebar3-open the default package. This change is backward-compatible, as (most) nix packages use rebar3 via a function wrapper rather than directly. Moreover, as nix-build runs with network isolation, I don't see much reason in keeping this patch at all: rebar3 won't be able to fetch any packages anyway.

tl;dr: Shipping a package with incompatible patches and not renaming it is wrong. Proposal: remove rebar3 hermetic patch

2) Nix build system tries to reinvent Erlang dependency handling. rebar3 get-deps function reproducibly downloads a fixed set packages with certain hashes validated against the release lock file. There is no reason in making nix wrappers for the packages for two reasons:

• Maintaining a mirror of hex repository index in semi-manual mode requires too much community effort, and therefore this approach is not sustainable, as witnessed by the last patch to hex-packages.nix from 2017 and lots of broken packages (e.g. try installing something as simple as cowboy). Integrating rebar3 get-deps into Nix will fix many problems.

• Last but not least: this goes against OTP design principles [1]. Killer feature of Erlang is hot bytecode patching. Most applications achieve it by packaging the bytecode in OTP releases [2], so they can benefit from Erlang's standard release management system. Note that building OTP releases can be made reproducible, so it doesn't go against Nix packaging principles. But the way NixOS packages Erlang bytecode today is not compatible with the OTP design principles and this is a deal-breaker. Going away from Nix-packaged Erlang applications to Nix-packaged Erlang releases + some nix-specific wrappers for transitioning a running system from one state to another has tremendous potential. But it requires rather fundamental changes in the way packaging happens.

  tl;dr: Hot code loading is a unique, yet extremely tricky feature and everyone should stick to the procedures recommended by the OTP team. It means treating Erlang releases as opaque blobs rather than trying to manage their moving parts separately.

Conclusion: I can put time and effort into maintaining the Erlang ecosystem in Nix, or at least try to investigate the live upgrade and reproducible dependency handling, but I want to have some confirmation from the current maintainers that moving closer to the OTP design principles is not going to be flat-out rejected on ideology grounds.

Steps to reproduce

N/A

Technical details

N/A

[1] http://erlang.org/doc/design_principles/users_guide.html
[2] http://erlang.org/doc/design_principles/release_structure.html#release-concept

[hedning]

ping erlang maintainers: @gleber @couchemar @sjmackenzie @the-kenny
https://github.com/NixOS/nixpkgs/blob/497e6f1705107a7d60e400e3b6dd6df5ca8bcdba/pkgs/development/interpreters/erlang/generic-builder.nix#L126

[sjmackenzie]

If you can do it, go for it. I'm currently working on something similar whereby the dynamic OTP like system calls nix to change its state. To me this is the correct way to do it.

[k32]

First step: #54115

It eliminates a major pain point of having an up-to-date hex registry snapshot that can't be used anyway.

Next step: investigate if it's possible to create an escript parsing rebar lock files and automagically turning them into derivations

[ankhers]

I have been meaning to write a thing to parse rebar and mix lock files for a while to get rid of the hex snapshot. I can get working through that.

[ankhers]

Just to make sure I understand your proposal here. #54115 is temporary for the new fetch-rebar-deps, and you want to eventually have a foo2nix tool that will read the rebar.lock file and generate a nix file for the build, correct?

[k32]

Yes

[ankhers]

Awesome. I can get behind this.

[ankhers]

I have started ankhers/rebar32nix which will be the escript to parse rebar3's lock file and generate a nix file for your project. I have also split out the actual building of the nix expression to ankhers/beam2nix so that there is a common library that can be taken advantage of when I build the equivalent for Elixir.

[k32]

Awesome. I will drop my own tool then. Here https://github.com/NixOS/nixpkgs/pull/54115#issuecomment-460096334 I gathered some useful information about hex hashes, it may help.

There is an extremely cursed way of using hex hashes to fetch hex packages:

1) Create an intermediate fixed output derivation A downloading tar from hex
2) Unpack it
3) cat VERSION metadata.config contents.tar.gz > $out
Hash of this derivation will match with the one from lock file, but the contents are unusable
4) Create another derivation that drops everything from A up to gzip magic bytes (1f 8b) yielding a valid tarball

[ankhers]

Since we have the name of the lib and the version, I think we can just shell out to nix-prefetch-url and parse the output to recieve the required sha. I haven't quite gotten to the point of pulling in deps yet, but that is the path I plan on taking. I saw the same approach taken in the carnix tool (rust's cargo2nix tool).

If that doesn't work, I will dig through what you have started there.

[k32]

So with your approach some tool will return hashes of the release dependencies that can be copy-pasted to the final derivation? I guess it will work.

[ankhers]

@k32 I have a simple working version of my rebar32nix tool. I just wanted some clarification from you. In rebar3-release.nix, the checkouts are meant to be the dependencies of the application, right? As in the packages from hex? If so, I had to make a modification to the handling of it and wanted your thoughts. Below is the change

diff --git a/pkgs/development/beam-modules/rebar3-release.nix b/pkgs/developmdiff --git a/pkgs/development/beam-modules/rebar3-release.nix b/pkgs/developm
ent/beam-modules/rebar3-release.nix
index 837d0ccf15c..37e616f691c 100644
--- a/pkgs/development/beam-modules/rebar3-release.nix
+++ b/pkgs/development/beam-modules/rebar3-release.nix
@@ -48,7 +48,15 @@ let
     configurePhase = ''
       runHook preConfigure
       ${if checkouts != null then
-          ''cp --no-preserve=all -R ${checkouts}/_checkouts .''
+          ''
+          mkdir _checkouts
+          for checkout in ${toString checkouts}; do
+            IFS='-' read -ra CHECKOUT <<< $checkout
+
+            cp --no-preserve=all -R $checkout _checkouts/''${CHECKOUT[-2]}
+          done
+          ''
         else
           ''''}
       runHook postConfigure

This places each of the dependencies into the _checkouts directory so that rebar3 can pick them up when attempting to compile. The version you had failed when passing in a list, which I assume _checkouts is supposed to be.

And for what it's worth, this is the default.nix file my tool spits out for rebar3's repository.

# Generated by rebar32nix 0.1.0: rebar32nix --escript
{ pkgs ? import <nixpkgs> { } }:

with pkgs;

let

  rebar_git = { rebar3Relx }:
    rebar3Relx {
      name = "rebar";
      version = "git";
      src = ./.;
      checkouts = [
        bbmustache_1_6_0
        certifi_2_3_1
        cf_0_2_2
        cth_readable_1_4_3
        erlware_commons_1_3_1
        eunit_formatters_0_5_0
        getopt_1_0_1
        hex_core_0_4_0
        parse_trans_3_3_0
        providers_1_7_0
        relx_3_28_0
        ssl_verify_fun_1_1_3
      ];
      releaseType = "escript";
    };

  bbmustache_1_6_0 = fetchHex {
    pkg = "bbmustache";
    version = "1.6.0";
    sha256 = "1pqm40rs0r13s6r31016614scb2dnd0ibjcq7bh7p98jclljvq2k";
  };

  certifi_2_3_1 = fetchHex {
    pkg = "certifi";
    version = "2.3.1";
    sha256 = "0s3xryibg2wgqhn6nv8yigz3cf76jw02pbjbb4qd249c0iyncbg1";
  };

  cf_0_2_2 = fetchHex {
    pkg = "cf";
    version = "0.2.2";
    sha256 = "08cvy7skn5d2k4manlx5k3anqgjdvajjhc5jwxbaszxw34q3na28";
  };

  cth_readable_1_4_3 = fetchHex {
    pkg = "cth_readable";
    version = "1.4.3";
    sha256 = "0wr0hba6ka74s3628jrrd7ynjdh7syxigkh7ildg8fgi20ab88fd";
  };

  erlware_commons_1_3_1 = fetchHex {
    pkg = "erlware_commons";
    version = "1.3.1";
    sha256 = "0rnikkyk0hxdh34wshzhvfa9xb5lgwdr6f9f28q082ld6qzskbbs";
  };

  eunit_formatters_0_5_0 = fetchHex {
    pkg = "eunit_formatters";
    version = "0.5.0";
    sha256 = "1jb3hzb216r29x2h4pcjwfmx1k81431rgh5v0mp4x5146hhvmj6n";
  };

  getopt_1_0_1 = fetchHex {
    pkg = "getopt";
    version = "1.0.1";
    sha256 = "174mb46c2qd1f4a7507fng4vvscjh1ds7rykfab5rdnfp61spqak";
  };

  hex_core_0_4_0 = fetchHex {
    pkg = "hex_core";
    version = "0.4.0";
    sha256 = "0zsxvsw6yqg7nk3lq29w40jf34388kvl4vw7psw4rpqhz9n8rkla";
  };

  parse_trans_3_3_0 = fetchHex {
    pkg = "parse_trans";
    version = "3.3.0";
    sha256 = "0q5r871bzx1a8fa06yyxdi3xkkp7v5yqazzah03d6yl3vsmn7vqp";
  };

  providers_1_7_0 = fetchHex {
    pkg = "providers";
    version = "1.7.0";
    sha256 = "19p4rbsdx9lm2ihgvlhxyld1q76kxpd7qwyqxxsgmhl5r8ln3rlb";
  };

  relx_3_28_0 = fetchHex {
    pkg = "relx";
    version = "3.28.0";
    sha256 = "1df79b9861ljc9s61hzc0p3pn86jcky06fcp7l3g09ra18f8gywa";
  };

  ssl_verify_fun_1_1_3 = fetchHex {
    pkg = "ssl_verify_fun";
    version = "1.1.3";
    sha256 = "1zljxashfhqmiscmf298vhr880ppwbgi2rl3nbnyvsfn0mjhw4if";
  };

in

  beamPackages.callPackage rebar_git { }

Let me know if you have any questions.

[k32]

Looks good. In the original rebar3-release _checkouts is a derivation already containing all packages in unpacked form, therefore it's not a list. But this change makes it better.

[ankhers]

In addition to the change I showed above for the rebar3-release.nix file, I have renamed checkouts to beamDeps to make it more understandable what it actually is. I'm also skipping the _checkouts directory and directly linking into the _build directory.

I am also trying to come up with a different name than rebar3Relx. My hesitation with that name is that relx is a specific tool to build releases. But that derivation also can build escripts. It is not immediately apparent based on the name that it does so. Unfortunately rebar3Release suffers from a similar issue because a release is a specific distribution type and an escript is different.

Does anyone have any thoughts on what it could be named that is not ambiguous in what it does? I just want to finalize the naming before releasing an initial version of my tool.

[k32]

I see some problems with linking to _build directory. Unlike using _checkouts which is a documented and supported rebar3 feature, manually managing stuff in _build is a hack.

[ankhers]

After re-testing, I am fine with using the _checkouts directory. I swear I tested previously and the link/directory name needed to be the exact same as the application name. That apparently is not the case. So I will revert to using the _checkouts directory.

I think the issue with the derivation name is still a problem though. We could have two derivations. rebar3Release and rebar3Escriptize that inherit from the same builder. Similar to what we are doing with building Erlang.

[k32]

Great. I have no problem with changing the name, in fact I agree that bringing relx into name was not correct, but the proper name was already taken by the old derivation.

[ankhers]

After a conversation in IRC, I'm thinking we should still use the _build directory. When you use the _checkouts directory, rebar will check on every compile whether or not it needs to recompile that application. This may not seem like a big deal when building for a deployment, but if you nix-shell for development, it will take additional time unnecessarily. rebar3 will also remove the entry for those applications from the dependency list in the rebar.lock file.

[k32]

I don't think we need to consider development in nix-shell as a common use case. I think of rebar wrapper mostly as a mean to package an existing release and I don't expect that a significant number of developers will use it "ab initio". Main problem with unpacking a package to _build is that rebar3 may consider build directory dirty, so it will attempt to sync hex registry, which will fail. It was a big problem with the previous version of rebar wrapper. So we'll get back to bootstrap scripts manipulating contents~/.cache and what not, and it's something I wouldn't touch with a 10 foot pole.
While checkouts directory is a documented way of telling rebar3 that dependencies are local. So this solution is simpler and therefore more reliable.

[elay123]

@k32 I just picked up on this because I am interested in running erlang / elixir applications on nixos. At the moment I am particularly interested in running zotonic cms on Nixos.
http://docs.zotonic.com/en/latest/index.html

[elay123]

Sorry I might have contravened use of issue tracker by asking a general question but I thought it might be relevant regarding the issue with rebar3 and otp standards.

[ankhers]

Would anyone mind giving my rebar32nix tool a go? Just clone the repo and rebar3 escriptize should take care of the rest. It technically works for hex.pm and git dependencies, but I am having some trouble on the building side with git deps. Please post any issues you find on the issue tracker.

[k32]

Sorry, I am having an intensive period at work right now. But I guess I can rewrite at least rebar3 derivation to use your tool in the nearest future.

[ankhers]

Right. #57909 actually has my changes. If you use my tool to generate a nix file, you can build using

nix-build --expr 'with import <nixpkgs> { }; callPackage ./. { }' -I nixpkgs=/path/to/nixpkgs

[alunduil]

Have the changes outlined in this issue been documented in the nixpkgs manual? Does that make sense to do? Also, with the mentioned pull request still open do we want to leave this issue open as well?

[ankhers]

To my knowledge, nothing really came out of this. I'm wondering if @k32 just became too busy and is unable to continue working on this?

Whatever the reason behind closing the issue, I am more than happy to continue the work I have already started on this.

[k32]

I encountered some other problems with Nix and NixOS, unrelated to Erlang. Mainly they are related to security and secret management. Unfortunately these are not as easy to fix, and they were deal-breakers for my usecase, so I halted my investigation.

Sorry for the radio silence, I should've make it clear earlier.