Nixpkgs: [Rust] How to package crate without Cargo.lock

[matthiasbeyer]

Issue description

I want to (locally) package a crate which does not (yet) have a Cargo.lock file. I asked how to do this before on the Mailinglist but I got no answer which helped me enough to get it packaged.

An example how I want to package it in my local configuration is in the initial email in this thread. There I try to package imag 0.3.0, 0.4.0 was released just a few days ago, so if you want to try yourself when helping me, chose either.

As far as I can tell, the cargoDepsHook does not get evaluated from my package definition and the cargoUpdateHook runs _after_ the check whether a Cargo.lock exists, so I cannot use it.

The documentation on this subject is rather sparse... so I guess if I can get it packaged I may update the documentation on the rust packaging a bit :smile:

[matthiasbeyer]

@wizeman @Mic92 @the-kenny Pinging you as some of the top-contributors to the rust compiler infrastructure in nixpkgs and due to no response in 7 days.

[the-kenny]

I'm a bit stale on our current rust infrastructure. I'll try to have a look into it when I find some time.

[bachp]

I'm not sure how you could package without a Cargo.lock as this would mean that tha dependencies can change over time.

Maybe you could try to provide a Cargo.lock file as part of your nix derivation if upstream doesn't provide one.

[matthiasbeyer]

Maybe you could try to provide a Cargo.lock file as part of your nix derivation if upstream doesn't provide one.

That's exactly what I tried, but if you studied my initial comment here you'll see that it does not work.

[Mic92]

See for an example https://github.com/NixOS/nixpkgs/pull/30088/commits/c3d5fd511c8c049bf61e49f2794a10c5369e2c49 you probably want cargoUpdateHook instead.

On the long run we should fix the patch phase instead.

[matthiasbeyer]

I tried the cargoUpdatesHook as well and it didn't work. See my initial message.

[Mic92]

How about passing a patched src to buildRustPackage, you can make it another derivation where you add a Cargo.lock file.

[matthiasbeyer]

How would I do that?

---

This is what I have so far:

{
  mkImagPackage = { pkgs, pname, version, cratePath, sha256, cargoLock }:
    pkgs.rustPlatform.buildRustPackage rec {
      inherit pname version;

      name = "${pname}-v${version}";
      sourceRoot = "imag-src/${cratePath}";

      src = pkgs.stdenv.mkDerivation {
        name = "${pname}-src";
        src = pkgs.fetchFromGitHub {
          inherit sha256;
          owner  = "matthiasbeyer";
          repo   = "imag";
          rev    = "v${version}";
        };

        phases = [ "unpackPhase" "buildPhase" ];

        buildPhase = ''
          cat ${cargoLock} > Cargo.lock
          mkdir $out -p
          cp ./* -rv $out
        '';
      };

      depsSha256 = "2pbb0swgpsbd6x3avxz6fv3q31dg801li47jibz721a4n9c0rssx";

      buildInputs = with pkgs;
        [ cmake curl gcc libpsl openssl pkgconfig which zlib ];

      meta = with pkgs.stdenv.lib; {
        description = "imag";
        homepage = https://imag-pim.org;
        license = with licenses; [ lgpl2 ];
        maintainers = with maintainers; [ matthiasbeyer ];
        platforms = platforms.linux;
      };
    };

  imag = { pkgs }: mkImagPackage {
    inherit pkgs;
    pname     = "imag";
    version   = "0.4.0";
    sha256    = "0vsfhark6l8g7vp4bd1dm24icy7a9kv40x8g7smh4lwf6m860m23";
    cratePath = "bin/core/imag";
    cargoLock = ./imag-cargolock-0.4.0;
  };
}

and it fails with

unpacking source archive /nix/store/40jwm1z31ly50ni9mhkns1jk44apypl3-imag-src
source root is imag-src/bin/core/imag
installing
Fetching /tmp/nix-build-imag-v0.4.0-fetch.drv-0/imag-src/bin/core/imag to /nix/store/209rpb8lky6n0gk2d8xw9bqz5za82zl9-imag-v0.4.0-fetch
Setting up git templatedir
Using rust registry from /nix/store/5blib6rllbs6c9jyqac9pbnmx3j9xaw8-rustRegistry-2017-10-03

ERROR: The Cargo.lock file doesn't exist

Cargo.lock is needed to make sure that depsSha256 doesn't change
when the registry is updated.

builder for ‘/nix/store/y12kvr1byii3slb5wzi07w4n7r2rkzp5-imag-v0.4.0-fetch.drv’ failed with exit code 1
killing process 5272
killing process 5273
error: build of ‘/nix/store/y12kvr1byii3slb5wzi07w4n7r2rkzp5-imag-v0.4.0-fetch.drv’ failed

... the line that irritates me is Fetching /tmp/nix-build-imag-v0.4.0-fetch.drv-0/imag-src/bin/core/imag to /nix/store/209rpb8lky6n0gk2d8xw9bqz5za82zl9-imag-v0.4.0-fetch here.

[Mic92]

@matthiasbeyer can you provide also the Cargo.lock file?

[matthiasbeyer]

Pasted in pastebin because of size: https://pastebin.com/VhzvkPVS

[Mic92]

The Cargo.lock is in the wrong location:

        buildPhase = ''
          mkdir -p $out
          cp ./* -r $out
          cp ${cargoLock} $out/${cratePath}/Cargo.lock
        '';

Now there is project-specific, different error.

[matthiasbeyer]

I wonder why that is (the wrong path). The project is a multi-crate project and the cargo.lock file should live at the top-level of the project. I will investigate!

[matthiasbeyer]

Now there is project-specific, different error.

This was not necessarily project specific: There exists a crate as a dotfile: ./.imag-documentation which was not copied to the sources before.

I now have

        buildPhase = ''
          mkdir $out -p
          cp ./* -r $out
          cp ./.imag-documentation -r $out

          echo "Copying ${cargoLock} to $out/${cratePath}/Cargo.lock" # dont know why I need this
          cp ${cargoLock} $out/${cratePath}/Cargo.lock

          echo "Copying ${cargoLock} to $out/Cargo.lock"
          cp ${cargoLock} $out/Cargo.lock

          chmod a+wr $out/${cratePath}/Cargo.lock
          chmod a+wr $out/Cargo.lock
        '';

but it fails with

error: failed to write /tmp/nix-build-imag-v0.4.0-fetch.drv-0/imag-src/Cargo.lock

Caused by:
  failed to open: /tmp/nix-build-imag-v0.4.0-fetch.drv-0/imag-src/Cargo.lock

Caused by:
  Permission denied (os error 13)

[Mic92]

you set $sourceRoot in your project, that's why. you have to copy the cargo file to the correct subdirectory. Just insert find, ls -la or build the project with nix-build -k and look at the failed build directory find out what is missing and also check if each directory is readable/executable. I also suspect that cargoUpdateHook would also work if you copy it to the right directory.

[matthiasbeyer]

But cargo should search for the Cargo.lock file at the project root, no matter where it is called! At least my cargo does that when building without nix!

[Mic92]

You build a subdirectory here, which has its own Cargo.toml.

[matthiasbeyer]

Yes, but this is a multi-crate project, it has a Cargo.toml for the whole workspace! Even when building locally, there is only _one_ Cargo.lock for all ~30 crates in this project, and it lives at the top-level directory of the project!

[Mic92]

So this is a wrong assumption by buildRustPackage, which expects the Cargo.lock in the same directory. At the moment only one top-level Cargo.toml and a corresponding Cargo.lock file is supported.

[matthiasbeyer]

Well, that explains a lot. I guess I have to dig deeper then.

[matthiasbeyer]

Hm, @Mic92 can you point me? From what I see the rust package helper simply calls cargo build, which should automatically find the project root and therefor the workspace configuration.

[Mic92]

We would need to add an exception here, to not check for Cargo.lock file since it will be found the by cargo eventually.

[matthiasbeyer]

Hm,... I would not do this,... I would rather check whether a Cargo.lock exists in the current directory or any directory above the sourceRoot... this way we would still guarantee that one is present. Though, its a bit harder to implement, I guess.

[Mic92]

a while loop, which looks for Cargo.lock until it hits /nix/store/<directory>

[vincentbernat]

I'd like to update svgcleaner but upstream stopped shipping Cargo.lock. What is the best way today for such situations?

[bachp]

@vincentbernat I think https://github.com/RazrFalcon/svgcleaner/issues/157 is the best way to deal with this.

[rasendubi]

(triage) @Mic92 @matthiasbeyer are there any action items left to close the issue?

[Mic92]

this was solved.

[monomadic]

I just want to chime in here, Cargo.lock in rust projects is NOT always necessary nor recommended. Core rust devs even mention this. See https://github.com/rust-lang/cargo/issues/315 - conventional wisdom is not to include a cargo.lock for libraries.

[Mic92]

Yes. We mostly build applications with very few exceptions. There are now mechanism to add missing Cargo.lock files: https://github.com/NixOS/nixpkgs/blob/d96959a6296b8ff44787236ee4ba138678360543/pkgs/tools/package-management/cargo-license/default.nix#L14

[SoftwareApe]

@Mic92 posted the answer that worked for me. For discoverability it would be great if this was suggested when the package doesn't have a Cargo.lock instead of leaving the user stranded looking for an answer.

[Mic92]

Added here: https://github.com/NixOS/nixpkgs/pull/89523