Nixpkgs: Can't get nokogiri, a rails dependency, to build with bundlerEnv

[jolmg]

Issue description

I'm trying to make a development environment for a rails project, but I can't get nokogiri, which seems to be a dependency of rails, to build.

Steps to reproduce

I simplified the gemfile to only require nokogiri. To reproduce the problem, put the following files in a directory and run nix-shell. BTW, I'm inlining them, as github doesn't seem to like the extensions, and I figure that using zip or having to later rename them from .txt might be more bothersome.

Gemfile

source "https://rubygems.org"
gem 'nokogiri'

Gemfile.lock

``` Gemfile.lock
GEM
remote: https://rubygems.org/
specs:
mini_portile2 (2.1.0)
nokogiri (1.6.8)
mini_portile2 (~> 2.1.0)
pkg-config (~> 1.1.7)
pkg-config (1.1.7)

PLATFORMS
ruby

DEPENDENCIES
nokogiri

BUNDLED WITH
1.10.5

gemset.nix

``` nix
{
  pkg-config = {
    version = "1.1.7";
    source = {
      type = "gem";
      remotes = ["https://rubygems.org"];
      sha256 = "0lljiqnm0b4z6iy87lzapwrdfa6ps63x2z5zbs038iig8dqx2g0z";
    };
  };
  nokogiri = {
    version = "1.6.8";
    source = {
      type = "gem";
      remotes = ["https://rubygems.org"];
      sha256 = "17pjhvm4yigriizxbbpx266nnh6nckdm33m3j4ws9dcg99daz91p";
    };
  };
  mini_portile2 = {
    source = {
      remotes = ["https://rubygems.org"];
      sha256 = "1y25adxb1hgg1wb2rn20g3vl07qziq6fz364jc5694611zz863hb";
      type = "gem";
    };
    version = "2.1.0";
  };
}

default.nix

with (import <nixpkgs> {});
let
  env = bundlerEnv {
    name = "test-package";
    inherit ruby;
    gemfile = ./Gemfile;
    lockfile = ./Gemfile.lock;
    gemset = ./gemset.nix;
  };
in stdenv.mkDerivation {
  name = "test-package";
  buildInputs = [env ruby];
}

This is the output of running nix-shell. It includes stderr.

Attempts at fixing

I think the exact reason why it failed is in the mkmf.log file mentioned in the output, but I can't find it under /nix/store/5jcpxbf2i4bgs3w7s7dfxl9467pla9ch-ruby2.3.0-p0-nokogiri-1.6.8/ which, according to the output, seems to have been left to inspect the failure of the build. Are the build directories the paths under /nix/store or is there another place I can look for said file?

I tried replacing nokogiri in the gemfile with each of its 2 dependencies in turn, and (after rebuilding Gemfile.lock and gemset.nix) they both lead to no problems with nix-shell. So I guess it's not a problem with nokogiri's dependencies. Because the output hinted that it might be due to missing libraries, I also tried including zlib, pkg-config, and something else (sorry I forget) in the buildInputs, but it didn't change the outcome.

Technical details

• System: NixOS 16.03
• Nix version: nix-env (Nix) 1.11.2
• Nixpkgs version: 16.03.git.0470d0d, though that's the hash of a personal commit. It's based on baf46b9 from the nixos-16.03 branch from the nixpkgs-channels repo. Changes made are to nixos/modules/config/update-users-groups.pl and pkgs/applications/networking/sync/unison/default.nix, so that's unrelated.

[cstrahan]

This is the important line from your logs:

/nix/store/s7skd7lkiaw1vnf5qjxkgy56r55plp1n-rubygems-2.4.8/lib/rubygems/dependency.rb:315:in `to_specs': Could not find 'pkg-config' (~> 1.1.7) among 5 total gem(s) (Gem::LoadError)

Nokogiri expects that the pkg-config gem is installed, which it then uses in its extconf.rb to invoke the pkg-config binary, in order to resolve CFLAGS and such during compilation of the native C extension.

Unfortunately, your gemset.nix appears to be broken: it doesn't show that pkg-config is a dependency of nokogiri, and therefore bundlerEnv doesn't know to pass along pkg-config when building nokogiri, which then results in the error message that I highlighted above.

Looks like there might be a bug in bundix that'll need to be fixed.

/cc @manveru

[jolmg]

I found this line:

https://github.com/NixOS/nixpkgs/blob/release-16.03/pkgs/development/interpreters/ruby/bundler-env/default.nix#L36

which seems to allow specifying other gems in the gemset as dependencies, passing their derivations as propagatedBuildInputs of the dependent. I tried changing the gemset.nix package to this:

{
  pkg-config = {
    version = "1.1.7";
    source = {
      type = "gem";
      remotes = ["https://rubygems.org"];
      sha256 = "0lljiqnm0b4z6iy87lzapwrdfa6ps63x2z5zbs038iig8dqx2g0z";
    };
  };
  nokogiri = {
    version = "1.6.8";
    dependencies = ["pkg-config" "mini_portile2"];
    source = {
      type = "gem";
      remotes = ["https://rubygems.org"];
      sha256 = "17pjhvm4yigriizxbbpx266nnh6nckdm33m3j4ws9dcg99daz91p";
    };
  };
  mini_portile2 = {
    version = "2.1.0";
    source = {
      type = "gem";
      remotes = ["https://rubygems.org"];
      sha256 = "1y25adxb1hgg1wb2rn20g3vl07qziq6fz364jc5694611zz863hb";
    };
  };
}

but that still gives me the same error.

I listed the files in /nix/store/nj7cq3zmci9vd3x1fzqmkw1dk252y48g-ruby2.3.0-p0-nokogiri-1.6.8, but I didn't find the files of its dependencies symlinked in there. I'm going to search to see if their gem paths are included in GEM_PATH before executing extconf.rb.

[jolmg]

It seems this line is what, at some point, ends up executing extconf.rb:

https://github.com/NixOS/nixpkgs/blob/release-16.03/pkgs/development/interpreters/ruby/build-ruby-gem/default.nix#L183

I echoed $RUBYLIB (found that's what was needed, not $GEM_PATH) right before that line and got /nix/store/s7skd7lkiaw1vnf5qjxkgy56r55plp1n-rubygems-2.4.8/lib:/nix/store/a28k21v4xikvs393bvmcrfhwfqcg9yrl-ruby2.3.0-p0-pkg-config-1.1.7/lib/ruby/gems/2.3.0/gems/pkg-config-1.1.7/lib:/nix/store/glbn70m5zmczgyay5jpgcp3chn80mf7y-ruby2.3.0-p0-mini_portile2-2.1.0/lib/ruby/gems/2.3.0/gems/mini_portile2-2.1.0/lib.

So, that's not it.

[jolmg]

Adding pkg.pkgconfig to buildInputs in either/both nokogiri and/or pkg-config failed to work. Did an strace on the gem install call, only found that the error came right after trying to stat a Makefile and finding it didn't exist. I guess extconf.rb makes Makefiles.. I'm going to read more on ruby's native extension mechanism tomorrow/today; going to bed now..

[jolmg]

I figured I can debug the gem package by using nix-shell with this default.nix:

with import <nixpkgs> {};

let simpleGem =
      { version
      , gemName
      , gemPath ? []
      , sha256
      } @ args:
      buildRubyGem (args // {
        inherit ruby rubygems;
        type = "gem";
        remotes = ["https://rubygems.org"];
      });
in

simpleGem {
  gemName = "nokogiri";
  sha256 = "17pjhvm4yigriizxbbpx266nnh6nckdm33m3j4ws9dcg99daz91p";
  version = "1.6.8";
  gemPath = [
    (simpleGem {
      sha256 = "0lljiqnm0b4z6iy87lzapwrdfa6ps63x2z5zbs038iig8dqx2g0z";
      version = "1.1.7";
      gemName = "pkg-config";
    })
    (simpleGem {
      sha256 = "1y25adxb1hgg1wb2rn20g3vl07qziq6fz364jc5694611zz863hb";
      version = "2.1.0";
      gemName = "mini_portile2";
    })
  ];
}

but running unpackPhase in the shell gives me this error:

unpacking source archive /nix/store/y8wf8kjaqqkkdbcwra3y5027kxi4bw4b-nokogiri-1.6.8.gem
do not know how to unpack source archive /nix/store/y8wf8kjaqqkkdbcwra3y5027kxi4bw4b-nokogiri-1.6.8.gem
exit

I ran typeset -f unpackPhase and got back what looks like the generic unpacker. Why didn't I get this? I thought maybe it was wrapping over it, but they both run the preUnpack and postUnpack hooks.

[jolmg]

Using nix-build gives me the same error as the one in the OP, where I'm using bundlerEnv. Running (import ./default.nix).unpackPhase in nix-repl using the default.nix with the buildRubyGem in the previous comment, does give me the correct unpackPhase.

Am I misunderstanding something about nix-shell, or is this another bug?

[johbo]

Just bumped into the same issue, did you manage to figure it out?

[johbo]

Found two bits which might be related:

• bundix -d will include the dependencies into gemset.nix
• bundlerEnv supports a parameter gemConfig which can be used if special tweaks are needed, default seems to be here https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/ruby-modules/gem-config/default.nix

For my case using -d to generate gemset.nix did the trick.

[jolmg]

No, I haven't tried again since that last comment. I'll look into what you suggest tonight.

[ivanbrennan]

@jolmg Were you able to resolve this? I just packaged (using bundix) a gem that depends on nokogiri, and found I had to run bundix in a nix-shell with zlib and libiconv,

nix-shell -p bundix zlib libiconv --run 'bundix --magic'

I wonder if you were missing libiconv?

[jolmg]

No, I gave up on this and moved back to Archlinux. I'll try to check this again sometime this week in a virtual machine.

[tylermauthe]

FWIW, I had the same issues and when I ran what @ivanbrennan posted it worked with no extra configs at all -- just like a normal bundle install. Glorious. Thank you!

[aanderse]

@jolmg seeing as a solution was found and you're no longer in a position to test can you please close the issue?