Nokogiri: Feedback and final to-do items for precompiled native gems on Linux and OSX

[flavorjones]

This issue exists to capture feedback and action items about the precompiled native gems for Linux and OSX that we're shipping in the Nokogiri v1.11 release candidates (prereleases).

(RC4 is out! https://github.com/sparklemotion/nokogiri/releases/tag/v1.11.0.rc4)

How do I try this out?

We're only shipping native gems for the following platforms (run bundle platform or ruby -e 'puts Gem::Platform.local.to_s' to identify your system):

• New platforms as of v1.11 release candidates:
  
  
  
  • OSX/Darwin: x86_64-darwin and arm64-darwin
  
  
  • Linux: x86-linux and x86_64-linux including musl platforms like alpine
  
  
• Previously existing native gems (note these have already been available for many years):
  
  
  
  • Windows: x86-mingw32 and x64-mingw32
  
  
  • JRuby: java
  
  

If you're on Linux or OSX, try updating your Gemfile with:

gem "nokogiri", "~> 1.11.0.rc4"`

If you're having trouble with bundler trying to install a native gem and it failing, you may want to set the specific_platform bundler config as described in bundler install should not fail when a native platform gem's required_ruby_version is not met, and a suitable ruby platform gem is available · Issue #4012 · rubygems/rubygems:

bundle config set --global specific_platform true

Please let us know how it works for you in the comments!

Known Issues

• Linux systems with glibc < 2.17 are not supported. See #2081 for context and #2156 to see our attempt to be helpful.

Providing Feedback

If you love this idea in a vague general sense, please __give this comment a :heart:__.

If you tried native gems on Linux or OSX and they worked just fine, please __give this comment a :+1:__.

If you're having problems installing or using Nokogiri's precompiled native gems, please __comment on this issue__ with the following information:

• output from ruby -v
• output from gem -v
• output from gem env
• if you're using bundler,
  
  
  
  • output from bundle version
  
  
  • output from bundle config (take care to redact any personal info or credentials)
  
  
• output from gem install or bundle install, including the command you ran

Why is a precompiled native gem useful?

Installing a precompiled native gem, __installation takes about 1 second__, and avoids the challenging bits of how to set up the compiler toolchain on your development system. This will reduce the support burden for Team Nokogiri and allow us to move a bit faster on bugfixes and features.

For comparison, compiling and installing the gem (and its vendored dependencies) from source, when only using a single core, my laptop takes 52 seconds. When using all 8 cores, this is cut down to 32 seconds.

Why would I not want to use a precompiled native gem?

I can imagine some folks might have trust issues; if this is you, please let us know in a comment here (or in #2013) what we could do to increase that trust (e.g., I can imagine providing a chain of custody including public build logs with cryptographic hashes of artifacts).

Anybody on a linux system old enough to not have glibc >= 2.17 will need to install (via compilation) from the ruby platform gem.

If you have other reasons for not wanting to use a precompiled native gem, let us know in the comments.

How can I avoid using a precompiled native gem?

If you want to avoid using a precompiled gem in your project, do one of the following:

• If you're not using bundler, then run gem install nokogiri -v 1.11.0.rc4 --platform=ruby
• If you are using bundler, then run bundle config set force_ruby_platform true and you'll never use a precompiled native gem

Shout-outs

Thanks to @larskanis for all the time he spends maintaining the gems rake-compiler and rake-compiler-dock, and for being the one who wrote the first PR for Linux native gem support.

Thanks to the GRPC team who already ships native gems (see https://github.com/grpc/grpc). They blazed the trail and provided inspiration and a how-to manual for some edge cases.

Thanks to @tenderlove who paired with me on the nasty OSX bits that I was struggling with.

Resources

• #1571 is the original PR from @larskanis for Linux native gems
• #1983 is an issue to polish Linux native gem support
• #2063 is an issue researching OSX/Darwin native gem support
• #2073 is a PR introducing OSX/Darwin native gem support
• #2013 is an issue to increase trust in published gems

[odlp]

Unfortunately my system seems to be building the native extensions still:

time gem install --prerelease nokogiri
Fetching nokogiri-1.11.0.rc3.gem
Fetching mini_portile2-2.5.0.gem
Successfully installed mini_portile2-2.5.0
Building native extensions. This could take a while...
Successfully installed nokogiri-1.11.0.rc3
2 gems installed
gem install --prerelease nokogiri  46.41s user 34.10s system 93% cpu 1:26.53 total

I'm on macOS Catalina (10.15). My Ruby / Rubygems version (via Rbenv):

ruby 2.7.0p0 (2019-12-25 revision 647ee6f091) [x86_64-darwin18]

gem -v
3.1.4

I noticed on Rubygems.org that RC3 only has darwin-19 build, yet my Ruby version states darwin-18. Is this a Catalina / BigSur mismatch?

[flavorjones]

@odlp - Thanks for giving it a try! I'm not building darwin18 gems at this point, because I don't have a system available to do so. Only darwin19 is supported right now.

(But I have an old Mac Mini sitting around, maybe I'll use that for darwin18!)

[odlp]

@flavorjones thanks for clarifying!

Thanks for all this work on precompiled Nokogiri, it's an awesome improvement. Happy thought: I wonder how many tons of CO2 this'll save per year 😃

[gsiener]

macOS Catalina (10.15.6)
ruby-2.6.5 [ x86_64 ]

gsiener$ time gem install --prerelease nokogiri
Successfully installed nokogiri-1.11.0.rc3-x86_64-darwin-19
1 gem installed

real    0m2.835s
user    0m1.159s
sys 0m0.651s

[jrochkind]

On MacOS 10.15.6, ruby 2.6.6

Installs fine. Ran tests in one app and two gems I work on, tests green in all of them.

[thbar]

Quick feedback:

• On one ruby 2.5.8p224 (2020-03-31 revision 67882) [x86_64-darwin18] Rails app, there is no regression. It is darwin18 as pointed above, so compiling like before, but I still wanted to point out the absence of regressions :smile:
• On one JRuby app (tested on both jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc OpenJDK 64-Bit Server VM 25.232-b09 on 1.8.0_232-b09 +jit [darwin-x86_64] and jruby 9.2.5.0 (2.5.0) 2018-12-06 6d5a228 OpenJDK 64-Bit Server VM 25.232-b09 on 1.8.0_232-b09 +jit [darwin-x86_64]), I see many test failures as soon as I upgrade from Nokogiri 1.10.10 to any of the 1.11.0.rc[1/2/3] releases. Something appears to have been botched there.

I will investigate more on the JRuby failures.

Thanks for your work on accelerating the installation, much appreciated!

[flavorjones]

@thbar Thanks for testing this. If you need to, please open a new GH issue for the jruby problems -- I don't think they're going to be related to the native gem work for Linux and OSX. I'd like to know more about what you're seeing, though!

[captn3m0]

Works perfectly on linux x86_64 (click for logs)

time gem install --prerelease nokogiri
Fetching nokogiri-1.11.0.rc3-x86_64-linux.gem
Nokogiri is built with the packaged libraries: libxml2-2.9.10, libxslt-1.1.34, zlib-1.2.11, libiconv-1.15.
Successfully installed nokogiri-1.11.0.rc3-x86_64-linux
Parsing documentation for nokogiri-1.11.0.rc3-x86_64-linux
Installing ri documentation for nokogiri-1.11.0.rc3-x86_64-linux
Done installing documentation for nokogiri after 1 seconds
1 gem installed

real 5.469  user 1.821  sys 0.218 pcpu 37.28

nokogiri -v
# Nokogiri (1.11.0.rc3)
    ---
    warnings: []
    nokogiri: 1.11.0.rc3
    ruby:
      version: 2.7.1
      platform: x86_64-linux
      gem_platform: x86_64-linux
      description: ruby 2.7.1p83 (2020-03-31 revision a0c7c23c9c) [x86_64-linux]
      engine: ruby
    libxml:
      source: packaged
      patches:
      - 0001-Revert-Do-not-URI-escape-in-server-side-includes.patch
      - 0002-Remove-script-macro-support.patch
      - 0003-Update-entities-to-remove-handling-of-ssi.patch
      - 0004-libxml2.la-is-in-top_builddir.patch
      - 0005-Fix-infinite-loop-in-xmlStringLenDecodeEntities.patch
      compiled: 2.9.10
      loaded: 2.9.10
    libxslt:
      source: packaged
      patches: []
      compiled: 1.1.34
      loaded: 1.1.34

[twk3]

Its not happy on centos6, ruby 2.6

 $ ldd /usr/local/lib/ruby/gems/2.6.0/gems/nokogiri-1.11.0.rc3-x86_64-linux/lib/nokogiri/nokogiri.so
/usr/local/lib/ruby/gems/2.6.0/gems/nokogiri-1.11.0.rc3-x86_64-linux/lib/nokogiri/nokogiri.so: /lib64/libz.so.1: version `ZLIB_1.2.3.3' not found (required by /usr/local/lib/ruby/gems/2.6.0/gems/nokogiri-1.11.0.rc3-x86_64-linux/lib/nokogiri/nokogiri.so)
/usr/local/lib/ruby/gems/2.6.0/gems/nokogiri-1.11.0.rc3-x86_64-linux/lib/nokogiri/nokogiri.so: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /usr/local/lib/ruby/gems/2.6.0/gems/nokogiri-1.11.0.rc3-x86_64-linux/lib/nokogiri/nokogiri.so)
/usr/local/lib/ruby/gems/2.6.0/gems/nokogiri-1.11.0.rc3-x86_64-linux/lib/nokogiri/nokogiri.so: /lib64/libc.so.6: version `GLIBC_2.17' not found (required by /usr/local/lib/ruby/gems/2.6.0/gems/nokogiri-1.11.0.rc3-x86_64-linux/lib/nokogiri/nokogiri.so)
    linux-vdso.so.1 =>  (0x00007ffc61dd8000)
    libruby.so.2.5 => not found
    libm.so.6 => /lib64/libm.so.6 (0x00007f64d7b0f000)
    libz.so.1 => /lib64/libz.so.1 (0x00007f64d78f9000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f64d76dc000)
    libc.so.6 => /lib64/libc.so.6 (0x00007f64d7348000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f64d81c5000)

I have the newer zlib in /usr/local/lib

[flavorjones]

@twk3 Interesting! Thank you for reporting this, I'll take a look.

[flavorjones]

@twk3 How did you compile/build/install that version of Ruby? Looks like yum install ruby on centos6 installs Ruby 1.8.7 (!). I'd like to reproduce what you're seeing.

[twk3]

@flavorjones compiled from source, can be seen here: https://gitlab.com/gitlab-org/gitlab-omnibus-builder/-/blob/5523332b2cd3a14b4f4a4aa2b215465499ea3617/docker/Dockerfile_centos_6#L85

That environment can be pulled using docker at: registry.gitlab.com/gitlab-org/gitlab-omnibus-builder/centos_6:0.0.72

[flavorjones]

@twk3 I've created #2081 to investigate the Centos6 issue you reported above. Thanks again!

[flavorjones]

@odlp and others running Darwin 18 or earlier: please jump over to #2079 where I've got a gem file I need tested. Thanks!

[bruno-]

Any chance you can release a 1.11 rc version that works with ruby 3.0 preview?

I'm getting this when I try to use 1.11.0.rc3 on a project with ruby 3.0.0-preview1:

nokogiri-1.11.0.rc3-x86_64-darwin-19 requires ruby version < 2.8.dev, >= 2.4, which is incompatible with the current version, ruby 3.0.0p-1

[flavorjones]

@bruno- I'll look into it this week.

[flavorjones]

Any chance you can release a 1.11 rc version that works with ruby 3.0 preview?

I'm getting this when I try to use 1.11.0.rc3 on a project with ruby 3.0.0-preview1:

nokogiri-1.11.0.rc3-x86_64-darwin-19 requires ruby version < 2.8.dev, >= 2.4, which is incompatible with the current version, ruby 3.0.0p-1

@bruno- Please note that I've opened an upstream Bundler issue (https://github.com/rubygems/rubygems/issues/4012) for this behavior. And I'm going to do my best to cut a native gem that's compatible with Ruby 3.0 in the next release candidate.

[flavorjones]

Ahoy, everybody. Release v1.11.0.rc4 / 2020-12-29 is out. This is really close to what I'd like to ship in a v1.11.0 final release, including native gem support for linux and darwin (even arm64!).

Please give it a whirl and comment if you have problems. Something like gem install nokogiri --prerelease should do the trick.

[thbar]

@flavorjones tested fine on the JRuby app I mentioned earlier. Thanks for this pre-release!

[flavorjones]

Heads up that I'm investigating an issue running the precompiled linux binaries on musl-based systems (e.g., alpine). If you're having issues on alpine hold tight while we look into it.

[flavorjones]

The musl issue is addressed on head as of 713e723 and I'm planning to cut v1.11.0 final tonight. So, I'm closing this. 💥

[rikikonikoff]

I was able to get around this issue by setting my bundle config as indicated above (see "How can I avoid using a precompiled native gem?"), but I still want to share in case this helps someone else.

I am running a mac OSX machine and I updated my Gemfile with gem 'nokogiri', '~> 1.11' and then ran bundle update nokogiri. This saved a file vendor/cache/nokogiri-1.11.0-x86_64-darwin.gem.

Heroku relies on vendor/cache during bundle install. Since Heroku is not running a mac OSX machine and we are not using a docker image, it gets confused; it says it cannot find nokogiri v1.11 in our cache and it therefore cannot build our app. See the following build output:

-----> Ruby app detected -----> Installing bundler 2.1.4 -----> Removing BUNDLED WITH version in the Gemfile.lock -----> Compiling Ruby/Rails -----> Using Ruby version: ruby-2.7.2 -----> Installing dependencies using bundler 2.1.4 Running: BUNDLE_WITHOUT='development:test' BUNDLE_PATH=vendor/bundle BUNDLE_BIN=vendor/bundle/bin BUNDLE_DEPLOYMENT=1 bundle install -j4 Some gems seem to be missing from your vendor/cache directory. Could not find nokogiri-1.11.0 in any of the sources Bundler Output: Some gems seem to be missing from your vendor/cache directory. Could not find nokogiri-1.11.0 in any of the sources ! ! Failed to install gems via Bundler. ! ! Push rejected, failed to compile Ruby app. ! Push failed

Again, for anyone needing advice on this situation, the solution is:

• Delete the offending vendor/cache file
• Run bundle config set force_ruby_platform true
• Re-run bundle install

[jzyeezy]

Similar to @rikikonikoff , we are finding that a gem with the exact same nokogiri version/dependency as a previous successfully installed gem cannot install nokogiri, with the following error:

"ERROR:  Error installing my-gem:\n\t\"nokogiri\" from nokogiri conflicts with /usr/local/bin/nokogiri"

Which is a symlink to our install-dir's pre-compiled native gem:

nokogiri -> /var/lib/gems/2.5.0/gems/nokogiri-1.11.0-x86_64-linux/bin/nokogiri*

Our workaround to stay on 1.11.0 was to delete/usr/local/bin/nokogiri before the subsequent gem is installed.

[larskanis]

@rikikonikoff There should be a better fix than force_ruby_platform. Bundler version 2.2 has much better multiplatform support and can package for multiple platforms like so:

gem inst "bundler:~>2.2"
bundle lock --add-platform ruby
bundle lock --add-platform x86_64-darwin
bundle lock --add-platform x86_64-linux
bundle package --all-platforms

This should fetch all required platform gems:

$ ls vendor/cache/
mini_portile2-2.5.0.gem  nokogiri-1.11.1.gem  nokogiri-1.11.1-x86_64-darwin.gem  nokogiri-1.11.1-x86_64-linux.gem  racc-1.5.2.gem

[rikikonikoff]

@rikikonikoff There should be a better fix than force_ruby_platform. Bundler version 2.2 has much better multiplatform support and can package for multiple platforms like so:

gem inst "bundler:~>2.2"
bundle lock --add-platform ruby
bundle lock --add-platform x86_64-darwin
bundle lock --add-platform x86_64-linux
bundle package --all-platforms

This should fetch all required platform gems:

$ ls vendor/cache/
mini_portile2-2.5.0.gem  nokogiri-1.11.1.gem  nokogiri-1.11.1-x86_64-darwin.gem  nokogiri-1.11.1-x86_64-linux.gem  racc-1.5.2.gem

@larskanis what is the benefit of what you just described? Same as the benefit of using a native gem to begin with?

[larskanis]

@rikikonikoff The benefit is that you enable the use of gem versions specific to the listed platforms, even if the target system is offline and running on a different platform. Specific platform gems have precompiled binaries, so that they install much faster, don't need a compiler to be installed and have less dependencies. See also our install guide.

On the other hand force_ruby_platform disables platform gems entirely and should be used as a workaround only.

@flavorjones What do you think - is the above bundler package description something for our install tutorial?

[flavorjones]

@larskanis Yes! I'm adding it to the installation doc right now. Actually, I'm just going to give a brief description of the problem and then link to https://blog.thegnar.co/caching-all-native-gem-platforms which is a terrific write-up.

[rikikonikoff]

Thanks for explaining, @larskanis!

[flavorjones]

Docs are up: https://nokogiri.org/tutorials/installing_nokogiri.html#using-vendorcache-to-deploy-to-another-architecture

[codener]

We like to compile Nokogiri with system libxml because it simplifies updating when libxml has another of its frequent security issues. Will this continue to be possible?

[flavorjones]

@codener Yes. Please read https://nokogiri.org/tutorials/installing_nokogiri.html#installing-using-standard-system-libraries and let me know -- in a new issue, please -- if it does not address your concerns.