Nokogiri: [install] bundle update pulls in racc with native extensions

[wlipa]

Doing a standard bundle update with a rails app that indirectly pulls in nokogiri has a new native gem dependency on racc with nokogiri 1.11.0. That is despite the fact that racc is a default gem in ruby 2.7 (albeit at an older version).

Before:

$ gem list racc

*** LOCAL GEMS ***

racc (default: 1.4.16)

After:

$ gem list racc

*** LOCAL GEMS ***

racc (1.5.2, default: 1.4.16)

Have you read and followed the installation tutorial at http://www.nokogiri.org/tutorials/installing_nokogiri.html?

• [x] Yes!

What is the complete output of gem install or bundle install?

$ bundle update
Fetching gem metadata from https://rubygems.org/......
Fetching gem metadata from https://rubygems.org/.
Resolving dependencies......
...
Fetching racc 1.5.2
Installing racc 1.5.2 with native extensions
Using nokogiri 1.11.0 (x86_64-darwin) (was 1.10.10)
Using rails-dom-testing 2.0.3
...
Using webpacker 5.2.1
Bundle updated!
Gems in the group production were not updated.

Tell us about your system!

What is the output from ruby -v?
ruby 2.7.2p137 (2020-10-01 revision 5445e04352) [x86_64-darwin20]
What is the output from gem -v?
3.1.4
What is the output from gem env?

RubyGems Environment:
  - RUBYGEMS VERSION: 3.1.4
  - RUBY VERSION: 2.7.2 (2020-10-01 patchlevel 137) [x86_64-darwin20]
  - INSTALLATION DIRECTORY: /Users/myuser/.gem/ruby/2.7.0
  - USER INSTALLATION DIRECTORY: /Users/myuser/.gem/ruby/2.7.0
  - RUBY EXECUTABLE: /opt/local/bin/ruby2.7
  - GIT EXECUTABLE: /opt/local/bin/git
  - EXECUTABLE DIRECTORY: /Users/myuser/.gem/ruby/2.7.0/bin
  - SPEC CACHE DIRECTORY: /Users/myuser/.gem/specs
  - SYSTEM CONFIGURATION DIRECTORY: /opt/local/etc
  - RUBYGEMS PLATFORMS:
    - ruby
    - x86_64-darwin-20
  - GEM PATHS:
     - /Users/myuser/.gem/ruby/2.7.0
     - /opt/local/lib/ruby2.7/gems/2.7.0
  - GEM CONFIGURATION:
     - :update_sources => true
     - :verbose => true
     - :backtrace => false
     - :bulk_threshold => 1000
     - "gem" => "--no-rdoc --no-ri"
  - REMOTE SOURCES:
     - https://rubygems.org/
  - SHELL PATH:
     - /opt/local/bin
     - /opt/local/sbin
     - ./bin
     - /usr/local/bin
     - /usr/bin
     - /bin
     - /usr/sbin
     - /sbin
     - /Library/Apple/usr/bin

If you're using Bundler:

• what is the output from bundle version?
  Bundler version 2.1.4 (2021-01-04 commit unknown)
• what is the output from bundle config? (Take care to redact any credentials)

Settings are listed in order of priority. The top value will be used.
without
Set for your local app (/Users/myuser/Code/ruby/apollo/.bundle/config): [:production]
Set for the current user (/Users/myuser/.bundle/config): [:production]

build.nokogiri
Set for the current user (/Users/myuser/.bundle/config): "--use-system-libraries"

If you're on MacOS, please note:

• the version of XCode you have installed (if you know)
  12.3
• the output of gcc -v or clang -v

Apple clang version 12.0.0 (clang-1200.0.32.28)
Target: x86_64-apple-darwin20.2.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

[k-brooks]

Seeing a similar issue in Chef / Windows, causes gcc dependency failure.

[flavorjones]

Hi, @wlipa! Thanks for reporting this, and sorry you're having issues.

It sounds like you're describing bundler behavior and not something that Nokogiri can directly control. If you'd prefer that bundler not update to the latest version of racc, can I suggest that you add a line like this to your Gemfile?

gem "racc", "~> 1.4.0"

@k-brooks without more information from you, I'm afraid I can't help you. "gcc dependency failure" does not sound like what @wlipa is describing; if you're seeing something different and not addressed by my comment above can you please open a new issue?

[wlipa]

It's not exactly an issue for me. It's just a little ironic that nokogiri apparently goes through a bit of trouble to avoid having native gem install issues for its direct native needs, but adds a dependency on racc for theoretical future-proofing which adds another native gem extension.

I just question whether the explicit racc dependency is actually a good idea if bundler users are going to have to compile a native racc extension by default, rather than relying on the one pre-packaged with ruby.

[voxik]

By default Bundler always installs the most recent version of package no matter what you have on system. But that is Bundler choice. If Bundler used the installed package, the would be no issues.

[k-brooks]

The pessimistic dependency on racc 1.4 from nokogiri introduces a (system) dependency on gcc to compile it.

[wlipa]

That's true. The Nokogiri maintainers should make a choice about a racc dependency based on the overall tradeoff between the impact on the developer ecosystem as it exists (including Bundler behavior) vs. future-proofing concerns.

[k-brooks]

Disregard, misread

[wlipa]

Presumably the version of racc that's needed for the workaround is tied to the version of Ruby you have installed.

It's certainly the case that this change makes it harder to install nokogiri for users today.

[flavorjones]

@k-brooks @wlipa the dependency on racc exists whether Nokogiri declares it or not. Can you help me understand why adding a racc version constraint to your Gemfile isn't a suitable way to address bundler's behavior here?

[wlipa]

I'm not a direct user of either racc or nokogiri - I am using it due to the Rails dependency. So in order to avoid installing an additional native extension, I have to add an extra line to my Gemfile with the precise version of a doubly indirect dependency that varies based on the version of Ruby that's in use.

I can certainly do that after having researched and understood the issue. All rails/bundler users would presumably need to go through the same process if they want to avoid the additional native code dependency. They would also need to update that line when switching ruby versions.

I thought one of the highlights of this release was to reduce the pain of installing native extensions for nokogiri. That seems to have been achieved for nokogiri itself, but adding an additional native gem dependency is somewhat of a step backwards, even if there is a fairly straightforward (if obscure) workaround.

From the discussion in https://github.com/sparklemotion/nokogiri/issues/1988, it seems like the benefits of declaring this dependency are rather abstract. I'm questioning if it's worth causing users to either have to know about this and do some maintenance to avoid it, or to have the extra native compilation step and potential install issues from that.

[voxik]

It seems to me you are trying to solve the wrong problem. The only issue is that you are using mix of prebuilt binary packages and packages you have to build by yourselves. Therefore I'd like to suggest you to try to request Racc upstream to provide binary gem the same way Nokogiri does that.

Just for the record, arguing against properly specifying dependencies is wrong. Just take look on Ruby 3.0. Whoever previously thought it is fine to depend on WEBrick, because it is part of the StdLib will have a problem now. Similarly, if somebody ever used ReXML, they have to change their Gemfiles now.

[flavorjones]

The Nokogiri maintainers should make a choice about a racc dependency based on the overall tradeoff between the impact on the developer ecosystem as it exists (including Bundler behavior) vs. future-proofing concerns.

@wlipa I understand what you're saying, but what I'm saying is that we _have_ made that decision weighing tradeoffs, and we're moving ahead with this dependency being properly declared in the gemspec.

It's certainly the case that this change makes it harder to install nokogiri for users today.

I'm not sure I understand how. Users upgrading from Nokgiri v1.10.x already have the toolchain in place to compile C extensions, and so my mental model is that this is not blocking anyone from upgrading right now. Please help me understand if upgrading is blocked for some users, and if so how.

[wlipa]

Fair enough. I just wanted to make it clear that the typical Rails use case will now have an additional native gem installation requirement via Nokogiri, in case that was not realized.

Installing an additional gem with native extensions can only be more difficult than not installing it at all, which was the case until the dependency was added.

[flavorjones]

Thanks again for opening this issue and talking to us about the situation.

[wlipa]

Not to beat a dead horse, but sadly I just got this in an email from a colleague asking for installation help.

Using mini_portile2 2.5.0
Fetching racc 1.5.2
Installing racc 1.5.2 with native extensions
Gem::Ext::BuildError: ERROR: Failed to build gem native extension.

current directory:
/Users/myuser2/.gem/ruby/2.7.0/gems/racc-1.5.2/ext/racc/cparse
/opt/local/bin/ruby2.7 -I /opt/local/lib/ruby2.7/2.7.0 -r
./siteconf20210104-30183-axgzet.rb extconf.rb
checking for rb_ary_subseq()... *** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of necessary
libraries and/or headers.  Check the mkmf.log file for more details.  You may
need configuration options.

Provided configuration options:
       --with-opt-dir
       --with-opt-include
       --without-opt-include=${opt-dir}/include
       --with-opt-lib
       --without-opt-lib=${opt-dir}/lib
       --with-make-prog
       --without-make-prog
       --srcdir=.
       --curdir
       --ruby=/opt/local/bin/$(RUBY_BASE_NAME)2.7
/opt/local/lib/ruby2.7/2.7.0/mkmf.rb:471:in `try_do': The compiler failed to
generate an executable file. (RuntimeError)
You have to install development tools first.
       from /opt/local/lib/ruby2.7/2.7.0/mkmf.rb:564:in `try_link0'
       from /opt/local/lib/ruby2.7/2.7.0/mkmf.rb:582:in `try_link'
       from /opt/local/lib/ruby2.7/2.7.0/mkmf.rb:794:in `try_func'
       from /opt/local/lib/ruby2.7/2.7.0/mkmf.rb:1083:in `block in have_func'
       from /opt/local/lib/ruby2.7/2.7.0/mkmf.rb:971:in `block in checking_for'
       from /opt/local/lib/ruby2.7/2.7.0/mkmf.rb:361:in `block (2 levels) in postpone'
       from /opt/local/lib/ruby2.7/2.7.0/mkmf.rb:331:in `open'
       from /opt/local/lib/ruby2.7/2.7.0/mkmf.rb:361:in `block in postpone'
       from /opt/local/lib/ruby2.7/2.7.0/mkmf.rb:331:in `open'
       from /opt/local/lib/ruby2.7/2.7.0/mkmf.rb:357:in `postpone'
       from /opt/local/lib/ruby2.7/2.7.0/mkmf.rb:970:in `checking_for'
       from /opt/local/lib/ruby2.7/2.7.0/mkmf.rb:1082:in `have_func'
       from extconf.rb:6:in `<main>'

To see why this extension failed to compile, please check the mkmf.log which can
be found here:

/Users/myuser2/.gem/ruby/2.7.0/extensions/arm64-darwin-20/2.7.0/racc-1.5.2/mkmf.log

extconf failed, exit code 1

Gem files will remain installed in
/Users/myuser2/.gem/ruby/2.7.0/gems/racc-1.5.2 for inspection.
Results logged to
/Users/myuser2/.gem/ruby/2.7.0/extensions/arm64-darwin-20/2.7.0/racc-1.5.2/gem_make.out

An error occurred while installing racc (1.5.2), and Bundler cannot
continue.
Make sure that `gem install racc -v '1.5.2' --source 'https://rubygems.org/'`
succeeds before bundling.

In Gemfile:
      rails was resolved to 6.0.3.4, which depends on
        actioncable was resolved to 6.0.3.4, which depends on
          actionpack was resolved to 6.0.3.4, which depends on
            actionview was resolved to 6.0.3.4, which depends on
              rails-dom-testing was resolved to 2.0.3, which depends on
                nokogiri was resolved to 1.11.0, which depends on
                  racc

[flavorjones]

I've added documentation about racc here: https://nokogiri.org/tutorials/installing_nokogiri.html#cannot-install-racc

[nightpool]

I also had a colleague ask me about problems compiling racc this afternoon, on a normal Mac laptop. Is there any way the Nokogiri team could reach out to the Bundler team about this? All else being equal, it's really unfortunate to add lots of extra complexity to people's build processes these ways.

[nightpool]

(I still haven't figured out what the actual compilation error is, unfortunately, but this was an unfortunate blocker for us upgrading. Will follow up once I have more details)

[flavorjones]

@nightpool I linked to this in an earlier comment: https://nokogiri.org/tutorials/installing_nokogiri.html#cannot-install-racc

If you're experiencing some other error, can you please provide specifics in a new issue created at https://github.com/ruby/racc ?