Nokogiri: [bug] Can't install nokogiri with Ruby 3.1.0.dev

[eileencodes]

Please describe the bug

I was benchmarking some Ruby changes I'm making with @tenderlove but I'm unable to install the latest Nokogiri with Ruby 3.1.0.dev (Ruby master). When I try to bundle Rails using Ruby built from source I get the following:

nokogiri-1.11.1-x86_64-darwin requires ruby version < 3.1.dev, >= 2.5, which is incompatible with the current version, ruby
3.1.0p-1

My bundler version is Bundler version 2.3.0.dev

I saw this previous issue https://github.com/sparklemotion/nokogiri/issues/2075 and tried specific_platform but that made no difference. I also tried the known "bundler hack" to work around platform specific issues but that also made no difference in being able to bundle Rails. I don't know if this is a bundler issue or a nokogiri issue but I wanted to start here.

module BundlerHack
  def __materialize__
    if name == "nokogiri"
      Bundler.settings.temporary(force_ruby_platform: true) do
        super
      end 
    else
      super
    end 
  end 
end
Bundler::LazySpecification.prepend(BundlerHack)

Help us reproduce what you're seeing

1) Clone Rails https://github.com/rails/rails on OSX
2) Clone the latest Ruby and make and make install
3) Change .ruby-version to ruby-trunk
4) Run bundle and see the error.

[eileencodes]

I'm so sorry, I accidentally hit enter before filling in the details 😱

[flavorjones]

:wave: Sorry you're dealing with this. I opened an upstream bundler issue here: https://github.com/rubygems/rubygems/issues/4012 and I believe a permanent fix has been committed to Bundler master/main.

I'm going to tag in the Bundler maintainer I've been working with on this and related issues, @deivid-rodriguez, who might have some advice and help.

[flavorjones]

In the meantime I'm going through building Ruby locally to try to reproduce on my system.

[deivid-rodriguez]

I believe this is expected, because 3.1.0.p1 is considered newer than 3.1.0.dev, so it doesn't match < 3.1.0.dev. If nokogiri wanted to allow development versions of ruby 3.1.0, then I believe the upper bound should be < 3.1.0?

I think the reason this wasn't detected in rubygems/rubygems#4012 is that nokogiri was using < 2.8.0.dev at the time, but ruby 2.8 ended up not being a thing, so all ruby 3.0.0 development versions would match that requirement.

[eileencodes]

I believe this is expected, because 3.1.0.p1 is considered newer than 3.1.0.dev, so it doesn't match < 3.1.0.dev. If nokogiri wanted to allow development versions of ruby 3.1.0, then I believe the upper bound should be < 3.1.0?

AFAICT Nokogiri isn't setting an upper bound, it only has a lower one. https://github.com/sparklemotion/nokogiri/blob/main/nokogiri.gemspec#L40

[deivid-rodriguez]

Oh, just ignore me, I'm tired today. I understand now, this is indeed the same issue. Bundler should've chosen the RUBY version since it has no upper bound.

I'll build ruby-head too and have a look :+1:.

[deivid-rodriguez]

Our comments crossed @eileencodes, yeah, you're right. I misunderstood the issue initially.

[deivid-rodriguez]

One question @eileencodes. Do you have a lockfile? If you do, does the issue still repro if you delete it?

[eileencodes]

I deleted rails/rails lockfile and it looks like Nokogiri did install in that case, but bundling failed to build native extensions for curses.

[flavorjones]

OK, running

• ruby 3.1.0dev
• bundler 2.3.0.dev
• starting from scratch (without a Gemfile.lock file)

installs nokogiri v1.11.1 (the ruby platform version) as expected for me.

[deivid-rodriguez]

I think the issue here is reusing a lockfile across rubies that resolve to different sets of gems. I assume you use that lockfile for older rubies that are resolved to the platform specific version. If that's the case, bundler will use the lockfile since it currently doesn't have the ability of detecting this situation and re-resolving. I think it's the same issue as https://github.com/rubygems/rubygems/issues/4282.

I coded a solution involving storing required_ruby_version and required_rubygems_version constraints in the lockfile so that the situation can be detected and a re-resolve is triggered if the constraints are not matched by the running ruby version, but I'm not sure yet how to move it forward since the change in the lockfile format is not backwards compatible.

but bundling failed to build native extensions for curses.

This bit is expected, right? (or at least not a bundler issue?)

[flavorjones]

I'm going to close this out since I don't think there's anything we can do within Nokogiri to address this, and there are some open Bundler conversations about it.