Manageiq: Support Ruby 2.7

Created on 2 Jan 2020 · 25Comments · Source: ManageIQ/manageiq

Motivation: ruby 2.5 is EOL 2021-03-31 so we need 2.7 and 2.6 to be the supported rubies.

Work to be done:

[x] [Allow Ruby 2.7 in Gemfile](https://github.com/ManageIQ/manageiq/pull/21023)
[x] [Fix automation engine tests](https://github.com/ManageIQ/manageiq-automation_engine/pull/471)
[x] Create lasker branch
[x] Change Gemfile to not allow ruby 2.5
- [x] master / lasker (backport) via 21095
[x] Change .travis.yml to add ruby 2.7/ drop ruby 2.5
- [x] master / lasker (backport) via 21095
[x] Lockdown release branch to test only ruby 2.6 in .travis.yml
- [x] lasker only PR
- [x] Update developer doc
- [x] Blast updates to each repository, dropping 2.5, adding 2.7 if not already there.
- [x] [Update cross-repo-test's .travis.yml](https://github.com/ManageIQ/manageiq-cross_repo-tests/pull/345)

Current maintenance schedule for ruby as of 2021-02-09 here:

Ruby 3.0
status: normal maintenance
release date: 2020-12-25

Ruby 2.7
status: normal maintenance
release date: 2019-12-25

Ruby 2.6
status: normal maintenance
release date: 2018-12-25

Ruby 2.5
status: security maintenance
release date: 2017-12-25
EOL date: 2021-03-31

Errors:

[x] Some standard libraries were extracted to gems:
- [x] [sync for core](https://github.com/ManageIQ/manageiq/pull/19682)
- [x] [sync for more_core_extensions](https://github.com/ManageIQ/more_core_extensions/pull/79)

CMath (cmath gem)
Scanf (scanf gem)
Shell (shell gem)
Synchronizer (sync gem)
ThreadsWait (thwait gem)
E2MM (e2mmap gem)

[x] config gem blows up on missing new_ostruct_member, fixed/triaged here (PR merged, gem with fix released)
[x] "warning: Capturing the given block using Proc.new is deprecated; use &block instead\n"
Causes many test failures checking the output to be empty:

  6) TaskHelpers::Imports::GenericObjectDefinitions#import when the source is a directory imports all .yaml files in a specified directory
     Failure/Error:
       expect do
         TaskHelpers::Imports::GenericObjectDefinitions.new.import(options)
       end.to_not output.to_stderr

       expected block to not output to stderr, but output "/Users/joerafaniello/.gem/ruby/2.7.0/gems/activerecord-5.1.7/lib/active_record/statement_cache.rb:90: warning: Capturing the given block using Proc.new is deprecated; use `&block` instead\n"

[x] undefined method `new' for BigDecimal:Class

   4) MiqWorker Monitor After Setup, A WorkerMonitor threshold validation for excessive memory should trigger memory threshold if worker is ready
     Failure/Error: :memory_usage     => ActiveSupport::NumberHelper.number_to_human_size(usage),

     NoMethodError:
       undefined method `new' for BigDecimal:Class
     # /Users/joerafaniello/.gem/ruby/2.7.0/gems/activesupport-5.1.7/lib/active_support/number_helper/rounding_helper.rb:37:in `round_significant'
     # /Users/joerafaniello/.gem/ruby/2.7.0/gems/activesupport-5.1.7/lib/active_support/number_helper/rounding_helper.rb:14:in `round'
     # /Users/joerafaniello/.gem/ruby/2.7.0/gems/activesupport-5.1.7/lib/active_support/number_helper/number_to_rounded_converter.rb:9:in `convert'
     # /Users/joerafaniello/.gem/ruby/2.7.0/gems/activesupport-5.1.7/lib/active_support/number_helper/number_converter.rb:132:in `execute'
     # /Users/joerafaniello/.gem/ruby/2.7.0/gems/activesupport-5.1.7/lib/active_support/number_helper/number_converter.rb:118:in `convert'
     # /Users/joerafaniello/.gem/ruby/2.7.0/gems/activesupport-5.1.7/lib/active_support/number_helper/number_to_human_size_converter.rb:21:in `convert'
     # /Users/joerafaniello/.gem/ruby/2.7.0/gems/activesupport-5.1.7/lib/active_support/number_helper/number_converter.rb:132:in `execute'
     # /Users/joerafaniello/.gem/ruby/2.7.0/gems/activesupport-5.1.7/lib/active_support/number_helper/number_converter.rb:118:in `convert'
     # /Users/joerafaniello/.gem/ruby/2.7.0/gems/activesupport-5.1.7/lib/active_support/number_helper.rb:262:in `number_to_human_size'
     # ./app/models/miq_server/worker_management/monitor/validation.rb:43:in `exceeded_memory_threshold?'
     # ./app/models/miq_server/worker_management/monitor/validation.rb:14:in `validate_worker'
     # ./spec/models/miq_server/worker_monitor_spec.rb:313:in `block (6 levels) in <top (required)>'

enhancement ruby2.7

Source

jrafanie

🎉1

Most helpful comment

Done 🎉

Fryguy on 19 Mar 2021

🚀2

All 25 comments

Warnings:

[x] ~~/Users/joerafaniello/.gem/ruby/2.7.0/gems/json-2.1.0/lib/json/common.rb:156: warning: Using the last argument as keyword parameters is deprecated~~ # No longer warning in 2.7.2
[x] ~~/Users/joerafaniello/.gem/ruby/2.7.0/gems/faker-1.8.7/lib/faker.rb:149: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call~~ # No longer warning in 2.7.2
[x] ~~/Users/joerafaniello/.gem/ruby/2.7.0/gems/i18n-1.7.0/lib/i18n.rb:179: warning: The called method `translate' is defined here. Various method names are found in test output with nearly 1000 warnings.~~ # No longer warning in 2.7.2
[x] ~~/Users/joerafaniello/Code/manageiq/lib/extensions/ar_href_slug.rb:5: warning: Passing the keyword argument as the last hash parameter is deprecated~~ # No longer warning in 2.7.2
[x] warning: URI.escape is obsolete file_depot_mixin.rb:42: warning: URI.escape is obsolete (replace with Addressable.escape/encode)
[x] warning: URI.unescape is obsolete # If found, use Addressable.unencode
[x] ~~11 warning: deprecated Object#=~ is called on FalseClass; it always returns nil~~ # No longer warning in 2.7.2
[x] ~~7080 warning: deprecated Object#=~ is called on Float; it always returns nil~~ # No longer warning in 2.7.2

jrafanie on 3 Jan 2020

@jrafanie : https://github.com/rubyconfig/config/releases/tag/2.2.1

Fryguy on 15 Jan 2020

👍1

ArgumentError: wrong number of arguments (given 0, expected 1)
/Users/duhlmann/manageiq/lib/extensions/ar_types.rb:4:in 'initialize_type_map'
/Users/duhlmann/.rvm/gems/ruby-2.7.0-preview1/gems/activerecord-5.2.4.1/lib/active_record/connection_adapters/postgresql_adapter.rb:235:in 'initialize'

fixed by passing the instance var in 235 of the adapter

d-m-u on 23 Jan 2020

the warning: Capturing the given block using Proc.new is deprecated; use&blockinstead errors I'm seeing are mostly from faraday which got updated

d-m-u on 23 Jan 2020

recursive-open-struct was broken on ruby2.7, fixed in 1.1.1.
Current Gemfile _allows_ 1.1.1 so bundle update takes it.
But it also allows older versions e.g. 1.0.4.

Do we want to force 1.1.1+ in one of our gemfiles to guarantee a ruby2.7-compatible version?
(Currently it's a transitive dependency, not direct)

I ~~might~~ EDIT: forced it in kubeclient, but we're unlikely bump kubeclient far enough to benefit (kubevirt currently pins kubeclient to 4.1.z, will happen in 4.6+ or 5.0)...

cben on 7 Apr 2020

Do you care about keyword arg warnings in gems outside manageiq?

kubeclient had a couple, fixed by https://github.com/abonas/kubeclient/pull/438

cben on 7 Apr 2020

Do we want to force 1.1.1+ in one of our gemfiles to guarantee a ruby2.7-compatible version?
(Currently it's a transitive dependency, not direct)
I ~might~ EDIT: forced it in kubeclient, but we're unlikely bump kubeclient far enough to benefit (kubevirt currently pins kubeclient to 4.1.z, will happen in 4.6+ or 5.0)...

I'd rather not have to pull in recursive-open-struct, an indirect dependency, into manageiq. I'd rather kubeclient/image inspector client enforced the new version.

jrafanie on 7 Apr 2020

Do you care about keyword arg warnings in gems outside manageiq?

kubeclient had a couple, fixed by abonas/kubeclient#438

yeah, we'll want to pull in any new versions that support ruby 2.7 where possible. Can we safely upgrade kubeclient to get more 2.7 support?

jrafanie on 7 Apr 2020

Working on updating to to a newer kubeclient version, updating manageiq-providers-kubevirt to work with a newer version of fog-kuebvirt so we're not stuck on kubeclient 4.1 https://github.com/ManageIQ/manageiq-providers-kubevirt/pull/160 and https://github.com/ManageIQ/manageiq-providers-kubernetes/pull/368

agrare on 20 Apr 2020

We need to move to at least kubeclient v4.5 due to this issue https://github.com/abonas/kubeclient/issues/442

We are currently limited to 4.3 until we can get a release of fog-kubevirt with https://github.com/fog/fog-kubevirt/pull/145

agrare on 25 Apr 2020

Released more_core_extensions v4.1.0 with Ruby 2.7 support

Fryguy on 1 May 2020

Cross repo tests are failing with:

Bundler could not find compatible versions for gem "bundler":
  In Gemfile:
    bundler (~> 2.1.4)
    manageiq-gems-pending (> 0) was resolved to 0.1.0, which depends on
      bundler (>= 1.8.4)
    rails (~> 5.1.7) was resolved to 5.1.7, which depends on
      bundler (>= 1.3.0)
  Current Bundler version:
    bundler (2.1.2)

This Gemfile requires a different version of Bundler.

Perhaps you need to update Bundler by running `gem install bundler`?

agrare on 6 May 2020

we have updated the rails version. I wonder if that affects the bundler version
Maybe it is rails 6.0 - that requires a much more recent bundler

kbrock on 8 May 2020

@jrafanie I know it's been a while since you've looked at this, but what is the current status?

Fryguy on 3 Feb 2021

Good question. I'll have to take a look. I have no idea.

jrafanie on 3 Feb 2021

I haven't kicked it in a while but everything was green except for automation-engine and content https://travis-ci.com/github/ManageIQ/manageiq-cross_repo-tests/builds/211809884
IIRC there was an issue with automate scripts raising exceptions not working right

On travis it hangs, locally I see:

Failures:

  1) MiqAeEngine::MiqAeMethod.invoke_inline_ruby (private) with a script that raises logs the error with file and line numbers changed in the stacktrace, and raises an exception
     Failure/Error: expect($miq_ae_logger).to receive(:error).with("Method STDERR: /my/automate/method:2:in `<main>': unhandled exception").at_least(:once)

       #<VMDBLogger:0x00005594811c6748 @level=1, @progname=nil, @default_formatter=#<Logger::Formatter:0x00005594811c66f8 @datetime_format=nil>, @formatter=#<VMDBLogger::Formatter:0x00005594811c65e0 @datetime_format=nil>, @logdev=#<Logger::LogDevice:0x00005594811c66a8 @shift_period_suffix="%Y%m%d", @shift_size=1048576, @shift_age=0, @filename=#<Pathname:/home/agrare/src/manageiq/manageiq/log/automation.log>, @dev=#<File:/home/agrare/src/manageiq/manageiq/log/automation.log>, @binmode=false, @mon_data=#<Monitor:0x00005594811c6680>, @mon_data_owner_object_id=61000>, @write_lock=#<Thread::Mutex:0x00005594811c65b8>, @local_levels={}, @thread_hash_level_key=:"ThreadSafeLogger#112520@level"> received :error with unexpected arguments
         expected: ("Method STDERR: /my/automate/method:2:in `<main>': unhandled exception")
              got: ("<AEMethod /my/automate/method> The following error occurred during method evaluation:") (1 time)
                   ("<AEMethod /my/automate/method>   RuntimeError: ") (1 time)
                   ("<AEMethod /my/automate/method>   ") (1 time)
     # ./spec/engine/miq_ae_method_spec.rb:56:in `block (4 levels) in <top (required)>'
     # /home/agrare/.gem/ruby-2.7.0/gems/webmock-3.11.2/lib/webmock/rspec.rb:37:in `block (2 levels) in <top (required)>'

  2) MiqAeEngine::MiqAeMethod.invoke_inline_ruby (private) with a script that raises in a nested method logs the error with file and line numbers changed in the stacktrace, and raises an exception
     Failure/Error: expect($miq_ae_logger).to receive(:error).with("Method STDERR: /my/automate/method:2:in `my_method': unhandled exception").at_least(:once)

       #<VMDBLogger:0x00005594811c6748 @level=1, @progname=nil, @default_formatter=#<Logger::Formatter:0x00005594811c66f8 @datetime_format=nil>, @formatter=#<VMDBLogger::Formatter:0x00005594811c65e0 @datetime_format=nil>, @logdev=#<Logger::LogDevice:0x00005594811c66a8 @shift_period_suffix="%Y%m%d", @shift_size=1048576, @shift_age=0, @filename=#<Pathname:/home/agrare/src/manageiq/manageiq/log/automation.log>, @dev=#<File:/home/agrare/src/manageiq/manageiq/log/automation.log>, @binmode=false, @mon_data=#<Monitor:0x00005594811c6680>, @mon_data_owner_object_id=61000>, @write_lock=#<Thread::Mutex:0x00005594811c65b8>, @local_levels={}, @thread_hash_level_key=:"ThreadSafeLogger#112520@level"> received :error with unexpected arguments
         expected: ("Method STDERR: /my/automate/method:2:in `my_method': unhandled exception")
              got: ("<AEMethod /my/automate/method> The following error occurred during method evaluation:") (1 time)
                   ("<AEMethod /my/automate/method>   RuntimeError: ") (1 time)
                   ("<AEMethod /my/automate/method>   /my/automate/method:2:in `my_method'") (1 time)
     # ./spec/engine/miq_ae_method_spec.rb:76:in `block (4 levels) in <top (required)>'
     # /home/agrare/.gem/ruby-2.7.0/gems/webmock-3.11.2/lib/webmock/rspec.rb:37:in `block (2 levels) in <top (required)>'

agrare on 3 Feb 2021

Thanks @agrare I'll take a look to see if I can find anything obvious.

jrafanie on 3 Feb 2021

For failed expectations on STDERR, it looks like the messages are now going to stdout instead of stderr, causing the failure. Looking.

jrafanie on 3 Feb 2021

I did a cross-repo with https://github.com/ManageIQ/manageiq/pull/21023 and https://github.com/ManageIQ/manageiq-automation_engine/pull/471 here: https://github.com/ManageIQ/manageiq-cross_repo-tests/pull/304

All tests pass with ruby 2.5, 2.6.6, and 2.7.2. The only remaining warning I see are the warning: URI.escape is obsolete ones, which mentioned above could be replaced with Addressable.escape/encode but since that's not a gem we directly rely on here, we can handle after we're on 2.7.

jrafanie on 9 Feb 2021

For failed expectations on STDERR, it looks like the messages are now going to stdout instead of stderr, causing the failure. Looking.

Was fixed in https://github.com/ManageIQ/manageiq-automation_engine/pull/471
We were replacing STDERR/STDOUT with an existing logger but not closing the original IO objects. By closing the objects, we get the correct standard error messages the tests were expecting.

jrafanie on 9 Feb 2021

[ ] 49 warning: URI.escape is obsolete file_depot_mixin.rb:42: warning: URI.escape is obsolete (replace with Addressable.escape/encode)

[ ] 732 warning: URI.unescape is obsolete # If found, use Addressable.unencode

opened https://github.com/ManageIQ/manageiq/pull/21036 to address these

jrafanie on 10 Feb 2021

@jrafanie https://www.youtube.com/watch?v=OrPT7h2hsok - Upgrading GitHub to Ruby 2.7