Manageiq: Infrastructure hosts are getting false authentication tick in quadicon

[dmetzger57]

Original BZ: https://bugzilla.redhat.com/show_bug.cgi?id=1704644

Description of problem:
Infrastructure hosts are getting false authentication tick in quadicon.

We saw this behavior with random infra provider -

1. With QE's RHV42 provider, host1 and host2 already authenticated after adding provider
2. With QE's vsphere65-nested provider

Version-Release number of selected component (if applicable):
5.11.0.1.20190424210327_58b5dbd

How reproducible:
Random

Steps to Reproduce:

1. Add provider and don't authenticate any host manually
2. Check host after inventory get loaded

Actual results:
Getting false authentication on host

Expected results:
Host authentication should display correctly

From last BZ comment, I think this is a back-end issue.

To reproduce:
Add dev-rhv42 provider
Drill down to dev-rhv42 Hosts
dell-r420-04 is reported as authenticated in the quadicon
Continuing on to Summary for dell-r420-04 shows it is in fact unauthenticated

dev-rhv42 is authenticated and has an id of 23, dell-r420-04 also has an id of 23. In the authentications table no objects of type Host are listed, and only dev-rhv42 is the only record with an id of 23. It looks like the Host is picking up the auth from the Provider?

[agrare]

I was able to reproduce this and I believe this is a UI issue.

I added this provider and ran a refresh with only rails s and rails c+simulate_queue_worker to isolate exactly what was happening (aka make sure no background process was running an auth check).

1. Added dev-rhv42 provider
2. Refresh relationships and power states
3. Go to hosts on the UI after refresh finishes

4. The second host shows a green checkmark
   

5. Check host authentication status:

>> pp Host.all.map { |h| [h.name, h.authentication_status_ok?] }
[["dell-r420-04", false],
 ["dell-r420-05", false],
 ["dell-r430-09.cloudforms.lab.eng.rdu2.redhat.com", false],
 ["dell-r430-10.cloudforms.lab.eng.rdu2.redhat.com", false]]

All are false.

[h-kataria]

@mzazrivec this is related to https://bugzilla.redhat.com/show_bug.cgi?id=1820012. Adam was able to recreate the issue.

[mzazrivec]

I'm able to reproduce the problem described here, though at the moment it doesn't seem like an UI issue. The GTL simply renders whatever comes from reporting and in this case the authentication_status indeed shows Valid. Hence the OK icon.

This is the minimal reproducer for the problem:

irb(main):030:0> host01 = Host.find 17
=> #<ManageIQ::Providers::Redhat::InfraManager::Host id: 17, name: "dell-r420-05", hostname: "10.8.96.15", ipaddress: "10.8.96.15", vmm_vendor: "redhat", vmm_version: "7.6", vmm_product: "rhel", vmm_buildnumber: nil, created_on: "2020-06-09 12:22:39", updated_on: "2020-06-09 12:22:39", guid: "04bc7b34-6885-4d51-95bd-7b06e0d2bdf8", ems_id: 83, user_assigned_os: nil, power_state: "on", smart: 1, settings: {}, last_perf_capture_on: nil, uid_ems: "0691b690-98e3-44d1-a7ac-1b71bc15236e", connection_state: "connected", ssh_permit_root_login: nil, admin_disabled: nil, service_tag: nil, asset_tag: nil, ipmi_address: nil, mac_address: nil, type: "ManageIQ::Providers::Redhat::InfraManager::Host", failover: nil, ems_ref: "/api/hosts/0691b690-98e3-44d1-a7ac-1b71bc15236e", hyperthreading: nil, ems_cluster_id: 8, next_available_vnc_port: nil, hypervisor_hostname: nil, availability_zone_id: nil, maintenance: false, maintenance_reason: nil, physical_server_id: nil, ems_ref_type: nil>

irb(main):031:0> host01.name
=> "dell-r420-05"

irb(main):032:0> host01.authentication_status
=> "None"

irb(main):033:0> search_options = {:parent=>{:class=>"ManageIQ::Providers::Redhat::InfraManager", :id=>83}, :userid=>"admin", :class=>"Host", :extra_cols=>['authentication_status'], :targets=>[17]}
=> {:parent=>{:class=>"ManageIQ::Providers::Redhat::InfraManager", :id=>83}, :userid=>"admin", :class=>"Host", :extra_cols=>["authentication_status"], :targets=>[17]}

irb(main):034:0> host02 = Rbac.search(search_options)[0][0]
=> #<ManageIQ::Providers::Redhat::InfraManager::Host id: 17, name: "dell-r420-05", hostname: "10.8.96.15", ipaddress: "10.8.96.15", vmm_vendor: "redhat", vmm_version: "7.6", vmm_product: "rhel", vmm_buildnumber: nil, created_on: "2020-06-09 12:22:39", updated_on: "2020-06-09 12:22:39", guid: "04bc7b34-6885-4d51-95bd-7b06e0d2bdf8", ems_id: 83, user_assigned_os: nil, power_state: "on", smart: 1, settings: {}, last_perf_capture_on: nil, uid_ems: "0691b690-98e3-44d1-a7ac-1b71bc15236e", connection_state: "connected", ssh_permit_root_login: nil, admin_disabled: nil, service_tag: nil, asset_tag: nil, ipmi_address: nil, mac_address: nil, type: "ManageIQ::Providers::Redhat::InfraManager::Host", failover: nil, ems_ref: "/api/hosts/0691b690-98e3-44d1-a7ac-1b71bc15236e", hyperthreading: nil, ems_cluster_id: 8, next_available_vnc_port: nil, hypervisor_hostname: nil, availability_zone_id: nil, maintenance: false, maintenance_reason: nil, physical_server_id: nil, ems_ref_type: nil, authentication_status: "Valid">

irb(main):035:0> host02.name
=> "dell-r420-05"

irb(main):036:0> host02.authentication_status
=> "Valid"
irb(main):037:0>

[agrare]

Hey @mzazrivec I was able to see a green checkbox with none of my hosts having a valid authentication status (see #5 in my comment above)

[mzazrivec]

@agrare yep, I was able to reproduce the same thing you're mentioning and my above comment shows that the incorrect data comes from reporting / Rbac.search()

[agrare]

Oh :+1: sorry I didn't notice that host01 and host02 were the same ID

[agrare]

@gtanzillo @lpichler any idea how Rbac might be messing with this?

[lpichler]

@NickLaMuro it looks like that it is something for you 👍

There is reproducer and this feature has been added here and here.

problematic host was different on @mzazrivec's laptop and on my, so there is script
which will find the host to reproduce issue.

#!/usr/bin/env ruby
require File.expand_path('../config/environment', __dir__)
require 'optimist'

Host.where(:type => "ManageIQ::Providers::Redhat::InfraManager::Host").all.each do |host01|
  puts host01.authentication_status
  search_options = {:parent=>{:class=>"ManageIQ::Providers::Redhat::InfraManager", :id=>host01.ems_id}, :userid=>"admin", :class=>"ManageIQ::Providers::Redhat::InfraManager::Host", :extra_cols=>['authentication_status'], :targets=>[host01.id]}
  host02 = Rbac.search(search_options)[0][0]
  puts host02.authentication_status
  puts "-#{host02.id}--"
end


exit

SQL of host01.authentication_status:
$1 = 25
$2 = Host

SELECT  "authentications".* 
FROM "authentications" 
WHERE "authentications"."resource_id" = $1 AND "authentications"."resource_type" = $2 
ORDER BY CASE 
WHEN LOWER("authentications"."status") = '' THEN -1 
WHEN LOWER("authentications"."status") = 'valid' THEN 0 
WHEN LOWER("authentications"."status") = 'none' THEN 1 
WHEN LOWER("authentications"."status") = 'incomplete' THEN 1 
WHEN LOWER("authentications"."status") = 'error' THEN 2 
WHEN LOWER("authentications"."status") = 'unreachable' THEN 2 
WHEN LOWER("authentications"."status") = 'invalid' THEN 3 ELSE -1 END DESC LIMIT 1

SQL of Rbac.search(search_options)[0][0].authentication_status:

SELECT "hosts".*, (SELECT  "authentications"."status" 
FROM "authentications" WHERE "authentications"."resource_id" = "hosts"."id" 
ORDER BY CASE 
WHEN LOWER("authentications"."status") = '' THEN -1 
WHEN LOWER("authentications"."status") = 'valid' THEN 0 
WHEN LOWER("authentications"."status") = 'none' THEN 1 
WHEN LOWER("authentications"."status") = 'incomplete' THEN 1 
WHEN LOWER("authentications"."status") = 'error' THEN 2 
WHEN LOWER("authentications"."status") = 'unreachable' THEN 2 
WHEN LOWER("authentications"."status") = 'invalid' THEN 3 ELSE -1 END DESC LIMIT 1) 
AS "authentication_status" FROM "hosts" WHERE (hosts.id IN (25))

and difference is that in second example is missing "authentications"."resource_type" = $2 so
"authentications"."resource_type" = 'Host'

I am not sure what is best way how to fix it, @NickLaMuro can your help with this ? thanks!

[mzazrivec]

ping -- any update on this one?

[NickLaMuro]

@NickLaMuro it looks like that it is something for you 👍

Sorry, this flew under my radar. Taking a look now.

[NickLaMuro]

tl; dr Still couldn't reproduce. Can someone check besides me?

---

Okay, sorry for the delay, but it took me a bit to get the "reproducer" going.

For me, I didn't have a host in my local DB for RHEV that had a status of "None", so I had to modify one for that to be the case:

irb(main):001:0> Host.find(10000000000004)
#=> #<ManageIQ::Providers::Redhat::InfraManager::Host id: 10000000000004, name: ... >
irb(main):002:0> Host.find(10000000000004).authentication_status
#=> "Valid"
irb(main):003:0> Host.find(10000000000004).authentication_status_severity_level.update_attribute :status, "None"
#=> true
irb(main):004:0> Host.find(10000000000004).authentication_status
#=> "None"

I also modified the script given by @lpichler just to make it a bit more readable and add a few features:

#!/usr/bin/env ruby
require File.expand_path('../config/environment', __dir__)
require 'optimist'

# Uncomment to view the queries inline with the rest of the output,
# though this makes seeing the `puts` output here harder
#
# ActiveRecord::Base.logger = Logger.new(STDOUT).tap { |l| l.level = Logger::DEBUG }
hosts = Host.where(:type => "ManageIQ::Providers::Redhat::InfraManager::Host").all

hosts.each do |host01|
  puts; puts
  puts "**** Host ##{host01.id} *****"
  puts ">>>>> .authentication_status 01##{host01.id}: #{host01.authentication_status}"

  # SELECT  "authentications".*
  # FROM "authentications"
  # WHERE "authentications"."resource_id" = $1
  #   AND "authentications"."resource_type" = $2
  # ORDER BY CASE
  #          WHEN LOWER("authentications"."status") = '' THEN -1
  #          WHEN LOWER("authentications"."status") = 'valid' THEN 0
  #          WHEN LOWER("authentications"."status") = 'none' THEN 1
  #          WHEN LOWER("authentications"."status") = 'incomplete' THEN 1
  #          WHEN LOWER("authentications"."status") = 'error' THEN 2
  #          WHEN LOWER("authentications"."status") = 'unreachable' THEN 2
  #          WHEN LOWER("authentications"."status") = 'invalid' THEN 3 ELSE -1
  #          END DESC
  # LIMIT $3

  search_options = {
    :parent     => {
      :class    => "ManageIQ::Providers::Redhat::InfraManager",
      :id       => host01.ems_id
    },
    :userid     => "admin",
    :class      => "ManageIQ::Providers::Redhat::InfraManager::Host",
    :extra_cols => ['authentication_status'],
    :targets    => [host01.id]
  }
  host02 = Rbac.search(search_options)[0][0]

  puts ">>>>> .authentication_status 02##{host02.id}: #{host02.authentication_status}"

  #SELECT "hosts".*,
  #       (SELECT  "authentications"."status"
  #        FROM "authentications"x
  #        WHERE "authentications"."resource_id" = "hosts"."id"
  #        ORDER BY CASE
  #                 WHEN LOWER("authentications"."status") = '' THEN -1
  #                 WHEN LOWER("authentications"."status") = 'valid' THEN 0
  #                 WHEN LOWER("authentications"."status") = 'none' THEN 1
  #                 WHEN LOWER("authentications"."status") = 'incomplete' THEN 1
  #                 WHEN LOWER("authentications"."status") = 'error' THEN 2
  #                 WHEN LOWER("authentications"."status") = 'unreachable' THEN 2
  #                 WHEN LOWER("authentications"."status") = 'invalid' THEN 3
  #                 ELSE -1
  #                 END DESC
  #        LIMIT 1) AS "authentication_status"
  # FROM "hosts"
  # WHERE "hosts"."type" IN ('ManageIQ::Providers::Redhat::InfraManager::Host')
  # AND (hosts.id IN (10000000000005))
end

However, it seems like I still get the following for output:

$ DB=XXX bundle exec ruby tmp/issue_20115.rb
** ManageIQ master, codename: Kasparov


**** Host #10000000000006 *****
>>>>> .authentication_status 01#10000000000006: Valid
>>>>> .authentication_status 02#10000000000006: Valid


**** Host #10000000000004 *****
>>>>> .authentication_status 01#10000000000004: None
>>>>> .authentication_status 02#10000000000004: None


**** Host #10000000000005 *****
>>>>> .authentication_status 01#10000000000005: Valid
>>>>> .authentication_status 02#10000000000005: Valid

The queries that @lpichler described in https://github.com/ManageIQ/manageiq/issues/20115#issuecomment-641958347 were still there, and I left them as a comment, but the inconsistent results shown in the https://github.com/ManageIQ/manageiq/issues/20115#issuecomment-641324311 by @mzazrivec are not present. Possibly there is something I am missing?

[mzazrivec]

My output from Nick's script:

**** Host #1 *****
>>>>> .authentication_status 01#1: None
>>>>> .authentication_status 02#1: None


**** Host #2 *****
>>>>> .authentication_status 01#2: None
>>>>> .authentication_status 02#2: Valid


**** Host #3 *****
>>>>> .authentication_status 01#3: None
>>>>> .authentication_status 02#3: None


**** Host #4 *****
>>>>> .authentication_status 01#4: None
>>>>> .authentication_status 02#4: None

Reproducing DB sent to Nick.

[NickLaMuro]

EDIT: BAH! Ignore this. This is exactly what @lpichler was talking about in his comment:

and difference is that in second example is missing "authentications"."resource_type" = $2 so "authentications"."resource_type" = 'Host'

So yes, this is the problem. Need to figure out how why the resource_type isn't getting added in the nested select. Might be a virtual_attribtues problem.

---

@mzazrivec So I took a look at this DB you sent, and all of the Host records you have provided don't have authentication records associated with them:

irb(main):001:0> Host.all.map {|host| host.authentications.count }
#=> [0, 0, 0, 0]

Which is really odd, even though there is an outlier it seems when using RBac that returns "Valid"... so that is weird.

What is also odd is that there is a single authentication record for the RHEV EMS that all the host are attached to, and they are all "Valid" as well:

irb(main):001:0> Host.all.map {|host| host.ext_management_system.id }
#=> [2, 2, 2, 2]
irb(main):002:0> Host.all.map {|host| host.ext_management_system.authentication_status }
#=> ["Valid", "Valid", "Valid", "Valid"]
irb(main):003:0> Host.all.map {|host| host.authentication_status }
#=> ["None", "None", "None", "None"]

So that is also a bit suspicious.

---

Can't draw any conclusious yet, just sharing what I have found so far and curious if it might be a combination of things going wrong.

[NickLaMuro]

@mzazrivec @lpichler the above linked PR should address this issue, but let me know if you still notice something wrong.