Kibana: [Ingest Manager] Fix /Enhance Enrollment version checks for Agent v Kibana

[mostlyjason]

We'd like to add a restriction on enrollment of Elastic Agents where the agent minor version is greater than the kibana minor version. This allows us to make upgrades to the agent while maintaining backwards compatibility for older agent versions (eg. 7.9 agent is not compatible with 7.8 kibana). It also allows us to ship patch updates to fix critical bugs in the agent without requiring the user to upgrade the entire stack.

When the user attempts to enroll an Agent with a higher version, it should print an error message in the log and on the console saying something like "Enrollment failed because the Kibana version must be greater than or equal to the major and minor version of Elastic Agent. We recommend you upgrade your Kibana version or use an Elastic Agent of the same version."

[elasticmachine]

Pinging @elastic/ingest-management (Team:Ingest Management)

[mostlyjason]

@ph to review

[ph]

@nchaulet or @jfsiii Can you confirm the current behavior of the logic we have? I think we have strict <= and we dont' allow patch level to diverge.

[ruflin]

@EricDavisX Would be great to get a test case for this. Also testing that 7.9.1 Elastic Agent works with 7.9.0 Kibana as bugfix releases are allowed.

[EricDavisX]

I'm catching up on tickets, can I ask that we clean up the spec here? Some explicit examples would help with the full semantic version cited. It is a bit confusing to me, thanks so much! @ph @mostlyjason thanks.

If we're clear, we can easily do a manual test and we can look into how to automate it, if feasible / worthwhile for the cost. @rahulgupta-qasource can you track this and write a test please?

[ruflin]

Here a few examples:

The following Agent - Kibana versions should work:

Agent 7.9.0. - Kibana 7.9.0
Agent 7.9.0. - Kibana 7.9.2
Agent 7.9.2 - Kibana 7.9.0
Agent 7.9.0 - Kibana 7.10.0
Agent 7.9.0 - Kibana 7.11.2

The following combinations should NOT work:

Agent 7.10.0. - Kibana 7.9.0
Agent 7.10.0. - Kibana 7.9.2

[rahulgupta-qasource]

Hi @EricDavisX /@ruflin /@mostlyjason

Thank you for the update.

We have validated this ticket and below are our observations:-

Observations:
S.No | Agent version | Kibana version | Error Message| Remark
-- | -- | -- | -- | --
1 | 7.9.0 | 7.9.0 | - | Pass
2 | 7.9.0 | 7.9.2 | - | Blocked as 7.9.2 Kibana is not available on staging cloud platform
3 | 7.9.2 | 7.9.0 | - | Blocked as 7.9.2 artifacts are not available.
4 | 7.9.0 | 7.10.0 | - | Pass
5 | 7.9.0 | 7.11.2 | - | Blocked as 7.11.2 Kibana is not available on staging cloud platform
6 | 7.10.0 | 7.9.0 | fail to enroll: fail to execute request to Kibana: Status code: 400, Kibana returned an error: Bad Request, message: Agent version is not compatible with kibana version | Pass
7 | 7.10.0 | 7.9.2 | - | Blocked as 7.9.2 Kibana is not available on staging cloud platform

Moreover, we have created 02 general testcases for the same :-
https://elastic.testrail.io/index.php?/cases/view/22507
https://elastic.testrail.io/index.php?/cases/view/22508

Queries:-

1. Could you please confirm the error message for 'Scenario 6' in the above table:-
   Actual error message: fail to enroll: fail to execute request to Kibana: Status code: 400, Kibana returned an error: Bad Request, message: Agent version is not compatible with kibana version

Screenshot:

Expected error message as per #75176_description : Enrollment failed because the Kibana version must be greater than or equal to the major and minor version of Elastic Agent. We recommend you upgrade your Kibana version or use an Elastic Agent of the same version.

Please share your feedback.

1. As per our understanding and validation, error message(as shared in 'Query 1' above) is displayed on enrolling higher 7.9.1 Snapshot Elastic Agent version on lower 7.9.0 BC9 Kibana. However, as per #75176_comment , it should work successfully.

Please share your feedback.

1. As per our understanding and #75176_description , 7.9 higher elastic-agent version is not compatible with lower 7.8 kibana. However, 7.9.0 BC9 elastic-agent is successfully enrolled on 7.8.0 kibana.

Screenshot:

Please share your feedback and let us know if anything is missing from our end.

[nchaulet]

Can you confirm the current behavior of the logic we have? I think we have strict <= and we dont' allow patch level to diverge.

Yes we have a strict <= comparaison so agent 7.9.2 should not be working with kibana 7.9.0

[ruflin]

@ph Some time ago we agree, that bugfix not in sync should be ok. I thin we should loosen this up for 7.10.

[EricDavisX]

Thanks Ruflin, that was very helpful.

@rahulgupta-qasource The feature-bug-fix has not been implemented yet so what you are seeing is as expected. The most interesting case to note is that the version restriction changes made thus far were after 7.8 and the changes were on the Kibana side, so the case of 7.9 Agent connected to 7.8 Kibana scenario is possible, however it is not supported or intended. Also then, it is expected behavior as of now.

We will need more than 2 test cases, tho this is a good start! ;)

• per https://semver.org/ and the major.minor.patch versioning scheme, I would suggest we can set the test cases as below. In practical terms we'll need to use the available builds so, during a "dot zero" release there will be no patch versions to use, this is acknowledged.

The following version comparisons need tests written, tho most can be automated and we will plan to do so (timeframe tbd) but for now let us get manual test coverage confirmed and we can update it later on.

section 1) very close versions, only patch release differences - all combos should work
Agent a.b.1 < Kibana a.b.2
Agent a.b.2 > Kibana a.b.1

section 2) somewhat close versions, Agent minor release is 1 or 2 versions back and is older than the stack:
Agent a.9.x < Kibana a.10.x
Agent a.9.x < Kibana a.11.x

section 3) versions where Agent is a minor version or major version greater (newer) than the stack should not work:
Agent 7.10.x > Kibana 7.9.x
Agent 8.0.x > Kibana 7.9.x

@ruflin @ph does that look fully correct and complete enough?

[ph]

@jfsiii Can you take a look at this issue to loosen up for bug fixes release? We can target that for 7.9.1 too. :)

[ph]

@EricDavisX This does look correct, but for section 2) and section 3) we would need to simulate the case with a curl command.

[EricDavisX]

Ack. We can manage that. The automation test issue to create this support is here: https://github.com/elastic/e2e-testing/issues/243

[rahulgupta-qasource]

Hi @EricDavisX

Thank you for the feedback.

We have created 05 and updated 02 testcases for above scenarios as per #75176_comment :-

Testcases created:
https://elastic.testrail.io/index.php?/cases/view/27049
https://elastic.testrail.io/index.php?/cases/view/27055
https://elastic.testrail.io/index.php?/cases/view/27056
https://elastic.testrail.io/index.php?/cases/view/27057
https://elastic.testrail.io/index.php?/cases/view/27060

Testcases updated:
https://elastic.testrail.io/index.php?/cases/view/22507
https://elastic.testrail.io/index.php?/cases/view/22508

Moreover we have executed 04 testcases under Enrollment should fail if Agent version > Kibana version TestRun.

As per discussion in today's Ingest weekly call, we will update the following testcases accordingly and execute them under above TestRun on next working day :
https://elastic.testrail.io/index.php?/cases/view/27049
https://elastic.testrail.io/index.php?/cases/view/27055
https://elastic.testrail.io/index.php?/cases/view/27057

Please let us know if anything is missing from our end.

[EricDavisX]

I confirmed with Rahul that we'll re-test this when the feature work / fixes land. We're aware, and this is still a helpful report to confirm where the product behaviors are currently.

[jfsiii]

@ph et al, I just created https://github.com/elastic/kibana/pull/76238 to fix this

[rahulgupta-qasource]

Hi @EricDavisX

Thank you for the feedback.

We have updated the following 03 testcases and executed them under Fix /Enhance Enrollment version checks for Agent v Kibana TestRun(as per current behavior).

https://elastic.testrail.io/index.php?/cases/view/27049
https://elastic.testrail.io/index.php?/cases/view/27055
https://elastic.testrail.io/index.php?/cases/view/27057

Sure we will re-validate this ticket once feature fix is merged.

Please let us know if anything is missing from our end.

[rahulgupta-qasource]

Hi @EricDavisX / @jfsiii

We have re-validated this ticket on 7.10.0-SNAPSHOT Kibana cloud environment as feature fix #76238 is merged.

Executed 06 and Blocked 01 testcase for agent patch version greater than Kibana patch version under Fix /Enhance Enrollment version checks for Agent v Kibana TestRun due to 7.10.1 artifacts not available.

We have also reported 01 bug #76613 while validating this ticket.

Query:
Actual Error message as in Query1(as per #75176 comment ) is displayed on enrollment when Agent minor version is greater than Kibana minor version or Agent major version is greater than Kibana major version for below testcases:
https://elastic.testrail.io/index.php?/cases/view/22508
https://elastic.testrail.io/index.php?/cases/view/27060

Please share your feedback and let us know if anything is missing from our end.

[EricDavisX]

This looks good to me @rahulgupta-qasource . As the fix is Kibana side, I don't see a need to re-do the tests with different Agent flavors, but it does imply that we've confirmed the Agent flavors are all returning the correct version # and format.

[stacey-gammon]

fwiw I was unable to enroll an agent running 7.9.3 with Kibana running 7.9.0. I upgraded Kibana to 7.9.3 and was able to successfully enroll.