Beats: [Ingest Manager] [7.10] Incorrect agent signing

[michalpristas]

When running elastic-agent BC3 we run into following warning

This is caused probably by incorrect signing using Macos natarizer, checking with Infra

[elasticmachine]

Pinging @elastic/ingest-management (Team:Ingest Management)

[Conky5]

We (infra-release) are going to need to add another layer of macOS codesigning + notarization to the release pipeline to accommodate the downstream elastic-agent artifact that includes the bundled artifacts.

[ph]

Just to clarify here, is that a regression from 7.9, based on @EricDavisX it is?

[EricDavisX]

I will cite it is changed / regressed in 7.10:
I verified the 'enroll' and 'help' commands of the Elastic Agent of 7.9.3 (GA) from the download site against my 7.10 BC3 cloud Kibana and didn't see any popup in the Mac on a Catalina darwin version; it executed without any pop up or fuss.

[EricDavisX]

we think this is fixed, Chris gave us a test .tar.gz for mac that looked fine to me.

[ph]

@Conky5 I can close this?

[Conky5]

I am waiting for the current snapshot build process to complete to confirm this is resolved, but I think it is fine to close this issue. Any further developments can be discussed on the internal issue.

[rahulgupta-qasource]

Hi @EricDavisX

We are blocked to execute the smoke testcases on 8.0.0-snapshot cloud environment and 7.10.0-snapshot cloud environment on mac sip disabled machines with Endpoint Security enabled in Policy due to bug #22224 and #22226

Given below are the build details:

Build Details:
Kibana Cloud : 8.0.0 SNAPSHOT
Commit : db0816f4a1fb7fb5a24106be53c86e5f8c5404e1
Build : 37600
Elastic-Agent Hash : c4b68cea
Agent download location:
https://snapshots.elastic.co/8.0.0-c4b68cea/downloads/beats/elastic-agent/elastic-agent-8.0.0-SNAPSHOT-darwin-x86_64.tar.gz

Build Details:
Kibana Cloud : 7.10.0 SNAPSHOT
Commit : 595e59e34a3802685133acbdbc89d39b100d83e5
Build : 35859
Elastic-Agent Hash : 81ee96e5
Agent download location:
https://snapshots.elastic.co/7.10.0-81ee96e5/downloads/beats/elastic-agent/elastic-agent-7.10.0-SNAPSHOT-darwin-x86_64.tar.gz

8.0.0 smoke Testrun: https://elastic.testrail.io/index.php?/runs/view/783
7.10.0 smoke Testrun: https://elastic.testrail.io/index.php?/runs/view/784

Query:
Currently, we are following the process of having mac disabled on kibana snapshot build and mac sip enabled on BC builds. Could you please share the exact steps for verifying the signing of the Agent artifacts so that we can create testcases for the same.

Please let us know if anything is missing from our end.

[EricDavisX]

You can execute the 7.10 Agent tests against the current 7.10 BC, by the way - the Kibana env has no bearing on the Agent signing.

Let us talk through and review SIP. I think testing with it enabled is appropriate in snapshot and BC/GA builds, it is not quite exactly a 1 to 1 comparison with Windows 'test signing' in that regards. SIP a safety feature and should be expected to be enabled by users. If the endpoint runs differently in an environment with SIP enabled, I do think it is highly likely to be related to signing of the artifacts, and we have 2 different teams contributing to signing... but for macOS I think all of it is now done by Infra jobs and their latest change which is why we are testing this, is applied to all snapshot and BCs and so we can test with the assumption it should 'work'.

What does it mean for Agent to 'work'? That is comprised of the Agent installing without any pop up about the risk of the signer not being known, that Agent can come alive and deploy Endpoint (and Endpoint can be validated to pull appropriate artifacts from its global server. The Endpoint artifact pull is something we had discussed prior, I don't have notes easily available, but if you want to chat it we can discuss with Madison Caldwell and Dan Ferullo in email, please feel free to send it so we can establish good test process.

[dikshachauhan-qasource]

Hi @EricDavisX

We have verified this with a 7.10 snapshot Agent against 7.10 BC3 build on MacOS having sip enabled.
Agent details: https://snapshots.elastic.co/7.10.0-81ee96e5/downloads/beats/elastic-agent/elastic-agent-7.10.0-SNAPSHOT-darwin-x86_64.tar.gz
Kibana Commit: ed66f41a8a60ad03426beff65ed270a743c46ac4

We were able to install agent successfully without any issues. Please refer Screenshot:

[dikshachauhan-qasource]

Hi @EricDavisX,

We have again verified it with 7.10 BC4 build. Please find the detailed observation as follows:
Artifacts: https://staging.elastic.co/7.10.0-c650b297/summary-7.10.0.html/elastic-agent-7.10.0-darwin-x86_64.tar.gz
Agent hash: c650b297
BUILD 35887
COMMIT 3f7bbe02a2f08f0ad1c7f5a9b35235a4aa066a32
OS : MACos Catalina 10.15
Sip Enabled

Precondition:

• 7.10.0 BC4 Kibana cloud environment should be available.
• Elastic endpoint Security integration should be added to Default Policy.

Steps to reproduce:

• Install Elastic agent with ./elastic-agent install command successfully.
• Access remote console for Mac VM, allow the permission to notifications for Elastic Security.
  Screenshot:
  

• Navigate to Agents tab in Kibana.
• Click on prerequisite host showing enrolling>online.
• Wait for the Activity logs.

Observation: Filebeat, Matricbeat and elastic security logs are available in running state.
Screenshot:

[dikshachauhan-qasource]

Hi @EricDavisX,

We have added a testcase covering this scenario in the Testrail. Testcase link : https://elastic.testrail.io/index.php?/cases/view/34994

Please let us know if anything else is required to be covered.

[rahulgupta-qasource]

Bug Conversion:

Below 01 Testcase already exists for this ticket:
https://elastic.testrail.io/index.php?/cases/view/34994