Beats: [Ingest Manager] 7.9 BC1 Agent is not able to download a corresponding 7.9 Endpoint-Security artifact from the BC location

[EricDavisX]

7.9 BC1 Agent is not able to download a corresponding 7.9 Endpoint-Security artifact from the BC location

For confirmed bugs, please report:

• Version: 7.9 BC1 (and latest 7.9 Stack code in cloud staging deploy)
• Operating System: win 7 x64 for Agent.
• Steps to Reproduce:
  1) deploy the latest 7.9 stack to cloud, i"m using SNAPSHOT available as of 5PM 7/15
  2) setup Ingest Manager / Fleet and copy the enrollment command for the default config
  3) download the BC agent from this location to the host: https://staging.elastic.co/7.9.0-33ee764c/summary-7.9.0.html
  4) unarchive and enroll and run agent. all is well, so far.
  5) create a config that has 'Endpoint' integration,
  6) then change your running Agent to use this new config.
  7) in Agent logs this is where I see the attempt to download Endpoint-security but it fails.

Application: endpoint-security--7.9.0[6c1f576d-ee62-4892-826d-f82ba2fe8371]: State changed to FAILED: 2 errors occurred: * package 'data\downloads\endpoint-security-7.9.0-windows-x86_64.zip' not found: open data\downloads\endpoint-security-7.9.0-windows-x86_64.zip: The system cannot find the file specified. * call to 'https://artifacts.elastic.co/downloads/endpoint-dev/endpoint-security-7.9.0-windows-x86_64.zip' returned unsuccessful status code: 404: /go/src/github.com/elastic/beats/x-pack/elastic-agent/pkg/artifact/download/http/downloader.go[142]: unknown error

[elasticmachine]

Pinging @elastic/ingest-management (Team:Ingest Management)

[ph]

# agent.download:
#   sourceURI: "https://artifacts.elastic.co/downloads/"

Using this allows you to configure the source of the binaries? Would that be fine for testing BC?
The artifact doesn't knows its a BC and I don't think we can solve it another way? (minus embedding)

@EricDavisX @Conky5 @blakerouse

[EricDavisX]

removing my redundant duplicate post.

I don't think we should have to modify anything to test the BCs, I think when we branch, we should enhance the Agent uri source for binaries to change from the SNAPSHOT location to the coming BC location. would this mean we'd need to update it with every BC? That's sub-optimal as well, but likely worth the time to do it (even if that was our only reason for doing the change).

[ph]

@EricDavisX what URI you are referring here we should enhance the Agent uri source for binaries to change from the SNAPSHOT location to the coming BC location ? That URI would be different for every BC?

[tonymeehan]

I agree with @ph. I think in order to avoid making configuration changes in BC's in staging, we'd have to update the agent binary to use the staging URL by default (not sure how we'd get the specific URL in there since it contains the git commit hash). If we did this, then when the BC is copied to production, users would have to change the sourceURI again.

Unrelated, @ph, what should sourceURI be? I tried this:

agent.download:
   sourceURI: "https://staging.elastic.co/7.9.0-33ee764c"

I got this error:

Application: endpoint-security--7.9.0[f403eef3-5f4d-4b4f-8c8c-fdad7ddbbb8c]: State changed to FAILED: 2 errors occurred: * package 'data\downloads\endpoint-security-7.9.0-windows-x86_64.zip' not found: open data\downloads\endpoint-security-7.9.0-windows-x86_64.zip: The system cannot find the file specified. * call to 'https://artifacts.elastic.co/downloads/endpoint-dev/endpoint-security-7.9.0-windows-x86_64.zip' returned unsuccessful status code: 404: /go/src/github.com/elastic/beats/x-pack/elastic-agent/pkg/artifact/download/http/downloader.go[142]: unknown error |  
-- | --

[tonymeehan]

Got it to work. My problem was that I was modifying the elastic-agent.yml file when the agent was running in Fleet mode. In that case, the configuration must be added to fleet.yml in order to be picked up by the agent. If it was running in standalone mode, you could add this configuration to elastic-agent.yml. It's maybe a bit too confusing for users?

I also had to update sourceURI to this:

agent.download:
  sourceURI: "https://staging.elastic.co/7.9.0-33ee764c/downloads/"

I've run into some new errors at this point, but I can confirm that the endpoint package is downloaded.

[michalpristas]

@tonymeehan can we close this?

[ph]

let's keep it open for now.

@michalpristas I though we merged both configuration files?

• Read elastic-agent.yml
• Read fleet.yml
• merge both?

If we change to that behavior that would remove some confusion?

@michalpristas @tonymeehan

I think we should do this for next BC https://github.com/elastic/beats/issues/19623 ?

The other workaround is to have the "data/download", both are a bit painful.

[ph]

discussing with @crowens over slack and he proposed to add a new flag at enrollment time to allow to configure the source of artifacts. I think this would be an acceptable choice we could keep as undocumented.

[crowens]

The idea was all @tsg's. He said elasticsearch has something similar for plugins. He can fill in details if needed.

[tsg]

The idea was all @tsg's. He said elasticsearch has something similar for plugins. He can fill in details if needed.

Yeah, Elasticsearch uses an env variable, e.g.:

ES_JAVA_OPTS="-Des.plugins.staging=33ee764c16febf8c45f80db1d148817650c49e7baaba87b56c8c7baa0c46e180fb9c881f6de203574177b5857f9c789f7964537bf4ad63204d8f7a1078940ae9" bin/elasticsearch-plugin install analysis-icu

you can see the instructions at the end of the artifacts HTML page, for example, this one. BTW, if we do this, it would be nice to add it to that HTML page as well.

[EricDavisX]

I put a work-around script into a private repo pr.

• it downloads the Agent and Endpoint (and sha) files and unarchives, copies Endpoint files in, and re-archives it - so it can be copied to a shared location, like a QA Google drive. The run-time enrollment option is definition preferred as it would still be testing the 'download' of the Endpoint, but this is not a high-risk area and the work-around script will be fine until the new option goes in.

[EricDavisX]

This will be testable in BC3 coming on July 22 estimated, unless one is cut sooner. Until then folks can use the QA Google Drive posted to chat with the re-packaged BC2 files