Beats: [Elastic-agent] .ASC files not being packaged into the Docker images

[mdelapenya]

Log error when running the image to enroll the agent:

[2020-09-24T07:06:01.287Z] elastic-agent_1 | 2020-09-24T07:03:38.989Z ERROR operation/operation_retryable.go:85 operation operation-verify failed
[2020-09-24T07:06:01.287Z] elastic-agent_1 | 2020-09-24T07:03:38.989Z DEBUG application/periodic.go:40 Failed to read configuration, error: could not emit configuration: operator: failed to execute step sc-run, error: operation 'operation-verify' failed to verify metricbeat.8.0.0-SNAPSHOT: 3 errors occurred:
[2020-09-24T07:06:01.287Z] elastic-agent_1 | * fetching asc file from '/usr/share/elastic-agent/data/elastic-agent-d816b6/downloads/metricbeat-8.0.0-SNAPSHOT-linux-x86_64.tar.gz.asc': open /usr/share/elastic-agent/data/elastic-agent-d816b6/downloads/metricbeat-8.0.0-SNAPSHOT-linux-x86_64.tar.gz.asc: no such file or directory
[2020-09-24T07:06:01.287Z] elastic-agent_1 | * check detached signature: openpgp: invalid signature: hash tag doesn't match
[2020-09-24T07:06:01.287Z] elastic-agent_1 |
[2020-09-24T07:06:01.287Z] elastic-agent_1 | : operation 'operation-verify' failed to verify metricbeat.8.0.0-SNAPSHOT: 3 errors occurred:
[2020-09-24T07:06:01.287Z] elastic-agent_1 | * fetching asc file from '/usr/share/elastic-agent/data/elastic-agent-d816b6/downloads/metricbeat-8.0.0-SNAPSHOT-linux-x86_64.tar.gz.asc': open /usr/share/elastic-agent/data/elastic-agent-d816b6/downloads/metricbeat-8.0.0-SNAPSHOT-linux-x86_64.tar.gz.asc: no such file or directory
[2020-09-24T07:06:01.287Z] elastic-agent_1 | * check detached signature: openpgp: invalid signature: hash tag doesn't match
It seems that the .ASC is not present in none of the expected locations: filesystem or artifactory. So no process is able to start

_Originally posted by @mdelapenya in https://github.com/elastic/beats/issues/21120#issuecomment-698499868_

[elasticmachine]

Pinging @elastic/ingest-management (Team:Ingest Management)

[ph]

@michalpristas @ruflin @blakerouse Any idea how we could solve that for dockers images? I am reluctant to disable the asc check in the docker images but maybe its the only solution.

[blakerouse]

I don't know enough about the build process, but I assume that the docker image comes with metricbeat and filebeat built into it, so I would think adding the .asc files to it should follow the same path?

[michalpristas]

copied from a slack thread:
this is what i see

* fetching asc file from '/usr/share/elastic-agent/data/elastic-agent-d816b6/downloads/metricbeat-8.0.0-SNAPSHOT-linux-x86_64.tar.gz.asc': open /usr/share/elastic-agent/data/elastic-agent-d816b6/downloads/metricbeat-8.0.0-SNAPSHOT-linux-x86_64.tar.gz.asc: no such file or directory
* check detached signature: openpgp: invalid signature: hash tag doesn't match
* fetching asc file from https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-8.0.0-SNAPSHOT-linux-x86_64.tar.gz.asc: call to 'https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-8.0.0-SNAPSHOT-linux-x86_64.tar.gz.asc' returned unsuccessful status code: 404: /go/src/github.com/elastic/beats/x-pack/elastic-agent/pkg/artifact/download/http/verifier.go[205]: unknown error

3 errors
first one from local verifier: asc is not there. dont know why, it should be other packages has asc files included, will check with infra

second one: verifier check downloaded signature for included binary invalid signature: hash tag doesn't match

third one it tries to download snapshot asc from official store, this one is expected.

cause for a second one might be that as asc file is missing it tries to download from latest snapshot, but it might differ (latest asc with stored tar)

[ph]

@michalpristas I _believe_ the docker images are created and bundled by us as part of the beat build process, so this could explain why they do not have their asc files?

[ruflin]

The template for the docker images can be found here: https://github.com/elastic/beats/blob/master/dev-tools/packaging/templates/docker/Dockerfile.elastic-agent.tmpl

[michalpristas]

this should fix the issue
https://github.com/elastic/infra/pull/23609/files
but todays asc nor endpoint is present (even on normal darwin artifacts) so there is probably some other issue going on on infra side

[ph]

@michalpristas Do you have an issue open with them?

[michalpristas]

build is ok now, i fetched some intermediate build before postprocess job finished bundling. all good now
chris is creating an issue to make this more visible to the user or fixed, will share it once it's created

[mdelapenya]

Are the images already created? I can confirm today's builds do not fetch the images with the ASC files: https://beats-ci.elastic.co/blue/organizations/jenkins/e2e-tests%2Fe2e-testing-mbp/detail/master/340/pipeline/337

[2020-09-30T09:44:15.273Z] elastic-agent_1 | 2020-09-30T09:43:29.971Z DEBUG application/periodic.go:40 Failed to read configuration, error: could not emit configuration: operator: failed to execute step sc-run, error: operation 'operation-verify' failed to verify metricbeat.8.0.0-SNAPSHOT: 3 errors occurred:
[2020-09-30T09:44:15.273Z] elastic-agent_1 | * fetching asc file from '/usr/share/elastic-agent/data/elastic-agent-30ff4d/downloads/metricbeat-8.0.0-SNAPSHOT-linux-x86_64.tar.gz.asc': open /usr/share/elastic-agent/data/elastic-agent-30ff4d/downloads/metricbeat-8.0.0-SNAPSHOT-linux-x86_64.tar.gz.asc: no such file or directory
[2020-09-30T09:44:15.273Z] elastic-agent_1 | * check detached signature: openpgp: invalid signature: hash tag doesn't match

[ph]

Not yet, the build process need to change to accommodate that. @mgreau @Conky5

[EricDavisX]

Manu confirmed in slack today that this is fixed as far as the e2e-testing run goes. closing it out