Kibana: [Ingest Manager] Add a raw YML box when configuring the Elasticsearch output. (CA, Certificate)

[ph]

Use case:

It would be nice to be able to configure the Elasticsearch output to use custom options like certificate_authorities or custom params. This would solve the issue of the Elastic Agent and self-signed certificate.

Solution:

Add a raw YAML box to the elasticsearch output settings, it will be a free form field for the users.

We should ensure that a user cannot set the api_key there because these keys are handled by fleet.

This can be done before having a complete self signed in fleet experience see https://github.com/elastic/kibana/issues/73483

[elasticmachine]

Pinging @elastic/ingest-management (Team:Ingest Management)

[ph]

@neptunian if you can take a look?

[ph]

@jfsiii Can you take this over?

[ph]

@ruflin @jfsiii After talking to @crowens Could we just retrieve the CA from the Kibana configuration and send it to the elastic agent? This assumes that the beats can understand the raw CA.

[ruflin]

@ph If we do that it would assume all the cert files on the Agent machine are in the same location as on the server where Kibana is hosted? So not sure this would work.

In any case, I would treat this as 2 separate features. I think the YAML box is a quick way to make us move forward and if we can have the Kibana CA as default set, even better.

[blakerouse]

@ruflin I think in this case it would the the actual certificate data in the YAML. Something we need to add support for Agent side.

I don't even know if it's worth adding the ability to set it to a path. It will be very hard for that path to be the same path across all platforms. I would recommand installing the certificate as a system certificate so Agent will look in the correct location for each system.

[ph]

@ruflin Its something I've discussed a long time ago, here a way to embed certificate authorities, certificate or keys in anything that support TLS options. https://github.com/elastic/beats/pull/21179

[ph]

I don't even know if it's worth adding the ability to set it to a path. It will be very hard for that path to be the same path across all platforms. I would recommand installing the certificate as a system certificate so Agent will look in the correct location for each system.

IIRC this is not the default behavior on windows though? Also I think its better if we don't mess around with the system certificate.

[ph]

@ruflin @jfsiii For adding the YAML box in the elasticsearch output options, A user would need to add the following in the yaml box.

certificate_authorities:
  - |
    -----BEGIN CERTIFICATE-----
    MIIDCjCCAfKgAwIBAgITJ706Mu2wJlKckpIvkWxEHvEyijANBgkqhkiG9w0BAQsF
    ADAUMRIwEAYDVQQDDAlsb2NhbGhvc3QwIBcNMTkwNzIyMTkyOTA0WhgPMjExOTA2
    MjgxOTI5MDRaMBQxEjAQBgNVBAMMCWxvY2FsaG9zdDCCASIwDQYJKoZIhvcNAQEB
    BQADggEPADCCAQoCggEBANce58Y/JykI58iyOXpxGfw0/gMvF0hUQAcUrSMxEO6n
    fZRA49b4OV4SwWmA3395uL2eB2NB8y8qdQ9muXUdPBWE4l9rMZ6gmfu90N5B5uEl
    94NcfBfYOKi1fJQ9i7WKhTjlRkMCgBkWPkUokvBZFRt8RtF7zI77BSEorHGQCk9t
    /D7BS0GJyfVEhftbWcFEAG3VRcoMhF7kUzYwp+qESoriFRYLeDWv68ZOvG7eoWnP
    PsvZStEVEimjvK5NSESEQa9xWyJOmlOKXhkdymtcUd/nXnx6UTCFgnkgzSdTWV41
    CI6B6aJ9svCTI2QuoIq2HxX/ix7OvW1huVmcyHVxyUECAwEAAaNTMFEwHQYDVR0O
    BBYEFPwN1OceFGm9v6ux8G+DZ3TUDYxqMB8GA1UdIwQYMBaAFPwN1OceFGm9v6ux
    8G+DZ3TUDYxqMA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQELBQADggEBAG5D
    874A4YI7YUwOVsVAdbWtgp1d0zKcPRR+r2OdSbTAV5/gcS3jgBJ3i1BN34JuDVFw
    3DeJSYT3nxy2Y56lLnxDeF8CUTUtVQx3CuGkRg1ouGAHpO/6OqOhwLLorEmxi7tA
    H2O8mtT0poX5AnOAhzVy7QW0D/k4WaoLyckM5hUa6RtvgvLxOwA0U+VGurCDoctu
    8F4QOgTAWyh8EZIwaKCliFRSynDpv3JTUwtfZkxo6K6nce1RhCWFAsMvDZL8Dgc0
    yvgJ38BRsFOtkRuAGSf6ZUwTO8JJRRIFnpUzXflAnGivK9M13D5GEQMmIl6U9Pvk
    sxSmbIUfc2SGJGCJD4I=
    -----END CERTIFICATE-----

YAML string is seriously a pain and the format of the certificate does need to match, newline and everything, maybe its okay to do that in 7.10 to at least enable that workflow.

[ph]

This box will be editable per Agent policy since "output" are linked to a policy.

@ruflin I had a chat with John over zoom, we should probably have some "rejected" keys so they don't mess up what fleet is actually using. I see theses keys as problematics:

• api_key
• username
• password
• index
• hosts

step 1, make it work,
step 2: add the reject or accept list.

@ruflin should we be more strict and have an "accept" list instead, WDYT? With great power comes great responsibility.

See the https://www.elastic.co/guide/en/beats/filebeat/current/elasticsearch-output.html

[blakerouse]

@ph Maybe the entire YAMl block is just to much, just input fields for the certificates and keys is what should be added instead?

[ph]

@blakerouse actually we need three?

1. certificates_authorities (list) (could probably be fetched from Kibana?)
2. certificate
3. key

Concerning 1, Beats accepts a list today, but seriously is that really used in the wild, I am not even sure there is a use case for it?

[ruflin]

My suggestion would be:

1. Offer raw YAML to have a quick solution for a common problem
2. Add specific fields + still have the extended yaml option for expert
3. Add validation for fields

The part I didn't get how the raw yaml is going to mess up with the Fleet part? You mean if the api_key for example is used? Which one does win in the case of yaml, first or last? If that is what you are getting at, I would think it is ok for a first iteration.

[ph]

I've merged the dependent PR on the beats side https://github.com/elastic/beats/pull/21179

[ph]

The part I didn't get how the raw yaml is going to mess up with the Fleet part? You mean if the api_key for example is used? Which one does win in the case of yaml, first or last? If that is what you are getting at, I would think it is ok for a first iteration.

Yes, I meant that would be unexpected behavior, I presume that the logic in fleet adding the output information would win. @nchaulet

[ph]

I am +1 for solving the case with a simple first implementation like 1 like ruflin suggested.

[nchaulet]

Yes, I meant that would be unexpected behavior, I presume that the logic in fleet adding the output information would win.

Yes we add the API key just before sending the config to the agent so this would replace any previously set property

[jfsiii]

I've opened a PR at https://github.com/elastic/kibana/pull/79019

[ferullo]

@k-g-elastic @EricDavisX once this is merged we should add testing this to the routine tests for Endpoint.

[rahulgupta-qasource]

Hi Eric

We have created a testcase https://elastic.testrail.io/index.php?/cases/view/34210 for this ticket.

We will validate this ticket once the changes are merged,

[nicpenning]

@ruflin @jfsiii For adding the YAML box in the elasticsearch output options, A user would need to add the following in the yaml box.

certificate_authorities:
  - |
    -----BEGIN CERTIFICATE-----
    MIIDCjCCAfKgAwIBAgITJ706Mu2wJlKckpIvkWxEHvEyijANBgkqhkiG9w0BAQsF
    ADAUMRIwEAYDVQQDDAlsb2NhbGhvc3QwIBcNMTkwNzIyMTkyOTA0WhgPMjExOTA2
    MjgxOTI5MDRaMBQxEjAQBgNVBAMMCWxvY2FsaG9zdDCCASIwDQYJKoZIhvcNAQEB
    BQADggEPADCCAQoCggEBANce58Y/JykI58iyOXpxGfw0/gMvF0hUQAcUrSMxEO6n
    fZRA49b4OV4SwWmA3395uL2eB2NB8y8qdQ9muXUdPBWE4l9rMZ6gmfu90N5B5uEl
    94NcfBfYOKi1fJQ9i7WKhTjlRkMCgBkWPkUokvBZFRt8RtF7zI77BSEorHGQCk9t
    /D7BS0GJyfVEhftbWcFEAG3VRcoMhF7kUzYwp+qESoriFRYLeDWv68ZOvG7eoWnP
    PsvZStEVEimjvK5NSESEQa9xWyJOmlOKXhkdymtcUd/nXnx6UTCFgnkgzSdTWV41
    CI6B6aJ9svCTI2QuoIq2HxX/ix7OvW1huVmcyHVxyUECAwEAAaNTMFEwHQYDVR0O
    BBYEFPwN1OceFGm9v6ux8G+DZ3TUDYxqMB8GA1UdIwQYMBaAFPwN1OceFGm9v6ux
    8G+DZ3TUDYxqMA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQELBQADggEBAG5D
    874A4YI7YUwOVsVAdbWtgp1d0zKcPRR+r2OdSbTAV5/gcS3jgBJ3i1BN34JuDVFw
    3DeJSYT3nxy2Y56lLnxDeF8CUTUtVQx3CuGkRg1ouGAHpO/6OqOhwLLorEmxi7tA
    H2O8mtT0poX5AnOAhzVy7QW0D/k4WaoLyckM5hUa6RtvgvLxOwA0U+VGurCDoctu
    8F4QOgTAWyh8EZIwaKCliFRSynDpv3JTUwtfZkxo6K6nce1RhCWFAsMvDZL8Dgc0
    yvgJ38BRsFOtkRuAGSf6ZUwTO8JJRRIFnpUzXflAnGivK9M13D5GEQMmIl6U9Pvk
    sxSmbIUfc2SGJGCJD4I=
    -----END CERTIFICATE-----

YAML string is seriously a pain and the format of the certificate does need to match, newline and everything, maybe its okay to do that in 7.10 to at least enable that workflow.

Is it strange that in 7.10 I had to have both this format and the ca.crt file for the agent to successfully send logs?

Note: I replaced my cert with your cert here to show what I am talking about. This is what I have in the new output settings.

# YAML settings here will be added to the Elasticsearch output section of each policy
ssl.certificate_authorities: ["C:\\Program Files\\Elastic\\Agent\\ca.crt"]
certificate_authorities:
  - |
    -----BEGIN CERTIFICATE-----
    MIIDCjCCAfKgAwIBAgITJ706Mu2wJlKckpIvkWxEHvEyijANBgkqhkiG9w0BAQsF
    ADAUMRIwEAYDVQQDDAlsb2NhbGhvc3QwIBcNMTkwNzIyMTkyOTA0WhgPMjExOTA2
    MjgxOTI5MDRaMBQxEjAQBgNVBAMMCWxvY2FsaG9zdDCCASIwDQYJKoZIhvcNAQEB
    BQADggEPADCCAQoCggEBANce58Y/JykI58iyOXpxGfw0/gMvF0hUQAcUrSMxEO6n
    fZRA49b4OV4SwWmA3395uL2eB2NB8y8qdQ9muXUdPBWE4l9rMZ6gmfu90N5B5uEl
    94NcfBfYOKi1fJQ9i7WKhTjlRkMCgBkWPkUokvBZFRt8RtF7zI77BSEorHGQCk9t
    /D7BS0GJyfVEhftbWcFEAG3VRcoMhF7kUzYwp+qESoriFRYLeDWv68ZOvG7eoWnP
    PsvZStEVEimjvK5NSESEQa9xWyJOmlOKXhkdymtcUd/nXnx6UTCFgnkgzSdTWV41
    CI6B6aJ9svCTI2QuoIq2HxX/ix7OvW1huVmcyHVxyUECAwEAAaNTMFEwHQYDVR0O
    BBYEFPwN1OceFGm9v6ux8G+DZ3TUDYxqMB8GA1UdIwQYMBaAFPwN1OceFGm9v6ux
    8G+DZ3TUDYxqMA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQELBQADggEBAG5D
    874A4YI7YUwOVsVAdbWtgp1d0zKcPRR+r2OdSbTAV5/gcS3jgBJ3i1BN34JuDVFw
    3DeJSYT3nxy2Y56lLnxDeF8CUTUtVQx3CuGkRg1ouGAHpO/6OqOhwLLorEmxi7tA
    H2O8mtT0poX5AnOAhzVy7QW0D/k4WaoLyckM5hUa6RtvgvLxOwA0U+VGurCDoctu
    8F4QOgTAWyh8EZIwaKCliFRSynDpv3JTUwtfZkxo6K6nce1RhCWFAsMvDZL8Dgc0
    yvgJ38BRsFOtkRuAGSf6ZUwTO8JJRRIFnpUzXflAnGivK9M13D5GEQMmIl6U9Pvk
    sxSmbIUfc2SGJGCJD4I=
    -----END CERTIFICATE-----