Beats: [libbeat] Kafka Output: Add support for SASL_SSL authentication

[ronanh]

Hello,

Kafka output support SASL_SSL scheme, but only with SASL/PLAIN, which is less secure. Additional support for SASL/SCRAM-SHA256 and SASL/SCRAM-SHA512 would be nice.
This can be an important limitation when one does not have admin rights over the Kafka Brokers it's using.

Any plans regarding this feature?
Thanks

[jsoriano]

Hi @ronanh,

We don't have specific plans to support these authentication methods yet, but this is something we will definitely consider for the next versions.

[cfriaszapater]

Adding also SASL/GSSAPI mechanism would be useful, thanks.

[MaximeGuerain]

Hello,
Since Shopify/sarama add SASL SCRAM-SHA-512 and SCRAM-SHA-256 mechanismes.
Is it possible to add it for the next release ?
Thanks.

[arnitolog]

Hi there,
Is there any update on this issue. We also need to have an ability to send data from Filebeat to Kafka with SASL/SCRAM-SHA512

[sarwarbhuiyan]

+1

[jsoriano]

Initial support for SASL/SCRAM-SHA256/512 has been merged into master (https://github.com/elastic/beats/pull/12867), I have created a follow up issue for pending things: https://github.com/elastic/beats/issues/16723

It'd be also nice if someone else could test this in their deployments to see if we are missing something before backporting it for an upcoming release.

[seawolf888]

Hi there,
May I know when this feature can be released officially?

[trevorprater]

Thank you for the implementation! Does anyone have an ETA for the release, by chance? Much appreciated.

[DavidIans]

Why does my 7.10 version of Filebeat still not support SASL/SCRAM-SHA-256？
This is the Filebeat output configuration:

output.kafka:
  enabled: true
  hosts: ["broker1:9093"]
  topic: 'test'
  username: user_rw
  password: davidrw
  mechanism: SCRAM-SHA-256

This is the filebeat log(debug):

2020-11-16T18:59:38.652+0800    DEBUG   [kafka] kafka/client.go:277     finished kafka batch
2020-11-16T18:59:38.652+0800    DEBUG   [kafka] kafka/client.go:291     Kafka publish failed with: circuit breaker is open
2020-11-16T18:59:38.653+0800    INFO    [publisher]     pipeline/retry.go:219   retryer: send unwait signal to consumer
2020-11-16T18:59:38.653+0800    INFO    [publisher]     pipeline/retry.go:223     done
2020-11-16T18:59:38.688+0800    DEBUG   [kafka] kafka/client.go:277     finished kafka batch
2020-11-16T18:59:38.688+0800    DEBUG   [kafka] kafka/client.go:291     Kafka publish failed with: circuit breaker is open
2020-11-16T18:59:38.688+0800    INFO    [publisher]     pipeline/retry.go:219   retryer: send unwait signal to consumer
2020-11-16T18:59:38.688+0800    INFO    [publisher]     pipeline/retry.go:223     done

This is the Kafka log:

2020-11-16 18:59:30,650 INFO org.apache.kafka.common.network.Selector: [SocketServer brokerId=1559] Failed authentication with /10.31.73.12 (Unsupported SASL mechanism PLAIN)
2020-11-16 18:59:32,155 INFO org.apache.kafka.common.network.Selector: [SocketServer brokerId=1559] Failed authentication with /10.31.73.12 (Unsupported SASL mechanism PLAIN)
2020-11-16 18:59:33,909 INFO org.apache.kafka.common.network.Selector: [SocketServer brokerId=1559] Failed authentication with /10.31.73.12 (Unsupported SASL mechanism PLAIN)
2020-11-16 18:59:35,665 INFO org.apache.kafka.common.network.Selector: [SocketServer brokerId=1559] Failed authentication with /10.31.73.12 (Unsupported SASL mechanism PLAIN)
2020-11-16 18:59:37,420 INFO org.apache.kafka.common.network.Selector: [SocketServer brokerId=1559] Failed authentication with /10.31.73.12 (Unsupported SASL mechanism PLAIN)

Why is it still PLAIN？

[jsoriano]

@DavidIans this feature hasn't been backported to 7.x branches yet, it is only available on master. Would you have the chance to test it with a build from master branch? Builds from master branch can be found here: https://github.com/elastic/beats/#snapshots

[DavidIans]

@jsoriano ok,thank you for that,I'm trying to build from source

[MacPower]

Why does my 7.10 version of Filebeat still not support SASL/SCRAM-SHA-256？
This is the Filebeat output configuration:

output.kafka:
  enabled: true
  hosts: ["broker1:9093"]
  topic: 'test'
  username: user_rw
  password: davidrw
  mechanism: SCRAM-SHA-256

This is the filebeat log(debug):

2020-11-16T18:59:38.652+0800    DEBUG   [kafka] kafka/client.go:277     finished kafka batch
2020-11-16T18:59:38.652+0800    DEBUG   [kafka] kafka/client.go:291     Kafka publish failed with: circuit breaker is open
2020-11-16T18:59:38.653+0800    INFO    [publisher]     pipeline/retry.go:219   retryer: send unwait signal to consumer
2020-11-16T18:59:38.653+0800    INFO    [publisher]     pipeline/retry.go:223     done
2020-11-16T18:59:38.688+0800    DEBUG   [kafka] kafka/client.go:277     finished kafka batch
2020-11-16T18:59:38.688+0800    DEBUG   [kafka] kafka/client.go:291     Kafka publish failed with: circuit breaker is open
2020-11-16T18:59:38.688+0800    INFO    [publisher]     pipeline/retry.go:219   retryer: send unwait signal to consumer
2020-11-16T18:59:38.688+0800    INFO    [publisher]     pipeline/retry.go:223     done

This is the Kafka log:

2020-11-16 18:59:30,650 INFO org.apache.kafka.common.network.Selector: [SocketServer brokerId=1559] Failed authentication with /10.31.73.12 (Unsupported SASL mechanism PLAIN)
2020-11-16 18:59:32,155 INFO org.apache.kafka.common.network.Selector: [SocketServer brokerId=1559] Failed authentication with /10.31.73.12 (Unsupported SASL mechanism PLAIN)
2020-11-16 18:59:33,909 INFO org.apache.kafka.common.network.Selector: [SocketServer brokerId=1559] Failed authentication with /10.31.73.12 (Unsupported SASL mechanism PLAIN)
2020-11-16 18:59:35,665 INFO org.apache.kafka.common.network.Selector: [SocketServer brokerId=1559] Failed authentication with /10.31.73.12 (Unsupported SASL mechanism PLAIN)
2020-11-16 18:59:37,420 INFO org.apache.kafka.common.network.Selector: [SocketServer brokerId=1559] Failed authentication with /10.31.73.12 (Unsupported SASL mechanism PLAIN)

Why is it still PLAIN？

Hi,
I have tested this with the filebeat-8.0.0-SNAPSHOT,

you have to actually set sasl object first:

output.kafka:
  enabled: true
  hosts: ["broker1:9093"]
  topic: 'test'
  username: user_rw
  password: davidrw
  sasl.mechanism: SCRAM-SHA-256

It worked for me with SCRAM-SHA-512 with Amazon MSK service. I have pushed this issue at the support for backporting.

Hopes it helps.

[faraya3m]

@MacPower I tried to use SCRAM-SHA-512 with Amazon MSK service and I'm also using the filebeat-8.0.0-SNAPSHOT, but is not working for me. I'm getting the same type of error:
2020-11-16T18:59:38.652+0800 DEBUG [kafka] kafka/client.go:291 Kafka publish failed with: circuit breaker is open 2020-11-16T18:59:38.653+0800 INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer 2020-11-16T18:59:38.653+0800 INFO [publisher] pipeline/retry.go:223 done
I'm using Amazon MSK 2.2.1. What version are you using? Do you had to specify the Kafka version on the filebeat configuration?
Can you please share your Amazon MSK configuration setup and the filebeat yml you used to make it work? I would really appreciate it. Thank you.

[trevorprater]

Try using SCRAM-SHA-256. It definitely works.

On Thu, Dec 24, 2020 at 11:17 fernando-araya-acn [e-mail]
wrote:

@MacPower https://github.com/MacPower I tried to use SCRAM-SHA-512 with
Amazon MSK service and I'm also using the filebeat-8.0.0-SNAPSHOT, but is
not working for me. I'm getting the same type of error:
2020-11-16T18:59:38.652+0800 DEBUG [kafka] kafka/client.go:291 Kafka
publish failed with: circuit breaker is open 2020-11-16T18:59:38.653+0800
INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to
consumer 2020-11-16T18:59:38.653+0800 INFO [publisher]
pipeline/retry.go:223 done
I'm using Amazon MSK 2.2.1. What version are you using? Do you had to
specify the Kafka version on the filebeat configuration?
Can you please share your Amazon MSK configuration setup and the filebeat
yml you used to make it work? I would really appreciate it. Thank you.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/elastic/beats/issues/8387#issuecomment-750920502, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AAJY34DRGJD54RKSLDPKXD3SWNSTPANCNFSM4FWQBQZA
.

[faraya3m]

I tried but “Amazon MSK only supports SCRAM-SHA-512 authentication.”

[MacPower]

@MacPower I tried to use SCRAM-SHA-512 with Amazon MSK service and I'm also using the filebeat-8.0.0-SNAPSHOT, but is not working for me. I'm getting the same type of error:
2020-11-16T18:59:38.652+0800 DEBUG [kafka] kafka/client.go:291 Kafka publish failed with: circuit breaker is open 2020-11-16T18:59:38.653+0800 INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer 2020-11-16T18:59:38.653+0800 INFO [publisher] pipeline/retry.go:223 done
I'm using Amazon MSK 2.2.1. What version are you using? Do you had to specify the Kafka version on the filebeat configuration?
Can you please share your Amazon MSK configuration setup and the filebeat yml you used to make it work? I would really appreciate it. Thank you.

I have used 2.4.1.1 Kafka version with MSK, and the I have already shared the output of filebeat.

[jsoriano]

SASL/SCRAM-SHA256 and SASL/SCRAM-SHA512 will be supported in 7.11. I am closing this by now.

@MacPower have you tried to use sasl.mechanism instead of mechanism in your config? Could you please open a new topic in https://discuss.elastic.co/c/elastic-stack/beats/28 if you still have issues?

@faraya3m how is the config you are using?

[MacPower]

SASL/SCRAM-SHA256 and SASL/SCRAM-SHA512 will be supported in 7.11. I am closing this by now.

@MacPower have you tried to use sasl.mechanism instead of mechanism in your config? Could you please open a new topic in https://discuss.elastic.co/c/elastic-stack/beats/28 if you still have issues?

@faraya3m how is the config you are using?

Nice work thanks !

@jsoriano I am using sasl.mechanism and it worked perfectly on 8.0.0-SNAPSHOT.