Fluentd-kubernetes-daemonset: Can't connect to AWS ES

[ixai]

I have an AWS ES instance protected by an IAM access policy. I added all my cluster nodes to said policy (by IP). Since I'm not using X-Pack, I figured I had to set the values for FLUENT_ELASTICSEARCH_USER and FLUENT_ELASTICSEARCH_PASSWORD to blank in the DaemonSet config file (fluentd-daemonset-elasticsearch.yml). Still, fluentd was unable to connect to ES:

2017-06-13 17:18:32 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2017-06-13 17:19:02 +0000 error_class="Elasticsearch::Transport::Transport::Errors::Forbidden" error="[403] " plugin_id="object:2ac3e37a42a8"

To confirm that my IAM policy was correctly set up, I curl'd my ES host from a fluentd pod and I got a successful response.

/home/fluent # curl https://<redacted>.us-west-2.es.amazonaws.com:443
{
  "name" : "Crimson Cavalier",
  "cluster_name" : "<redacted>",
  "version" : {
    "number" : "2.3.2",
    "build_hash" : "72aa8010df1a4fc849da359c9c58acba6c4d9518",
    "build_timestamp" : "2016-11-14T15:59:50Z",
    "build_snapshot" : false,
    "lucene_version" : "5.5.0"
  },
  "tagline" : "You Know, for Search"
}

The solution I found was to manually edit the /fluentd/etc/fluent.conf file, remove the user and password lines and start the fluentd service again.

-   user "#{ENV['FLUENT_ELASTICSEARCH_USER']}"
-   password "#{ENV['FLUENT_ELASTICSEARCH_PASSWORD']}"

---

I'm not sure what's the best way to get this into the code though.

[edsiper]

are you connecting directly or through a proxy that implements the IAM handling ?

[ixai]

I'm connecting directly.

Playing with the fluent-bit daemonset I learned I could add a k8s ConfigMap and change my daemonset to add it as a volume, so I could do the same here.

I guess my issue was that I wanted a way for the default image to know when FLUENT_ELASTICSEARCH_USER and FLUENT_ELASTICSEARCH_PASSWORD were not set and remove it from the erb template. But maybe that's too specific of a use-case?

[dileepfrog]

Can we please reopen this, PR #40 was meant to fix but as it turns out is not correct

[zopanix]

@ixai: hey, I'm also using aws es and I have it working using this: https://hub.docker.com/r/fxinnovation/aws-es-proxy/

It is a small authentication proxy to whom you can give the account credentials for accessing AWS ES and inside your cluster you define this service a being elastic search. For fluentd, heapster, etc it is completely transparent and it leverages AWS IAM auth for you. But you can't use the FLUENT_ELASTICSEARCH_USER and FLUENT_ELASTICSEARCH_PASSWORD as these are elasticsearch specific.

By the way, I have 5 of those proxies launched otherwhize you might get throttle by AWS...

[ixai]

@zopanix thanks for the info, I'll definitely check it out.

[glesperance]

Instead of using a proxy or modifying the config based on ENV variables at run-time ; wouldn't it be simpler to just create a child Docker image removing the lines in /fluentd/etc/fluent.conf like OP did manually?

[glesperance]

Alternatively ; an image supporting type aws-elasticsearch-service could be created specifically for the AWS use case

[zopanix]

Sure everybody can make an image based on this one and add the plugin. Only, it adds an other image to maintain. I currently maintain 20-25 images up to date. And I'd like to keep this to a minimum. Additionally, if you use AWS ES, fluentd wont be the only thing trying to access it. I currently have 3-4 apps in total that go through that proxy without me needing to add a new image to maintain in my list. I'm my case I benefit from having this image. Security wise it also means that only that proxy has my AWS access which makes it simpler in general. I'm not saying that it's a bad idea. Both systems have their advantages...

[mooperd]

+1

[max-rocket-internet]

Hey,
I had this same error:

fluentd Elasticsearch::Transport::Transport::Errors::Forbidden error=[403]

It was a problem with the AWS access keys. I am using the docker image cheungpat/fluentd-elasticsearch-aws:1.22 and getting the values for AWS_SECRET_ACCESS_KEY and AWS_ACCESS_KEY_ID from a k8s secret like this:

- name: AWS_ACCESS_KEY_ID
  valueFrom:
    secretKeyRef:
      name: logging-iam-user-staging
      key: aws_access_key_id
- name: AWS_SECRET_ACCESS_KEY
  valueFrom:
    secretKeyRef:
      name: logging-iam-user-staging
      key: aws_secret_access_key

The problem for me was that there was quotes around the access key.

It's worth noting that if you just see the error=[403] message with no further details then authentication wasn't successful. In this scenario it's nothing to do with IAM policy as it hasn't even got that far. If you have a problem with your IAM policy you should get a more specific error like this:

2017-10-27 13:49:23 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2017-10-27 13:49:23 +0000 error_class="Elasticsearch::Transport::Transport::Errors::Forbidden" error="[403] {\"Message\":\"User: arn:aws:iam::99999999999:user/staging-logs-es is not authorized to perform: es:ESHttpPost on resource: staging-logs\"}

[toddams]

Any update on this?

[edsiper]

@toddams which help specifically do you need ?

are we OK to close this one since it's a setup issue ?, should we remove auth from plugin config ?

[mooperd]

AWS ES does not have any user auth. To make this work I had to make a
customised container with all the user / password stuff disabled to allow
connection to aws ES. Was annoying!

On 22 Jan 2018 4:59 PM, "Eduardo Silva" [e-mail] wrote:

@toddams https://github.com/toddams which help specifically do you need ?

are we OK to close this one since it's a setup issue ?, should we remove
auth from plugin config ?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/fluent/fluentd-kubernetes-daemonset/issues/33#issuecomment-359469575,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AC3exTIztbhFsCrwn7tIDr47Tcg1nhIlks5tNLBogaJpZM4N42P_
.

[edsiper]

@mooperd are you deploying in Kubernetes ?

[mooperd]

@edsiper Yes.

[onlydole]

This is working since this PR was merged in: https://github.com/fluent/fluentd-kubernetes-daemonset/pull/54

[jicowan]

Curious if anyone has tried this using kube2iam. I've tried, but I still get the 403 error.