Charts: [incubator/fluentd-cloudwatch] AWS authorization errors

[tbtommyb]

Is this a request for help?: Yes

---

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT

Version of Helm and Kubernetes:

Client: &version.Version{SemVer:"v2.8.2", GitCommit:"a80231648a1473929271764b920a8e346f6de844", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.8.2", GitCommit:"a80231648a1473929271764b920a8e346f6de844", GitTreeState:"clean"}

Kubernetes 1.10

Which chart: incubator/fluentd-cloudwatch

What happened: Fluentd does not seem to be able to authorize with AWS.

With my AWS access key and secret key added in the secrets directory:

$ helm install --set awsRegion=eu-west-1 --set rbac.create=true --set image.tag=v0.12.33-cloudwatch@sha256:0a6763c174ac9456ae3b71ae4485ff5f9ab7ecd5a1542e71248a72c54666f02c ./workspace/charts/incubator/fluentd-cloudwatch

2018-05-29 14:52:14 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2018-05-29 14:52:15 +0000 error_class="Aws::CloudWatchLogs::Errors::IncompleteSignatureException" error="Authorization header requires 'Signature' parameter. Authorization header requires 'SignedHeaders' parameter. Authorization=AWS4-HMAC-SHA256 Credential=[redacted]" plugin_id="object:2b10a5734ca0"
  2018-05-29 14:52:14 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/aws-sdk-core-2.10.50/lib/seahorse/client/plugins/raise_response_errors.rb:15:in `call'
  2018-05-29 14:52:14 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/aws-sdk-core-2.10.50/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:20:in `call'
  2018-05-29 14:52:14 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/aws-sdk-core-2.10.50/lib/aws-sdk-core/plugins/idempotency_token.rb:18:in `call'
  2018-05-29 14:52:14 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/aws-sdk-core-2.10.50/lib/aws-sdk-core/plugins/param_converter.rb:20:in `call'
  2018-05-29 14:52:14 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/aws-sdk-core-2.10.50/lib/aws-sdk-core/plugins/response_paging.rb:26:in `call'
  2018-05-29 14:52:14 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/aws-sdk-core-2.10.50/lib/seahorse/client/plugins/response_target.rb:21:in `call'
  2018-05-29 14:52:14 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/aws-sdk-core-2.10.50/lib/seahorse/client/request.rb:70:in `send_request'
  2018-05-29 14:52:14 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/aws-sdk-core-2.10.50/lib/seahorse/client/base.rb:207:in `block (2 levels) in define_operation_methods'
  2018-05-29 14:52:14 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluent-plugin-cloudwatch-logs-0.4.0/lib/fluent/plugin/out_cloudwatch_logs.rb:298:in `log_group_exists?'
  2018-05-29 14:52:14 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluent-plugin-cloudwatch-logs-0.4.0/lib/fluent/plugin/out_cloudwatch_logs.rb:121:in `block in write'
  2018-05-29 14:52:14 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluent-plugin-cloudwatch-logs-0.4.0/lib/fluent/plugin/out_cloudwatch_logs.rb:113:in `each'
  2018-05-29 14:52:14 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluent-plugin-cloudwatch-logs-0.4.0/lib/fluent/plugin/out_cloudwatch_logs.rb:113:in `write'
  2018-05-29 14:52:14 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.33/lib/fluent/buffer.rb:354:in `write_chunk'
  2018-05-29 14:52:14 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.33/lib/fluent/buffer.rb:333:in `pop'
  2018-05-29 14:52:14 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.33/lib/fluent/output.rb:342:in `try_flush'
  2018-05-29 14:52:14 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.33/lib/fluent/output.rb:149:in `run'

With kube2iam installed and using the awsRole flag:

$ helm install --set awsRegion=eu-west-1 --set rbac.create=true --set image.tag=v0.12.33-cloudwatch@sha256:0a6763c174ac9456ae3b71ae4485ff5f9ab7ecd5a1542e71248a72c54666f02c --set awsRole=ecs_rw incubator/fluentd-cloudwatch

2018-05-29 14:09:25 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2018-05-29 14:09:26 +0000 error_class="Aws::Errors::MissingCredentialsError" error="unable to sign request without credentials set" plugin_id="object:2ab1696c42fc"
  2018-05-29 14:09:25 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/aws-sdk-core-2.10.50/lib/aws-sdk-core/plugins/request_signer.rb:104:in `require_credentials'
  2018-05-29 14:09:25 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/aws-sdk-core-2.10.50/lib/aws-sdk-core/plugins/request_signer.rb:94:in `sign_authenticated_requests'
  2018-05-29 14:09:25 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/aws-sdk-core-2.10.50/lib/aws-sdk-core/plugins/request_signer.rb:87:in `call'
  2018-05-29 14:09:25 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/aws-sdk-core-2.10.50/lib/aws-sdk-core/plugins/helpful_socket_errors.rb:10:in `call'
  2018-05-29 14:09:25 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/aws-sdk-core-2.10.50/lib/aws-sdk-core/plugins/retry_errors.rb:89:in `call'
  2018-05-29 14:09:25 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/aws-sdk-core-2.10.50/lib/aws-sdk-core/json/handler.rb:11:in `call'
  2018-05-29 14:09:25 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/aws-sdk-core-2.10.50/lib/aws-sdk-core/plugins/user_agent.rb:12:in `call'
  2018-05-29 14:09:25 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/aws-sdk-core-2.10.50/lib/seahorse/client/plugins/endpoint.rb:41:in `call'
  2018-05-29 14:09:25 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/aws-sdk-core-2.10.50/lib/aws-sdk-core/plugins/param_validator.rb:21:in `call'
  2018-05-29 14:09:25 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/aws-sdk-core-2.10.50/lib/seahorse/client/plugins/raise_response_errors.rb:14:in `call'
  2018-05-29 14:09:25 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/aws-sdk-core-2.10.50/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:20:in `call'
  2018-05-29 14:09:25 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/aws-sdk-core-2.10.50/lib/aws-sdk-core/plugins/idempotency_token.rb:18:in `call'
  2018-05-29 14:09:25 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/aws-sdk-core-2.10.50/lib/aws-sdk-core/plugins/param_converter.rb:20:in `call'
  2018-05-29 14:09:25 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/aws-sdk-core-2.10.50/lib/aws-sdk-core/plugins/response_paging.rb:26:in `call'
  2018-05-29 14:09:25 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/aws-sdk-core-2.10.50/lib/seahorse/client/plugins/response_target.rb:21:in `call'
  2018-05-29 14:09:25 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/aws-sdk-core-2.10.50/lib/seahorse/client/request.rb:70:in `send_request'
  2018-05-29 14:09:25 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/aws-sdk-core-2.10.50/lib/seahorse/client/base.rb:207:in `block (2 levels) in define_operation_methods'
  2018-05-29 14:09:25 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluent-plugin-cloudwatch-logs-0.4.0/lib/fluent/plugin/out_cloudwatch_logs.rb:298:in `log_group_exists?'
  2018-05-29 14:09:25 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluent-plugin-cloudwatch-logs-0.4.0/lib/fluent/plugin/out_cloudwatch_logs.rb:121:in `block in write'
  2018-05-29 14:09:25 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluent-plugin-cloudwatch-logs-0.4.0/lib/fluent/plugin/out_cloudwatch_logs.rb:113:in `each'
  2018-05-29 14:09:25 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluent-plugin-cloudwatch-logs-0.4.0/lib/fluent/plugin/out_cloudwatch_logs.rb:113:in `write'
  2018-05-29 14:09:25 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.33/lib/fluent/buffer.rb:354:in `write_chunk'
  2018-05-29 14:09:25 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.33/lib/fluent/buffer.rb:333:in `pop'
  2018-05-29 14:09:25 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.33/lib/fluent/output.rb:342:in `try_flush'
  2018-05-29 14:09:25 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.33/lib/fluent/output.rb:149:in `run'

What you expected to happen: Fluentd should be able to connect correctly.

How to reproduce it (as minimally and precisely as possible): See commands used above.

Anything else we need to know: My AWS user account has permissions to view and create Cloudwatch log groups. The ecs_rw role has permissions like so:

{
 "Version": "2012-10-17",
 "Statement": [
   {
     "Effect": "Allow",
     "Action": [
       "logs:*"
     ],
     "Resource": "*"
   }
 ]
}

I am specifying the image tag as a workaround recommended here.

If this a permissions problem, is it possible to document the required permissions here? I am happy to do a PR if helpful.

[hsinhoyeh]

hi @tbtommyb, this may fix your issue. A quick workaround is to encode your AWS id/key with base64 and modify the secret file on k8s. You can find it useful

[tbtommyb]

Hey @hsinhoyeh , thanks for the suggestion. I found encoding and manually changing the secret didn't make a difference, but changing the image to v1.2 of fluentd and passing the IDs in directly as env vars works. I'm not sure what is causing the secrets problem.

[arun-esure]

hi @tbtommyb , did you pass the aws-key and secret_key raw values or base64 values in v1.2 fluentd image env vars? I'm stuck with this same problem. Passing raw values didn't work for me.

[tbtommyb]

Hi @arun-esure what I ended up having to do was create my own fluentd.yaml file from the templates in the chart and pass in the env vars directly as raw values in DaemonSet.spec.template.spec.containers.fluentd-cloudwatch.env. This isn't a great solution but I couldn't get anything involving secrets to work (raw or encoded). Perhaps I was just doing it wrong.

[hsinhoyeh]

@tbtommyb @arun-esure have you tried my patch? https://github.com/kubernetes/charts/pull/6209
It works for me well on v1.2 fluentd image

[arun-esure]

@hsinhoyeh your patch is working for me too! thanks.
Thanks for your input @tbtommyb .

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[stale[bot]]

This issue is being automatically closed due to inactivity.

[jcoletaylor]

Hi, I'm sorry to comment on an old ticket, but I don't want to open a new issue that seems so clearly related. I am getting the following error if I look at the pod logs:
error_class="Aws::Errors::MissingCredentialsError" error="unable to sign request without credentials set"
My values.yaml file looks like this:

annotations: {}

awsRegion: us-east-1
awsRole: arn:aws:iam::601762473314:role/k8s-logger
logGroupName: k8s-staging

rbac:
  create: true

extraVars:
  - "{ name: FLUENT_UID, value: '0' }"

And I am installing the helm chart like this:

helm install --name fluentd -f values.yaml incubator/fluentd-cloudwatch

I am not explicitly assigning my awsAccessKeyId or awsSecretAccessKey because I want it to use the role and not my personal account credentials, especially since the log group has already been created via the aws cli. But is it necessary that these values are set? I can't seem to find a clear answer.

Any help would really be appreciated. I'm happy to open another issue since this one is closed but I feel like it's probably not a bug, just my own confusion about what to configure.

[bhvishal9]

We are still getting this issue, fluentd reports following error when we use IAM role with kube2iam.

[warn]: #0 [out_cloudwatch_logs] failed to flush the buffer. retry_time=7 next_retry_seconds=2020-04-01 16:50:29 +0000 chunk="5a23d6fcf2f27ff0c593dd009f3c9d60" error_class=Aws::Errors::MissingCredentialsError error="unable to sign request without credentials set"

Any help is appreciated.

[dmanchikalapudi]

Did anyone figure out the issue? I have the exact same issue and I cannot get fluentd to write to cloudwatch without explicitly providing secret and key