Charts: standard_init_linux.go:178: exec user process caused "no such file or directory
$ kubectl --namespace=default get pods -l "app=fluentd-cloudwatch,release=fluentd-cw"
NAME READY STATUS RESTARTS AGE
fluentd-cw-fluentd-cloudwatch-cw5cm 0/1 CrashLoopBackOff 1 22s
fluentd-cw-fluentd-cloudwatch-zplcp 0/1 Error 2 22s
$ kubectl logs -f fluentd-cw-fluentd-cloudwatch-zplcp
standard_init_linux.go:178: exec user process caused "no such file or directory
please help
rajkumar49
👍1
All 24 comments
Also, I want to send logs to AWS cloudwatch, I am using the kube2iam pods in kubernetes. how to give access to AWS to create AWS role automatically by Kube2iam?
rajkumar49
on 17 Jul 2018
Seeing this same issue. Is this related to the latest release of busybox?
swibrow
on 18 Jul 2018
Add this to your values file
image:
repository: fluent/fluentd-kubernetes-daemonset
tag: v0.12.43-cloudwatch
https://github.com/fluent/fluentd-kubernetes-daemonset/issues/164
swibrow
on 18 Jul 2018
2018-07-18 09:18:58 +0000 [error]: unexpected error error_class=Errno::EACCES error=#
2018-07-18 09:18:58 +0000 [error]: /fluentd/vendor/bundle/ruby/2.4.0/gems/fluentd-0.12.43/lib/fluent/plugin/in_tail.rb:145:in initialize'
2018-07-18 09:18:58 +0000 [error]: /fluentd/vendor/bundle/ruby/2.4.0/gems/fluentd-0.12.43/lib/fluent/plugin/in_tail.rb:145:inopen'
2018-07-18 09:18:58 +0000 [error]: /fluentd/vendor/bundle/ruby/2.4.0/gems/fluentd-0.12.43/lib/fluent/plugin/in_tail.rb:145:in start'
2018-07-18 09:18:58 +0000 [error]: /fluentd/vendor/bundle/ruby/2.4.0/gems/fluentd-0.12.43/lib/fluent/root_agent.rb:115:inblock in start'
2018-07-18 09:18:58 +0000 [error]: /fluentd/vendor/bundle/ruby/2.4.0/gems/fluentd-0.12.43/lib/fluent/root_agent.rb:114:in each'
2018-07-18 09:18:58 +0000 [error]: /fluentd/vendor/bundle/ruby/2.4.0/gems/fluentd-0.12.43/lib/fluent/root_agent.rb:114:instart'
2018-07-18 09:18:58 +0000 [error]: /fluentd/vendor/bundle/ruby/2.4.0/gems/fluentd-0.12.43/lib/fluent/engine.rb:237:in start'
2018-07-18 09:18:58 +0000 [error]: /fluentd/vendor/bundle/ruby/2.4.0/gems/fluentd-0.12.43/lib/fluent/engine.rb:187:inrun'
2018-07-18 09:18:58 +0000 [error]: /fluentd/vendor/bundle/ruby/2.4.0/gems/fluentd-0.12.43/lib/fluent/supervisor.rb:570:in run_engine'
2018-07-18 09:18:58 +0000 [error]: /fluentd/vendor/bundle/ruby/2.4.0/gems/fluentd-0.12.43/lib/fluent/supervisor.rb:162:inblock in start'
2018-07-18 09:18:58 +0000 [error]: /fluentd/vendor/bundle/ruby/2.4.0/gems/fluentd-0.12.43/lib/fluent/supervisor.rb:366:in main_process'
2018-07-18 09:18:58 +0000 [error]: /fluentd/vendor/bundle/ruby/2.4.0/gems/fluentd-0.12.43/lib/fluent/supervisor.rb:339:inblock in supervise'
2018-07-18 09:18:58 +0000 [error]: /fluentd/vendor/bundle/ruby/2.4.0/gems/fluentd-0.12.43/lib/fluent/supervisor.rb:338:in fork'
2018-07-18 09:18:58 +0000 [error]: /fluentd/vendor/bundle/ruby/2.4.0/gems/fluentd-0.12.43/lib/fluent/supervisor.rb:338:insupervise'
2018-07-18 09:18:58 +0000 [error]: /fluentd/vendor/bundle/ruby/2.4.0/gems/fluentd-0.12.43/lib/fluent/supervisor.rb:156:in start'
2018-07-18 09:18:58 +0000 [error]: /fluentd/vendor/bundle/ruby/2.4.0/gems/fluentd-0.12.43/lib/fluent/command/fluentd.rb:173:inrequire'
2018-07-18 09:18:58 +0000 [error]: /usr/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:inrequire'
2018-07-18 09:18:58 +0000 [error]: /fluentd/vendor/bundle/ruby/2.4.0/gems/fluentd-0.12.43/bin/fluentd:8:in <top (required)>'
2018-07-18 09:18:58 +0000 [error]: /fluentd/vendor/bundle/ruby/2.4.0/bin/fluentd:23:inload'
2018-07-18 09:18:58 +0000 [error]: /fluentd/vendor/bundle/ruby/2.4.0/bin/fluentd:23:in `
2018-07-18 09:18:58 +0000 [info]: shutting down fluentd
2018-07-18 09:18:58 +0000 [info]: shutting down filter type="kubernetes_metadata" plugin_id="object:2aace451889c"
2018-07-18 09:18:58 +0000 [info]: shutting down output type="null" plugin_id="object:2aace3de15d8"
2018-07-18 09:18:58 +0000 [info]: shutting down output type="cloudwatch_logs" plugin_id="object:2aace4ac8d64"
2018-07-18 09:18:58 +0000 [info]: process finished code=0
2018-07-18 09:18:58 +0000 [error]: fluentd main process died unexpectedly. restarting.
rajkumar49
on 18 Jul 2018
hi @swibrow ,
i changed as above with tag: v0.12.43-cloudwatch
getting above error
rajkumar49
on 18 Jul 2018
hi @swibrow , please update the "fluentd-cloudwatch" incubator repo with the correct docker image tag.
rajkumar49
on 18 Jul 2018
Hi @rajkumar49, not too sure about the error you're getting now. Are you using all the chart defaults?
Regarding the PR, I've opened one with the updated tag.
swibrow
on 18 Jul 2018
@swibrow , i have used both options - 1. all defaults with --set awsRole flag , 2. with values.yaml with small edits like aws region , aws log group name etc...
for example :
helm install --name fluentd-cloudwatch2 --set awsRole=CloudwatchPodLogs incubator/fluentd-cloudwatch
rajkumar49
on 18 Jul 2018
awsRole needs to be the role ARN eg: arn:aws:iam::1234567890:role/fluentd-cloudwatch which also requires kube2iam running in your cluster https://github.com/kubernetes/charts/tree/master/stable/kube2iam
swibrow
on 18 Jul 2018
hi @swibrow , i have used the full ARN of role in the --set awsRole flag , also i am using the kube2iam with my AWS accesskey and secret access key.
still getting this below ERROR:
rajkumar49
on 18 Jul 2018
I'm also seeing these errors but the logs are still being forwarded. I will look into it in the next days.
swibrow
on 19 Jul 2018
@swibrow , thanks . in my case the fluentd process is getting shut down.
rajkumar49
on 19 Jul 2018
So a temp fix is to run the fluentd user as root.
add this line to your values file
extraVars:
- "{ name: FLUENT_UID, value: '0' }"
this should get you up and running.
swibrow
on 19 Jul 2018
👍2
@swibrow please help to solve this . the extraVars above helped to solve the permission denied error.
rajkumar49
on 19 Jul 2018
Seems like your IAM role doesn't have access to Cloudwatch
swibrow
on 19 Jul 2018
Hi @swibrow , thanks , i given the Cloudwatch access permission to that role , now i can see logs in cloudwatch logs .
rajkumar49
on 19 Jul 2018
👍1
@rajkumar49 I guess you should close this issue now.
swibrow
on 25 Jul 2018
thanks for the help @swibrow
rajkumar49
on 25 Jul 2018
@rajkumar49 I'm dealing with error error_class="Aws::Errors::MissingCredentialsError" error="unable to sign request without credentials set" right now. I've already added the following:
image:
repository: fluent/fluentd-kubernetes-daemonset
tag: v0.12.43-cloudwatch
extraVars:
- "{ name: FLUENT_UID, value: '0' }"
awsRole: arn:aws:iam::xxxxxx:role/k8s-nodes-fluentd
awsRegion: us-east-1
logGroupName: kubernetes-logs
rbac:
create: true
Can you describe (censoring account id) here how your IAM policy and IAM role looks like that you used for kube2iam?
Here is the __IAM Role__:
{
"Role": {
"Path": "/",
"RoleName": "k8s-nodes-fluentd",
"RoleId": "xxxx",
"Arn": "arn:aws:iam::xxxx:role/k8s-nodes-fluentd",
"CreateDate": "2018-07-25T15:30:21Z",
"AssumeRolePolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "ec2.amazonaws.com"
},
"Action": "sts:AssumeRole"
},
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::xxxx:role/k8s-nodes-fluentd"
},
"Action": "sts:AssumeRole"
}
]
},
"MaxSessionDuration": 3600
}
}
And the attached __IAM policy__ to it:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"logs:DescribeLogGroups"
],
"Resource": [
"arn:aws:logs:us-east-1:xxxx:log-group::log-stream:*"
]
},
{
"Effect": "Allow",
"Action": [
"logs:DescribeLogStreams"
],
"Resource": [
"arn:aws:logs:us-east-1:xxxx:log-group:kubernetes-logs:log-stream:*"
]
},
{
"Effect": "Allow",
"Action": [
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Resource": [
"arn:aws:logs:us-east-1:xxxx:log-group:kubernetes-logs:log-stream:fluentd-cloudwatch"
]
}
]
}
I can also verify that kube2iam has placed the IAM role in the pod:
# inside the daemonset pod
wget http://169.254.169.254/latest/meta-data/iam/security-credentials/ && cat index.html
arn:aws:iam::xxxx:role/k8s-nodes-fluentd
bzon
on 25 Jul 2018
Hi,
No need to use kube 2 IAM. Just configure in values.yaml of fluent-
cloudwatch repo.
thanks,
Rajkumar Selvaraj,
On Wed, Jul 25, 2018, 10:19 PM John Bryan Sazon notifications@github.com
wrote:
@rajkumar49 https://github.com/rajkumar49 I'm dealing with error error_class="Aws::Errors::MissingCredentialsError"
error="unable to sign request without credentials set" right now..Can you describe (censoring account id) here how your IAM policy and IAM
role looks like that you used for kube2iam?—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/helm/charts/issues/6669#issuecomment-407821601, or mute
the thread
https://github.com/notifications/unsubscribe-auth/APlrLg9CsNjvWGPDM92o1x2ynVcY5gQRks5uKKGXgaJpZM4VSXCk
.
rajkumar49
on 26 Jul 2018
@rajkumar49, do you mean just use AWS variables or disable kube2iam?
My other apps/pods uses kube2iam..
bzon
on 26 Jul 2018
@bzon
Here is our role we use with kube2iam
{
"Role": {
"Path": "/",
"RoleName": "fluentd-cloudwatch-k8s",
"RoleId": "XXXXXXXXXXXXXXXXX",
"Arn": "arn:aws:iam::111111111111:role/fluentd-cloudwatch-k8s",
"CreateDate": "2018-03-14T11:01:44Z",
"AssumeRolePolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"Service": "ec2.amazonaws.com"
},
"Action": "sts:AssumeRole"
},
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"AWS": [
"arn:aws:iam::111111111111:role/nodes.k8s.cluster1",
"arn:aws:iam::111111111111:role/nodes.k8s.cluster2",
"arn:aws:iam::111111111111:role/nodes.k8s.cluster3",
"arn:aws:iam::111111111111:role/nodes.k8s.cluster4",
"arn:aws:iam::111111111111:role/nodes.k8s.cluster5",
"arn:aws:iam::111111111111:role/nodes.k8s.cluster6"
]
},
"Action": "sts:AssumeRole"
}
]
},
"Description": "Role for fluentd in k8s",
"MaxSessionDuration": 3600
}
}
Start with this policy to see if it is a permissions issue. Fluentd will create multiple log streams.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"logs:*"
],
"Resource": [
"arn:aws:logs:eu-west-1:*:*"
]
}
]
}
Thanks a lot @swibrow I just realised that I have a __wrong Trust Relationship__ in the IAM role that I used for the fluentd daemonset.
_AssumeRolePolicyDocument.Principal.AWS_ __should be the IAM role of the kubernetes cluster nodes__ and __not the IAM role of the fluentd pod__..
bzon
on 26 Jul 2018
👍1
@bzon No worries
swibrow
on 26 Jul 2018
🎉1
Was this page helpful?
0 / 5 - 0 ratings
Related issues
paskal
·
3Comments
richardalberto
·
3Comments
out-of-band
·
3Comments
harshmehta93
·
3Comments
onyxet
·
3Comments
Most helpful comment
So a temp fix is to run the fluentd user as root.
add this line to your values file
this should get you up and running.