Helm-charts: AccessDeniedException /usr/share/elasticsearch/data/nodes
Chart version:
version: 7.0.1-alpha1
Kubernetes version:
1.11
Kubernetes provider: E.g. GKE (Google Kubernetes Engine)
IBM cloud
Helm Version:
2.9.1
Values.yaml:
clusterName: "elasticsearch"
nodeGroup: "master"
# The service that non master groups will try to connect to when joining the cluster
# This should be set to clusterName + "-" + nodeGroup for your master group
masterService: ""
# Elasticsearch roles that will be applied to this nodeGroup
# These will be set as environment variables. E.g. node.master=true
roles:
master: "true"
ingest: "true"
data: "true"
replicas: 3
minimumMasterNodes: 2
esMajorVersion: 7
# Allows you to add any config files in /usr/share/elasticsearch/config/
# such as elasticsearch.yml and log4j2.properties
esConfig: {}
# elasticsearch.yml: |
# key:
# nestedkey: value
# log4j2.properties: |
# key = value
# Extra environment variables to append to this nodeGroup
# This will be appended to the current 'env:' key. You can use any of the kubernetes env
# syntax here
extraEnvs: []
# - name: MY_ENVIRONMENT_VAR
# value: the_value_goes_here
# A list of secrets and their paths to mount inside the pod
# This is useful for mounting certificates for security and for mounting
# the X-Pack license
secretMounts: []
# - name: elastic-certificates
# secretName: elastic-certificates
# path: /usr/share/elasticsearch/config/certs
image: docker.elastic.co/elasticsearch/elasticsearch
imageTag: 7.0.1
imagePullPolicy: IfNotPresent
podAnnotations: {}
# iam.amazonaws.com/role: es-cluster
esJavaOpts: "-Xmx1g -Xms1g"
resources:
requests:
cpu: "100m"
memory: "2Gi"
limits:
cpu: "1000m"
memory: "2Gi"
initResources: {}
# limits:
# cpu: "25m"
# # memory: "128Mi"
# requests:
# cpu: "25m"
# memory: "128Mi"
networkHost: "0.0.0.0"
volumeClaimTemplate:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 30Gi
persistence:
enabled: true
annotations: {}
extraVolumes: []
# - name: extras
# emptyDir: {}
extraVolumeMounts: []
# - name: extras
# mountPath: /usr/share/extras
# readOnly: true
extraInitContainers: []
# - name: do-something
# image: busybox
# command: ['do', 'something']
# This is the PriorityClass settings as defined in
# https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/#priorityclass
priorityClassName: ""
# By default this will make sure two pods don't end up on the same node
# Changing this to a region would allow you to spread pods across regions
antiAffinityTopologyKey: "kubernetes.io/hostname"
# Hard means that by default pods will only be scheduled if there are enough nodes for them
# and that they will never end up on the same node. Setting this to soft will do this "best effort"
antiAffinity: "hard"
# This is the node affinity settings as defined in
# https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#node-affinity-beta-feature
nodeAffinity: {}
# The default is to deploy all pods serially. By setting this to parallel all pods are started at
# the same time when bootstrapping the cluster
podManagementPolicy: "Parallel"
protocol: http
httpPort: 9200
transportPort: 9300
updateStrategy: RollingUpdate
# This is the max unavailable setting for the pod disruption budget
# The default value of 1 will make sure that kubernetes won't allow more than 1
# of your pods to be unavailable during maintenance
maxUnavailable: 1
# GroupID for the elasticsearch user. The official elastic docker images always have the id of 1000
fsGroup: 1000
# How long to wait for elasticsearch to stop gracefully
terminationGracePeriod: 120
sysctlVmMaxMapCount: 262144
readinessProbe:
failureThreshold: 3
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 3
timeoutSeconds: 5
# https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-health.html#request-params wait_for_status
clusterHealthCheckParams: "wait_for_status=green&timeout=1s"
nodeSelector: {}
tolerations: []
# Enabling this will publically expose your Elasticsearch instance.
# Only enable this if you have security enabled on your cluster
ingress:
enabled: false
annotations: {}
# kubernetes.io/ingress.class: nginx
# kubernetes.io/tls-acme: "true"
path: /
hosts:
- chart-example.local
tls: []
# - secretName: chart-example-tls
# hosts:
# - chart-example.local
nameOverride: ""
fullnameOverride: ""
Describe the bug:
I initially had some issues with the image and tag due to ""
so i did this: to fix that: image: docker.elastic.co/elasticsearch/elasticsearch:{{ .Values.imageTag }}
but now the kubernetes pods are in a crashloopbackoff state with this in their logs:
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
{"type": "server", "timestamp": "2019-05-16T21:50:15,565+0000", "level": "WARN", "component": "o.e.b.ElasticsearchUncaughtExceptionHandler", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "uncaught exception in thread [main]" ,
"stacktrace": ["org.elasticsearch.bootstrap.StartupException: ElasticsearchException[failed to bind service]; nested: AccessDeniedException[/usr/share/elasticsearch/data/nodes];",
"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.0.1.jar:7.0.1]",
"at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.0.1.jar:7.0.1]",
"at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.0.1.jar:7.0.1]",
"at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-7.0.1.jar:7.0.1]",
"at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.0.1.jar:7.0.1]",
"at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.0.1.jar:7.0.1]",
"at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.0.1.jar:7.0.1]",
"Caused by: org.elasticsearch.ElasticsearchException: failed to bind service",
"at org.elasticsearch.node.Node.
"at org.elasticsearch.node.Node.
"at org.elasticsearch.bootstrap.Bootstrap$5.
"at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:211) ~[elasticsearch-7.0.1.jar:7.0.1]",
"at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:325) ~[elasticsearch-7.0.1.jar:7.0.1]",
"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.0.1.jar:7.0.1]",
"... 6 more",
"Caused by: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data/nodes",
"at sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) ~[?:?]",
"at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]",
"at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) ~[?:?]",
"at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:389) ~[?:?]",
"at java.nio.file.Files.createDirectory(Files.java:692) ~[?:?]",
"at java.nio.file.Files.createAndCheckIsDirectory(Files.java:799) ~[?:?]",
"at java.nio.file.Files.createDirectories(Files.java:785) ~[?:?]",
"at org.elasticsearch.env.NodeEnvironment.lambda$new$0(NodeEnvironment.java:270) ~[elasticsearch-7.0.1.jar:7.0.1]",
"at org.elasticsearch.env.NodeEnvironment$NodeLock.
"at org.elasticsearch.env.NodeEnvironment.
"at org.elasticsearch.node.Node.
"at org.elasticsearch.node.Node.
"at org.elasticsearch.bootstrap.Bootstrap$5.
"at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:211) ~[elasticsearch-7.0.1.jar:7.0.1]",
"at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:325) ~[elasticsearch-7.0.1.jar:7.0.1]",
"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.0.1.jar:7.0.1]",
"... 6 more"] }
Steps to reproduce:
1.
2.
3.
Expected behavior:
Provide logs and/or server output (if relevant):
Any additional context:
ceastman-ibm
All 23 comments
I had same problem on Kubernetes 1.14.2 with ES 6.2.4 and 6.8.0.
I saw an almost identical issue here for 6.2.4 and here for 6.3.2.
I saw many comments on permissions for elasticsearch user and also found comments on Elastic discuss, but I really didn't find a practical solution.
staticdev
on 22 May 2019
I don't have an IBM Kubernetes cluster to test with. It's also a setup that we haven't tested at all yet, however if it has persistent storage it should work just fine. Are you able to give some more debugging output:
kubectl get pv
kubectl get pvc
kubectl get storageclass
kubectl describe statefulset elasticsearch-master
kubectl get statefulset elasticsearch-master -o yaml
And if possible could you try to attach into the container while it is starting up and check if the volumes are mounted correctly?
ls -lhat /usr/share/elasticsearch/data
df -h | grep /usr/share/elasticsearch/data
mount | grep /usr/share/elasticsearch/data
I initially had some issues with the image and tag due to ""
so i did this: to fix that: image: docker.elastic.co/elasticsearch/elasticsearch:{{ .Values.imageTag }}
I believe this is because you didn't quote imageTag: 7.0.1 properly. It needs to be a string so should have quotes like: imageTag: "7.0.1"
Crazybus
on 27 May 2019
@Crazybus i just pulled the latest helm chart and will try again.
ceastman-ibm
on 28 May 2019
there is something with parsing the yaml. im getting the following error and here is the image tag:
elasticsearch-master-0 0/2 Init:InvalidImageName
Image: map[pullPolicy:IfNotPresent pullSecret:docker]:7.1.0
here is the describe pod contents:
kubectl describe pod elasticsearch-master-0
Name: elasticsearch-master-0
Namespace: observability
Priority: 0
PriorityClassName: <none>
Node: 10.176.72.100/10.176.72.100
Start Time: Tue, 28 May 2019 13:53:24 -0500
Labels: app=elasticsearch-master
chart=elasticsearch-new-7.1.0
controller-revision-hash=elasticsearch-master-8669cbb44c
heritage=Tiller
release=elasticsearch
statefulset.kubernetes.io/pod-name=elasticsearch-master-0
Annotations: kubernetes.io/psp: ibm-privileged-psp
Status: Pending
IP: 172.30.45.180
Controlled By: StatefulSet/elasticsearch-master
Init Containers:
configure-sysctl:
Container ID:
Image: map[pullPolicy:IfNotPresent pullSecret:docker]:7.1.0
Image ID:
Port: <none>
Host Port: <none>
Command:
sysctl
-w
vm.max_map_count=262144
State: Waiting
Reason: InvalidImageName
Ready: False
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-7wv9b (ro)
Containers:
elasticsearch-new:
Container ID:
Image: map[pullPolicy:IfNotPresent pullSecret:docker]:7.1.0
Image ID:
Ports: 9200/TCP, 9300/TCP
Host Ports: 0/TCP, 0/TCP
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Limits:
cpu: 1
memory: 2Gi
Requests:
cpu: 100m
memory: 2Gi
Readiness: exec [sh -c #!/usr/bin/env bash -e
# If the node is starting up wait for the cluster to be ready (request params: 'wait_for_status=green&timeout=1s' )
# Once it has started only check that the node itself is responding
START_FILE=/tmp/.es_start_file
http () {
local path="${1}"
if [ -n "${ELASTIC_USERNAME}" ] && [ -n "${ELASTIC_PASSWORD}" ]; then
BASIC_AUTH="-u ${ELASTIC_USERNAME}:${ELASTIC_PASSWORD}"
else
BASIC_AUTH=''
fi
curl -XGET -s -k --fail ${BASIC_AUTH} http://127.0.0.1:9200${path}
}
if [ -f "${START_FILE}" ]; then
echo 'Elasticsearch is already running, lets check the node is healthy'
http "/"
else
echo 'Waiting for elasticsearch cluster to become cluster to be ready (request params: "wait_for_status=green&timeout=1s" )'
if http "/_cluster/health?wait_for_status=green&timeout=1s" ; then
touch ${START_FILE}
exit 0
else
echo 'Cluster is not yet ready (request params: "wait_for_status=green&timeout=1s" )'
exit 1
fi
fi
] delay=10s timeout=5s period=10s #success=3 #failure=3
Environment:
node.name: elasticsearch-master-0 (v1:metadata.name)
cluster.initial_master_nodes: elasticsearch-master-0,elasticsearch-master-1,elasticsearch-master-2,
discovery.seed_hosts: elasticsearch-master-headless
cluster.name: elasticsearch
network.host: 0.0.0.0
ES_JAVA_OPTS: -Xmx1g -Xms1g
node.data: true
node.ingest: true
node.master: true
Mounts:
/usr/share/elasticsearch/data from elasticsearch-master (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-7wv9b (ro)
elasticsearch-master-graceful-termination-handler:
Container ID:
Image: map[pullPolicy:IfNotPresent pullSecret:docker]:7.1.0
Image ID:
Port: <none>
Host Port: <none>
Command:
sh
-c
#!/usr/bin/env bash
set -eo pipefail
http () {
local path="${1}"
if [ -n "${ELASTIC_USERNAME}" ] && [ -n "${ELASTIC_PASSWORD}" ]; then
BASIC_AUTH="-u ${ELASTIC_USERNAME}:${ELASTIC_PASSWORD}"
else
BASIC_AUTH=''
fi
curl -XGET -s -k --fail ${BASIC_AUTH} http://elasticsearch-master:9200${path}
}
cleanup () {
while true ; do
local master="$(http "/_cat/master?h=node")"
if [[ $master == "elasticsearch-master"* && $master != "${NODE_NAME}" ]]; then
echo "This node is not master."
break
fi
echo "This node is still master, waiting gracefully for it to step down"
sleep 1
done
exit 0
}
trap cleanup SIGTERM
sleep infinity &
wait $!
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Environment:
NODE_NAME: elasticsearch-master-0 (v1:metadata.name)
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-7wv9b (ro)
Conditions:
Type Status
Initialized False
Ready False
ContainersReady False
PodScheduled True
Volumes:
elasticsearch-master:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: elasticsearch-master-elasticsearch-master-0
ReadOnly: false
default-token-7wv9b:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-7wv9b
Optional: false
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 49s (x21 over 3m38s) default-scheduler pod has unbound immediate PersistentVolumeClaims (repeated 3 times)
Normal Scheduled 49s default-scheduler Successfully assigned observability/elasticsearch-master-0 to 10.176.72.100
Warning InspectFailed 9s (x5 over 48s) kubelet, 10.176.72.100 Failed to apply default image tag "map[pullPolicy:IfNotPresent pullSecret:docker]:7.1.0": couldn't parse image reference "map[pullPolicy:IfNotPresent pullSecret:docker]:7.1.0": invalid reference format: repository name must be lowercase
Warning Failed 9s (x5 over 48s) kubelet, 10.176.72.100 Error: InvalidImageName
i changed the image: in the yaml to be the following to get by that error:
image: "docker.elastic.co/elasticsearch/elasticsearch:{{ .Values.imageTag }}"
ceastman-ibm
on 28 May 2019
here is the debugging info you requested:
Carls-MBP:helm carleastman$ kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-f177e32b-817c-11e9-8124-5e459b2de3b0 30Gi RWO Delete Bound observability/elasticsearch-master-elasticsearch-master-0 ibmc-file-bronze 6m39s
pvc-f17be671-817c-11e9-8124-5e459b2de3b0 30Gi RWO Delete Bound observability/elasticsearch-master-elasticsearch-master-1 ibmc-file-bronze 6m23s
pvc-f17fbca7-817c-11e9-8124-5e459b2de3b0 30Gi RWO Delete Bound observability/elasticsearch-master-elasticsearch-master-2 ibmc-file-bronze 6m51s
Carls-MBP:helm carleastman$ kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
elasticsearch-master-elasticsearch-master-0 Bound pvc-f177e32b-817c-11e9-8124-5e459b2de3b0 30Gi RWO ibmc-file-bronze 9m36s
elasticsearch-master-elasticsearch-master-1 Bound pvc-f17be671-817c-11e9-8124-5e459b2de3b0 30Gi RWO ibmc-file-bronze 9m36s
elasticsearch-master-elasticsearch-master-2 Bound pvc-f17fbca7-817c-11e9-8124-5e459b2de3b0 30Gi RWO ibmc-file-bronze 9m36s
Carls-MBP:helm carleastman$ kubectl get storageclass
NAME PROVISIONER AGE
default ibm.io/ibmc-file 11d
ibmc-file-bronze (default) ibm.io/ibmc-file 11d
ibmc-file-custom ibm.io/ibmc-file 11d
ibmc-file-gold ibm.io/ibmc-file 11d
ibmc-file-retain-bronze ibm.io/ibmc-file 11d
ibmc-file-retain-custom ibm.io/ibmc-file 11d
ibmc-file-retain-gold ibm.io/ibmc-file 11d
ibmc-file-retain-silver ibm.io/ibmc-file 11d
ibmc-file-silver ibm.io/ibmc-file 11d
Carls-MBP:helm carleastman$ kubectl describe statefulset elasticsearch-master
Name: elasticsearch-master
Namespace: observability
CreationTimestamp: Tue, 28 May 2019 14:15:23 -0500
Selector: app=elasticsearch-master
Labels: app=elasticsearch-master
chart=elasticsearch-new-7.1.0
heritage=Tiller
release=elasticsearch
Annotations: <none>
Replicas: 3 desired | 3 total
Update Strategy: RollingUpdate
Pods Status: 3 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
Labels: app=elasticsearch-master
chart=elasticsearch-new-7.1.0
heritage=Tiller
release=elasticsearch
Init Containers:
configure-sysctl:
Image: docker.elastic.co/elasticsearch/elasticsearch:7.1.0
Port: <none>
Host Port: <none>
Command:
sysctl
-w
vm.max_map_count=262144
Environment: <none>
Mounts: <none>
Containers:
elasticsearch-new:
Image: docker.elastic.co/elasticsearch/elasticsearch:7.1.0
Ports: 9200/TCP, 9300/TCP
Host Ports: 0/TCP, 0/TCP
Limits:
cpu: 1
memory: 2Gi
Requests:
cpu: 100m
memory: 2Gi
Readiness: exec [sh -c #!/usr/bin/env bash -e
# If the node is starting up wait for the cluster to be ready (request params: 'wait_for_status=green&timeout=1s' )
# Once it has started only check that the node itself is responding
START_FILE=/tmp/.es_start_file
http () {
local path="${1}"
if [ -n "${ELASTIC_USERNAME}" ] && [ -n "${ELASTIC_PASSWORD}" ]; then
BASIC_AUTH="-u ${ELASTIC_USERNAME}:${ELASTIC_PASSWORD}"
else
BASIC_AUTH=''
fi
curl -XGET -s -k --fail ${BASIC_AUTH} http://127.0.0.1:9200${path}
}
if [ -f "${START_FILE}" ]; then
echo 'Elasticsearch is already running, lets check the node is healthy'
http "/"
else
echo 'Waiting for elasticsearch cluster to become cluster to be ready (request params: "wait_for_status=green&timeout=1s" )'
if http "/_cluster/health?wait_for_status=green&timeout=1s" ; then
touch ${START_FILE}
exit 0
else
echo 'Cluster is not yet ready (request params: "wait_for_status=green&timeout=1s" )'
exit 1
fi
fi
] delay=10s timeout=5s period=10s #success=3 #failure=3
Environment:
node.name: (v1:metadata.name)
cluster.initial_master_nodes: elasticsearch-master-0,elasticsearch-master-1,elasticsearch-master-2,
discovery.seed_hosts: elasticsearch-master-headless
cluster.name: elasticsearch
network.host: 0.0.0.0
ES_JAVA_OPTS: -Xmx1g -Xms1g
node.data: true
node.ingest: true
node.master: true
Mounts:
/usr/share/elasticsearch/data from elasticsearch-master (rw)
elasticsearch-master-graceful-termination-handler:
Image: docker.elastic.co/elasticsearch/elasticsearch:7.1.0
Port: <none>
Host Port: <none>
Command:
sh
-c
#!/usr/bin/env bash
set -eo pipefail
http () {
local path="${1}"
if [ -n "${ELASTIC_USERNAME}" ] && [ -n "${ELASTIC_PASSWORD}" ]; then
BASIC_AUTH="-u ${ELASTIC_USERNAME}:${ELASTIC_PASSWORD}"
else
BASIC_AUTH=''
fi
curl -XGET -s -k --fail ${BASIC_AUTH} http://elasticsearch-master:9200${path}
}
cleanup () {
while true ; do
local master="$(http "/_cat/master?h=node")"
if [[ $master == "elasticsearch-master"* && $master != "${NODE_NAME}" ]]; then
echo "This node is not master."
break
fi
echo "This node is still master, waiting gracefully for it to step down"
sleep 1
done
exit 0
}
trap cleanup SIGTERM
sleep infinity &
wait $!
Environment:
NODE_NAME: (v1:metadata.name)
Mounts: <none>
Volumes: <none>
Volume Claims:
Name: elasticsearch-master
StorageClass:
Labels: <none>
Annotations: <none>
Capacity: 30Gi
Access Modes: [ReadWriteOnce]
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SuccessfulCreate 9m46s statefulset-controller create Claim elasticsearch-master-elasticsearch-master-0 Pod elasticsearch-master-0 in StatefulSet elasticsearch-master success
Normal SuccessfulCreate 9m46s statefulset-controller create Pod elasticsearch-master-0 in StatefulSet elasticsearch-master successful
Normal SuccessfulCreate 9m46s statefulset-controller create Claim elasticsearch-master-elasticsearch-master-1 Pod elasticsearch-master-1 in StatefulSet elasticsearch-master success
Normal SuccessfulCreate 9m46s statefulset-controller create Pod elasticsearch-master-1 in StatefulSet elasticsearch-master successful
Normal SuccessfulCreate 9m46s statefulset-controller create Claim elasticsearch-master-elasticsearch-master-2 Pod elasticsearch-master-2 in StatefulSet elasticsearch-master success
Normal SuccessfulCreate 9m46s statefulset-controller create Pod elasticsearch-master-2 in StatefulSet elasticsearch-master successful
Carls-MBP:helm carleastman$ kubectl get statefulset elasticsearch-master -o yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
creationTimestamp: "2019-05-28T19:15:23Z"
generation: 1
labels:
app: elasticsearch-master
chart: elasticsearch-new-7.1.0
heritage: Tiller
release: elasticsearch
name: elasticsearch-master
namespace: observability
resourceVersion: "11595522"
selfLink: /apis/apps/v1/namespaces/observability/statefulsets/elasticsearch-master
uid: f1705885-817c-11e9-8124-5e459b2de3b0
spec:
podManagementPolicy: Parallel
replicas: 3
revisionHistoryLimit: 10
selector:
matchLabels:
app: elasticsearch-master
serviceName: elasticsearch-master-headless
template:
metadata:
creationTimestamp: null
labels:
app: elasticsearch-master
chart: elasticsearch-new-7.1.0
heritage: Tiller
release: elasticsearch
name: elasticsearch-master
spec:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: app
operator: In
values:
- elasticsearch-master
topologyKey: kubernetes.io/hostname
containers:
- env:
- name: node.name
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: cluster.initial_master_nodes
value: elasticsearch-master-0,elasticsearch-master-1,elasticsearch-master-2,
- name: discovery.seed_hosts
value: elasticsearch-master-headless
- name: cluster.name
value: elasticsearch
- name: network.host
value: 0.0.0.0
- name: ES_JAVA_OPTS
value: -Xmx1g -Xms1g
- name: node.data
value: "true"
- name: node.ingest
value: "true"
- name: node.master
value: "true"
image: docker.elastic.co/elasticsearch/elasticsearch:7.1.0
imagePullPolicy: IfNotPresent
name: elasticsearch-new
ports:
- containerPort: 9200
name: http
protocol: TCP
- containerPort: 9300
name: transport
protocol: TCP
readinessProbe:
exec:
command:
- sh
- -c
- |
#!/usr/bin/env bash -e
# If the node is starting up wait for the cluster to be ready (request params: 'wait_for_status=green&timeout=1s' )
# Once it has started only check that the node itself is responding
START_FILE=/tmp/.es_start_file
http () {
local path="${1}"
if [ -n "${ELASTIC_USERNAME}" ] && [ -n "${ELASTIC_PASSWORD}" ]; then
BASIC_AUTH="-u ${ELASTIC_USERNAME}:${ELASTIC_PASSWORD}"
else
BASIC_AUTH=''
fi
curl -XGET -s -k --fail ${BASIC_AUTH} http://127.0.0.1:9200${path}
}
if [ -f "${START_FILE}" ]; then
echo 'Elasticsearch is already running, lets check the node is healthy'
http "/"
else
echo 'Waiting for elasticsearch cluster to become cluster to be ready (request params: "wait_for_status=green&timeout=1s" )'
if http "/_cluster/health?wait_for_status=green&timeout=1s" ; then
touch ${START_FILE}
exit 0
else
echo 'Cluster is not yet ready (request params: "wait_for_status=green&timeout=1s" )'
exit 1
fi
fi
failureThreshold: 3
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 3
timeoutSeconds: 5
resources:
limits:
cpu: "1"
memory: 2Gi
requests:
cpu: 100m
memory: 2Gi
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /usr/share/elasticsearch/data
name: elasticsearch-master
- command:
- sh
- -c
- |
#!/usr/bin/env bash
set -eo pipefail
http () {
local path="${1}"
if [ -n "${ELASTIC_USERNAME}" ] && [ -n "${ELASTIC_PASSWORD}" ]; then
BASIC_AUTH="-u ${ELASTIC_USERNAME}:${ELASTIC_PASSWORD}"
else
BASIC_AUTH=''
fi
curl -XGET -s -k --fail ${BASIC_AUTH} http://elasticsearch-master:9200${path}
}
cleanup () {
while true ; do
local master="$(http "/_cat/master?h=node")"
if [[ $master == "elasticsearch-master"* && $master != "${NODE_NAME}" ]]; then
echo "This node is not master."
break
fi
echo "This node is still master, waiting gracefully for it to step down"
sleep 1
done
exit 0
}
trap cleanup SIGTERM
sleep infinity &
wait $!
env:
- name: NODE_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
image: docker.elastic.co/elasticsearch/elasticsearch:7.1.0
imagePullPolicy: IfNotPresent
name: elasticsearch-master-graceful-termination-handler
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
dnsPolicy: ClusterFirst
initContainers:
- command:
- sysctl
- -w
- vm.max_map_count=262144
image: docker.elastic.co/elasticsearch/elasticsearch:7.1.0
imagePullPolicy: IfNotPresent
name: configure-sysctl
resources: {}
securityContext:
privileged: true
procMount: Default
runAsUser: 0
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
restartPolicy: Always
schedulerName: default-scheduler
securityContext:
fsGroup: 1000
terminationGracePeriodSeconds: 120
updateStrategy:
type: RollingUpdate
volumeClaimTemplates:
- metadata:
creationTimestamp: null
name: elasticsearch-master
spec:
accessModes:
- ReadWriteOnce
dataSource: null
resources:
requests:
storage: 30Gi
status:
phase: Pending
status:
collisionCount: 0
currentReplicas: 3
currentRevision: elasticsearch-master-64cbf7f58c
observedGeneration: 1
replicas: 3
updateRevision: elasticsearch-master-64cbf7f58c
updatedReplicas: 3
Carls-MBP:helm carleastman$ kubectl exec -it elasticsearch-master-0 bash -c elasticsearch-master-graceful-termination-handler
[root@elasticsearch-master-0 elasticsearch]# ls -lhat /usr/share/elasticsearch/data
total 8.0K
drwxrwxr-x 1 elasticsearch root 4.0K May 15 20:10 ..
drwxrwxr-x 2 elasticsearch root 4.0K May 15 20:09 .
[root@elasticsearch-master-0 elasticsearch]# df -h | grep /usr/share/elasticsearch/data
[root@elasticsearch-master-0 elasticsearch]# mount | grep /usr/share/elasticsearch/data
One problem seems to be with the formatting of the image. In the pod spec I can see this:
Image: map[pullPolicy:IfNotPresent pullSecret:docker]:7.1.0
This implies that you have something like this set:
image:
imagePullPolicy: "IfNotPresent"
pullSecret: "docker"
However I don't see this in the values.yaml you posted earlier. And this isn't the right way to specify the image pull secrets for this chart. It should look like: https://github.com/elastic/helm-charts/blob/4d6d768ce9e222b48791158c760d5e831d1331f6/elasticsearch/tests/elasticsearch_test.py#L384-L390
Could you post the output of helm get elasticsearch (or whatever your helm release is called). This will show me all of the values that have been set.
kubectl exec -it elasticsearch-master-0 bash -c elasticsearch-master-graceful-termination-handler
This container shouldn't have the disks mounted into it. Can you run this again from the elasticsearch container?
Crazybus
on 3 Jun 2019
Having the same problem. Running kube v1.14.2 and es v7.1.0. All 3 pods can create the elasticsearch-master-graceful-termination-handler container but not the elasticsearch container. Only thing I customized when deploying was the namespace. Here are the logs for elasticsearch container:
➜ ~ klogs elasticsearch-master-2
1) elasticsearch
2) elasticsearch-master-graceful-termination-handler
Please select a container:1
Getting you a shell in elasticsearch...
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
{"type": "server", "timestamp": "2019-06-03T13:29:16,629+0000", "level": "WARN", "component": "o.e.b.ElasticsearchUncaughtExceptionHandler", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-2", "message": "uncaught exception in thread [main]" ,
"stacktrace": ["org.elasticsearch.bootstrap.StartupException: ElasticsearchException[failed to bind service]; nested: AccessDeniedException[/usr/share/elasticsearch/data/nodes];",
"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.1.0.jar:7.1.0]",
"at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.1.0.jar:7.1.0]",
"at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.1.0.jar:7.1.0]",
"at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-7.1.0.jar:7.1.0]",
"at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.1.0.jar:7.1.0]",
"at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.1.0.jar:7.1.0]",
"at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.1.0.jar:7.1.0]",
"Caused by: org.elasticsearch.ElasticsearchException: failed to bind service",
"at org.elasticsearch.node.Node.<init>(Node.java:582) ~[elasticsearch-7.1.0.jar:7.1.0]",
"at org.elasticsearch.node.Node.<init>(Node.java:252) ~[elasticsearch-7.1.0.jar:7.1.0]",
"at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:211) ~[elasticsearch-7.1.0.jar:7.1.0]",
"at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:211) ~[elasticsearch-7.1.0.jar:7.1.0]",
"at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:325) ~[elasticsearch-7.1.0.jar:7.1.0]",
"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.1.0.jar:7.1.0]",
"... 6 more",
"Caused by: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data/nodes",
"at sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) ~[?:?]",
"at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]",
"at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) ~[?:?]",
"at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:389) ~[?:?]",
"at java.nio.file.Files.createDirectory(Files.java:692) ~[?:?]",
"at java.nio.file.Files.createAndCheckIsDirectory(Files.java:799) ~[?:?]",
"at java.nio.file.Files.createDirectories(Files.java:785) ~[?:?]",
"at org.elasticsearch.env.NodeEnvironment.lambda$new$0(NodeEnvironment.java:270) ~[elasticsearch-7.1.0.jar:7.1.0]",
"at org.elasticsearch.env.NodeEnvironment$NodeLock.<init>(NodeEnvironment.java:207) ~[elasticsearch-7.1.0.jar:7.1.0]",
"at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:267) ~[elasticsearch-7.1.0.jar:7.1.0]",
"at org.elasticsearch.node.Node.<init>(Node.java:272) ~[elasticsearch-7.1.0.jar:7.1.0]",
"at org.elasticsearch.node.Node.<init>(Node.java:252) ~[elasticsearch-7.1.0.jar:7.1.0]",
"at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:211) ~[elasticsearch-7.1.0.jar:7.1.0]",
"at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:211) ~[elasticsearch-7.1.0.jar:7.1.0]",
"at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:325) ~[elasticsearch-7.1.0.jar:7.1.0]",
"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.1.0.jar:7.1.0]",
"... 6 more"] }
ghostymun
on 3 Jun 2019
@ghostymun Thanks for the extra report.
Could you also provide me with this information?
kubectl get pv
kubectl get pvc
kubectl get storageclass
kubectl describe statefulset elasticsearch-master
helm get elasticsearch # (or whatever your helm release is called)
And if possible could you try to attach into the container while it is starting up and check if the volumes are mounted correctly?
ls -lhat /usr/share/elasticsearch/data
df -h | grep /usr/share/elasticsearch/data
mount | grep /usr/share/elasticsearch/data
I to have the same issue! A lot of forum posts suggest this a UID or GID issue. I manually created the $DATA/elasticsearch. Set permissions 666 and created the PV's.
clusterName: "elasticsearch"
nodeGroup: "master"
# The service that non master groups will try to connect to when joining the cluster
# This should be set to clusterName + "-" + nodeGroup for your master group
masterService: ""
# Elasticsearch roles that will be applied to this nodeGroup
# These will be set as environment variables. E.g. node.master=true
roles:
master: "true"
ingest: "true"
data: "true"
replicas: 3
minimumMasterNodes: 2
esMajorVersion: 7
# Allows you to add any config files in /usr/share/elasticsearch/config/
# such as elasticsearch.yml and log4j2.properties
esConfig: {}
# elasticsearch.yml: |
# key:
# nestedkey: value
# log4j2.properties: |
# key = value
# Extra environment variables to append to this nodeGroup
# This will be appended to the current 'env:' key. You can use any of the kubernetes env
# syntax here
extraEnvs: []
# - name: MY_ENVIRONMENT_VAR
# value: the_value_goes_here
# A list of secrets and their paths to mount inside the pod
# This is useful for mounting certificates for security and for mounting
# the X-Pack license
secretMounts: []
# - name: elastic-certificates
# secretName: elastic-certificates
# path: /usr/share/elasticsearch/config/certs
image: "docker.elastic.co/elasticsearch/elasticsearch"
imageTag: "7.1.0"
imagePullPolicy: "IfNotPresent"
podAnnotations: {}
# iam.amazonaws.com/role: es-cluster
esJavaOpts: "-Xmx1g -Xms1g"
resources:
requests:
cpu: "100m"
memory: "2Gi"
limits:
cpu: "1000m"
memory: "2Gi"
initResources: {}
# limits:
# cpu: "25m"
# # memory: "128Mi"
# requests:
# cpu: "25m"
# memory: "128Mi"
networkHost: "0.0.0.0"
volumeClaimTemplate:
accessModes: [ "ReadWriteOnce" ]
storageClassName: "manual"
resources:
requests:
storage: 10Gi
persistence:
enabled: true
annotations: {}
extraVolumes: []
# - name: extras
# emptyDir: {}
extraVolumeMounts: []
# - name: extras
# mountPath: /usr/share/extras
# readOnly: true
extraInitContainers: []
# - name: do-something
# image: busybox
# command: ['do', 'something']
# This is the PriorityClass settings as defined in
# https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/#priorityclass
priorityClassName: ""
# By default this will make sure two pods don't end up on the same node
# Changing this to a region would allow you to spread pods across regions
antiAffinityTopologyKey: "kubernetes.io/hostname"
# Hard means that by default pods will only be scheduled if there are enough nodes for them
# and that they will never end up on the same node. Setting this to soft will do this "best effort"
antiAffinity: "hard"
# This is the node affinity settings as defined in
# https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#node-affinity-beta-feature
nodeAffinity: {}
# The default is to deploy all pods serially. By setting this to parallel all pods are started at
# the same time when bootstrapping the cluster
podManagementPolicy: "Parallel"
protocol: http
httpPort: 9200
transportPort: 9300
updateStrategy: RollingUpdate
# This is the max unavailable setting for the pod disruption budget
# The default value of 1 will make sure that kubernetes won't allow more than 1
# of your pods to be unavailable during maintenance
maxUnavailable: 1
# GroupID for the elasticsearch user. The official elastic docker images always have the id of 1000
fsGroup: 1000
# How long to wait for elasticsearch to stop gracefully
terminationGracePeriod: 120
sysctlVmMaxMapCount: 262144
readinessProbe:
failureThreshold: 3
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 3
timeoutSeconds: 5
# https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-health.html#request-params wait_for_status
clusterHealthCheckParams: "wait_for_status=green&timeout=1s"
imagePullSecrets: []
nodeSelector: {}
tolerations: []
# Enabling this will publically expose your Elasticsearch instance.
# Only enable this if you have security enabled on your cluster
ingress:
enabled: false
annotations: {}
# kubernetes.io/ingress.class: nginx
# kubernetes.io/tls-acme: "true"
path: /
hosts:
- chart-example.local
tls: []
# - secretName: chart-example-tls
# hosts:
# - chart-example.local
nameOverride: ""
fullnameOverride: ""
savelono
on 7 Jun 2019
version 7.1.0 and I also installed in the non default name space.
savelono
on 7 Jun 2019
savelono
on 7 Jun 2019
I manually changed the permissions to my own "matt" UID '1000' on each node after creating the PV's. Now it installed correctly!
savelono
on 7 Jun 2019
@Crazybus i think this has helped - at least for my IBM cloud 1.13 kube cluster: https://cloud.ibm.com/docs/containers?topic=containers-cs_troubleshoot_storage#file_app_failures
in the initContainer section of the statefulset.yaml i added:
chown -R 1000:1000 /usr/share/elasticsearch/data
this seems to have gotten around the access denied issue.
ceastman-ibm
on 19 Jun 2019
Glad you figured it out. It's really surprising that IBM Kubernetes does not support this and actively encourages people to just run everything as root: https://cloud.ibm.com/docs/containers?topic=containers-cs_troubleshoot_storage#cs_storage_nonroot
The deployment or Helm chart configuration specifies the security context for the pod's fsGroup (group ID) and runAsUser (user ID). Currently, IBM Cloud Kubernetes Service does not support the fsGroup specification, and supports only runAsUser set as 0 (root permissions).
There are other Kubernetes providers that do the opposite, that prevent you from running anything as root. Since the Elasticsearch chart does work in a non-root environment I think its best to keep it as is and instead expect that users requiring root only volumes add something like this to extraInitContainers:
(not tested)
extraInitContainers: |
- name: file-permissions
image: "{{ .Values.image }}:{{ .Values.imageTag }}"
command: ['chown', '-R', '1000:1000', '/usr/share/elasticsearch/data']
@Crazybus sounds good.
ceastman-ibm
on 20 Jun 2019
i have checked official es operator and used
securityContext:
fsGroup: 1000
from it.
it worked well.
dzmitry-lahoda
on 25 Sep 2019
Having the same issue but with a path '/usr/share/elasticsearch/data/nodes/0' and cannot wrap my head around why it happens. Tried with several extraInitContainers ways but they either get stuck on PodInitiating or have no effect at all.
I am using a custom created local-storage class, on custom Persistent Volume and the Custer is Bare metal, which means deployed on our own machine.
vidolch
on 26 Feb 2020
This is the result of kubectl -n elasticsearch describe pod elasticsearch-master-0
Name: elasticsearch-master-0
Namespace: elasticsearch
Priority: 0
Node: minion-1/100.81.10.110
Start Time: Wed, 26 Feb 2020 14:00:40 +0000
Labels: app=elasticsearch-master
chart=elasticsearch
controller-revision-hash=elasticsearch-master-5cd9cb4dbc
heritage=Helm
release=elasticsearch
statefulset.kubernetes.io/pod-name=elasticsearch-master-0
Annotations: <none>
Status: Pending
IP: 10.244.0.156
IPs:
IP: 10.244.0.156
Controlled By: StatefulSet/elasticsearch-master
Init Containers:
create:
Container ID: docker://20f220d52f2c660a36b3f697a44625ce794a5b9b278abbf62cf5f8ae4d04a342
Image: busybox:1.28
Image ID: docker-pullable://busybox@sha256:141c253bc4c3fd0a201d32dc1f493bcf3fff003b6df416dea4f41046e0f37d47
Port: <none>
Host Port: <none>
Command:
mkdir
/usr/share/elasticsearch/data/nodes/
State: Terminated
Reason: Error
Exit Code: 1
Started: Wed, 26 Feb 2020 14:03:44 +0000
Finished: Wed, 26 Feb 2020 14:03:44 +0000
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Wed, 26 Feb 2020 14:02:11 +0000
Finished: Wed, 26 Feb 2020 14:02:11 +0000
Ready: False
Restart Count: 5
Environment: <none>
Mounts:
/usr/share/elasticsearch/data from elasticsearch-master (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-cwdpr (ro)
file-permissions:
Container ID:
Image: busybox:1.28
Image ID:
Port: <none>
Host Port: <none>
Command:
chown
-R
1000:1000
/usr/share/elasticsearch/
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Environment: <none>
Mounts:
/usr/share/elasticsearch/data from elasticsearch-master (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-cwdpr (ro)
Containers:
elasticsearch:
Container ID:
Image: docker.elastic.co/elasticsearch/elasticsearch:7.6.0
Image ID:
Ports: 9200/TCP, 9300/TCP
Host Ports: 0/TCP, 0/TCP
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Limits:
cpu: 1
memory: 2Gi
Requests:
cpu: 1
memory: 2Gi
Readiness: exec [sh -c #!/usr/bin/env bash -e
# If the node is starting up wait for the cluster to be ready (request params: 'wait_for_status=green&timeout=1s' )
# Once it has started only check that the node itself is responding
START_FILE=/tmp/.es_start_file
http () {
local path="${1}"
if [ -n "${ELASTIC_USERNAME}" ] && [ -n "${ELASTIC_PASSWORD}" ]; then
BASIC_AUTH="-u ${ELASTIC_USERNAME}:${ELASTIC_PASSWORD}"
else
BASIC_AUTH=''
fi
curl -XGET -s -k --fail ${BASIC_AUTH} http://127.0.0.1:9200${path}
}
if [ -f "${START_FILE}" ]; then
echo 'Elasticsearch is already running, lets check the node is healthy and there are master nodes available'
http "/_cluster/health?timeout=0s"
else
echo 'Waiting for elasticsearch cluster to become ready (request params: "wait_for_status=green&timeout=1s" )'
if http "/_cluster/health?wait_for_status=green&timeout=1s" ; then
touch ${START_FILE}
exit 0
else
echo 'Cluster is not yet ready (request params: "wait_for_status=green&timeout=1s" )'
exit 1
fi
fi
] delay=10s timeout=5s period=10s #success=3 #failure=3
Environment:
node.name: elasticsearch-master-0 (v1:metadata.name)
cluster.initial_master_nodes: elasticsearch-master-0,
discovery.seed_hosts: elasticsearch-master-headless
cluster.name: elasticsearch
network.host: 0.0.0.0
ES_JAVA_OPTS: -Xmx1g -Xms1g
node.data: true
node.ingest: true
node.master: true
Mounts:
/usr/share/elasticsearch/data from elasticsearch-master (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-cwdpr (ro)
Conditions:
Type Status
Initialized False
Ready False
ContainersReady False
PodScheduled True
Volumes:
elasticsearch-master:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: elasticsearch-master-elasticsearch-master-0
ReadOnly: false
default-token-cwdpr:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-cwdpr
Optional: false
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled <unknown> default-scheduler Successfully assigned elasticsearch/elasticsearch-master-0 to minion-1
Normal Pulled 99s (x5 over 3m7s) kubelet, minion-1 Container image "busybox:1.28" already present on machine
Normal Created 98s (x5 over 3m6s) kubelet, minion-1 Created container create
Normal Started 97s (x5 over 3m6s) kubelet, minion-1 Started container create
Warning BackOff 70s (x10 over 3m2s) kubelet, minion-1 Back-off restarting failed container
This is happening for me every time I restart my minikube node... Is there any solution to this?
brandoncollins7
on 1 Jun 2020
@brandoncollins7 It's a permission issue. What's your environment? If you are on local you can just do chmod -R 777 on that directory(simplest but not the secure). If not on local then you will have to see that user who is running this service has permission to write into that directory, if not then give permission to that user.
mohitpawar10
on 13 Jun 2020
I had the same issue and found an elegant solution using IBM documentations (https://cloud.ibm.com/docs/containers?topic=containers-file_storage#file_storageclass_reference)
First of all, you can make a little check :
make a kubectl get pvc | grep elasticsearch
if indicated STORAGECLASS is something like "ibmc-file-gold" (or silver/bronze), the right issues is expected to occur.
The solution implies modification of the StorageClass to something like "ibmc-file-gold-gid"
Before that, you should delete the PVCs ( as they will keep their StorageClass even with an helm uninstall/reinstall.).
clone the helm chart from git to get the elasticsearch/values.yaml.
In the values.yaml, find the volumeClaimTemplate and add the storageClassName line
volumeClaimTemplate:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 30Gi
storageClassName: ibmc-file-bronze-gid
Find the podSecurityContext and modify the fsGroup :
podSecurityContext:
fsGroup: 65531
runAsUser: 1000
Then, you can perform your helm install with the custom values.yaml.
Right Issues should have disappear
Maalshekto
on 10 Feb 2021
Related issues
davidkarlsen
·
5Comments
melissajenner22
·
6Comments
dperetti
·
4Comments
4nte
·
4Comments
naseemkullah
·
6Comments
Most helpful comment
@Crazybus i think this has helped - at least for my IBM cloud 1.13 kube cluster: https://cloud.ibm.com/docs/containers?topic=containers-cs_troubleshoot_storage#file_app_failures
in the initContainer section of the statefulset.yaml i added:
chown -R 1000:1000 /usr/share/elasticsearch/data
this seems to have gotten around the access denied issue.