Charts: Can not deploy the charts in kuberente v1.11

[huikang]

Steps to reproduce the issue:

1. helm install bitnami/etcd --set persistence.enabled=false

Describe the results you received:

 kubectl get pod
NAME                        READY     STATUS             RESTARTS   AGE
vigilant-bumblebee-etcd-0   0/1       CrashLoopBackOff   5          5m

kubectl logs -f vigilant-bumblebee-etcd-0
==> The ID of the host is 0
==> Creating data dir...
==> There is no data at all. Creating new cluster
==> Configuring RBAC authentication!
stty: 'standard input': Inappropriate ioctl for device
stty: 'standard input': Inappropriate ioctl for device
New password: 


client: etcd cluster is unavailable or misconfigured; error #0: client: endpoint http://vigilant-bumblebee-etcd-0.vigilant-bumblebee-etcd-headless.default.svc.cluster.local:2379 exceeded header timeout



md5-82555cea8f347d49ca3cf04a54380de3



Client: &version.Version{SemVer:"v2.11.0", GitCommit:"2e55dbe1fdb5fdb96b75ff144a339489417b146b", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.11.0", GitCommit:"2e55dbe1fdb5fdb96b75ff144a339489417b146b", GitTreeState:"clean"}



md5-62a501ae9172f51f46215b676fc07e57



Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.3", GitCommit:"a4529464e4629c21224b3d52edfe0ea91b072862", GitTreeState:"clean", BuildDate:"2018-09-09T18:02:47Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.6+IKS", GitCommit:"002d263ed027db260968616b951fb46f2bab9bb1", GitTreeState:"clean", BuildDate:"2019-01-09T08:07:22Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}

[carrodher]

What is the output of the helm install command?

$ helm install bitnami/etcd --set persistence.enabled=false
NAME:   existing-pig
LAST DEPLOYED: Fri Feb  1 12:43:12 2019
NAMESPACE: default
STATUS: DEPLOYED

RESOURCES:
==> v1/Secret
NAME               TYPE    DATA  AGE
existing-pig-etcd  Opaque  1     1s

==> v1/Service
NAME                        TYPE       CLUSTER-IP     EXTERNAL-IP  PORT(S)            AGE
existing-pig-etcd-headless  ClusterIP  None           <none>       2379/TCP,2380/TCP  1s
existing-pig-etcd           ClusterIP  10.105.60.206  <none>       2379/TCP,2380/TCP  1s

==> v1beta2/StatefulSet
NAME               DESIRED  CURRENT  AGE
existing-pig-etcd  1        1        1s

==> v1/Pod(related)
NAME                 READY  STATUS   RESTARTS  AGE
existing-pig-etcd-0  0/1    Pending  0         0s


NOTES:


** Please be patient while the chart is being deployed **

etcd can be accessed via port 2379 on the following DNS name from within your cluster:

    existing-pig-etcd.default.svc.cluster.local

To set a key run the following command:

    export POD_NAME=$(kubectl get pods --namespace default -l "app=etcd" -o jsonpath="{.items[0].metadata.name}")
    kubectl exec -it $POD_NAME -- etcdctl set /message Hello

To get a key run the following command:

    export POD_NAME=$(kubectl get pods --namespace default -l "app=etcd" -o jsonpath="{.items[0].metadata.name}")
    kubectl exec -it $POD_NAME -- etcdctl get /message

To connect to your etcd server from outside the cluster execute the following commands:

    kubectl port-forward --namespace default svc/existing-pig-etcd 2379:2379 &
    echo "etcd URL: http://127.0.0.1:2379"



 * As rbac is enabled you should add the flag `-u root:$ETCD_ROOT_PASSWORD` to the etcdctl commands. Export the password with this command:

    export ETCD_ROOT_PASSWORD=$(kubectl get secret --namespace default existing-pig-etcd -o jsonpath="{.data.etcd-root-password}" | base64 --decode)

In my case above output is shown. I want to check if there is some error or warning message

[nrxr]

Hi @huikang, this issue is fixed in the PR #1039. See more info about this in issue #1040.

[carrodher]

PR #1039 was merged, can you try again using this new version?

[huikang]

sire, will try and report back. Thanks

Sent from my iPhone

On Feb 4, 2019, at 5:56 AM, Carlos Rodríguez Hernández [e-mail] wrote:

PR #1039 was merged, can you try again using this new version?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[krwrang]

Having similar issues here, even after PR #1039.

helm install output:

NAME:   config-store
LAST DEPLOYED: Mon Feb  4 14:17:39 2019
NAMESPACE: default
STATUS: DEPLOYED

RESOURCES:
==> v1/Secret
NAME               TYPE    DATA  AGE
config-store-etcd  Opaque  1     1s

==> v1/Service
NAME                        TYPE       CLUSTER-IP    EXTERNAL-IP  PORT(S)            AGE
config-store-etcd-headless  ClusterIP  None          <none>       2379/TCP,2380/TCP  1s
config-store-etcd           ClusterIP  10.43.122.29  <none>       2379/TCP,2380/TCP  1s

==> v1beta2/StatefulSet
NAME               DESIRED  CURRENT  AGE
config-store-etcd  5        1        1s

==> v1/Pod(related)
NAME                 READY  STATUS             RESTARTS  AGE
config-store-etcd-0  0/1    ContainerCreating  0         1s


NOTES:


** Please be patient while the chart is being deployed **

etcd can be accessed via port 2379 on the following DNS name from within your cluster:

    config-store-etcd.default.svc.cluster.local

To set a key run the following command:

    export POD_NAME=$(kubectl get pods --namespace default -l "app=etcd" -o jsonpath="{.items[0].metadata.name}")
    kubectl exec -it $POD_NAME -- etcdctl set /message Hello

To get a key run the following command:

    export POD_NAME=$(kubectl get pods --namespace default -l "app=etcd" -o jsonpath="{.items[0].metadata.name}")
    kubectl exec -it $POD_NAME -- etcdctl get /message

To connect to your etcd server from outside the cluster execute the following commands:

    kubectl port-forward --namespace default svc/config-store-etcd 2379:2379 &
    echo "etcd URL: http://127.0.0.1:2379"



 * As rbac is enabled you should add the flag -u root:$ETCD_ROOT_PASSWORD` to the etcdctl commands. Export the password with this command:

    export ETCD_ROOT_PASSWORD=$(kubectl get secret --namespace default config-store-etcd -o jsonpath="{.data.etcd-root-password}" | base64 --decode)

Getting the same log entries as huikang. If I run with persistence (ceph), node-0 will start failing with this error after restarting once from the previous error:

==> The ID of the host is 0
==> Data exists. Re-joining etcd member
cat: /bitnami/etcd/data/member_id: No such file or directory

[nrxr]

@krwrang does it works with more nodes? Does it fail only when there's one node?

[krwrang]

@krwrang does it works with more nodes? Does it fail only when there's one node?

@nrxr I'm running with 5 nodes, haven't tested a 1 node scenario.

[nrxr]

@krwrang what version of kubernetes are you using? I tried it with a server on v1.13.3.

Could you try by adding echo's to the statefulset.yaml template? That way is easier to see in which of the steps it's actually failing.

It's failing somewhere around here: https://github.com/bitnami/charts/blob/dc28ef439a987808cd924d1186c7f933723db8e3/bitnami/etcd/templates/statefulset.yaml#L110-L113

So what I suggest is cloning this repo, adding echos in each step with different messages so you know before and after what step it's failing. I think is not in line 107 which is the one I modified but after, that's why adding an echo in line 110 saying something like echo "reached 110 would clear up if the PR actually added a regression or something.

[huikang]

It is working now. The command I was using is helm install bitnami/etcd --set persistence.enabled=false --set statefulset.replicaCount=3

[krwrang]

@nrxr I'm running on v1.11.5.

Step from line 110 is the one failing. I'll try and find out if the etcd is starting properly.

[krwrang]

Some more findings:

• Single node etcd cluster works
• A multi-node etcd cluster fails on default settings. When I try to run etcdctl cluster-health during the bootstrap, this is what I get:

[foranamo@latlng ~]$ kubectl --kubeconfig=/tmp/kube_config_cluster.yml exec -it config-store-etcd-0 -- etcdctl cluster-health
cluster may be unhealthy: failed to list members
Error:  client: etcd cluster is unavailable or misconfigured; error #0: client: endpoint http://127.0.0.1:2379 exceeded header timeout
; error #1: dial tcp 127.0.0.1:4001: connect: connection refused

error #0: client: endpoint http://127.0.0.1:2379 exceeded header timeout
error #1: dial tcp 127.0.0.1:4001: connect: connection refused

command terminated with exit code 4

After the couple of seconds of sleep pass, the pod init fails.
I tried to set a longer sleep and a somewhat short readinessProbe, so that another node starts while node-0 is still sleeping before adding the root user.

Running cluster-health in a loop, this is what I get when other nodes come up: (while still waiting to add root user):

[foranamo@latlng ~]$ kubectl --kubeconfig=/tmp/kube_config_cluster.yml exec -it config-store-etcd-0 -- etcdctl cluster-health
member 1202ac0529dfffb0 is unreachable: no available published client urls
member 37a71c129b43d4b0 is healthy: got healthy result from http://config-store-etcd-1.config-store-etcd-headless.default.svc.cluster.local:2379
member 9536ccc490fe4903 is healthy: got healthy result from http://config-store-etcd-2.config-store-etcd-headless.default.svc.cluster.local:2379
member e1adfa04c78dea09 is healthy: got healthy result from http://config-store-etcd-0.config-store-etcd-headless.default.svc.cluster.local:2379
member effe63defd99ec4e is unreachable: no available published client urls
cluster is degraded
command terminated with exit code 5

Some seconds pass for the sleep and it now successfully adds the root user and other authentication details. I'm open to suggestions on things to test.

Update: So turns out disabling the readinessProbe will enable the cluster to start, if persistence is disabled. If the probe is enabled, it will delay starting the next nodes long enough that there is no cluster for etcdctl to work on, resulting in the crash, which then results in node-0 never starting if persistence is enabled. This will continue to be broken if persistence is enabled, since the creation of the volume claims takes long enough (at least with ceph) that the second node does not start until the 5 second sleep has passed before trying etcdctl.

[stale[bot]]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[carrodher]

Hi @krwrang thanks for all detailed explanation, once #1043 was merged, is still the issue present in the latest version?

[krwrang]

Hi @carrodher! After merging the PR I haven't had any issues getting etcd up and running with chart on multiple different clusters.

[carrodher]

Nice, thanks, it's good to know. @huikang Do you think is there something pending or can we close the issue?

[stale[bot]]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[stale[bot]]

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.