Kind: Error: secret "memberlist" not found

Created on 28 Mar 2020 · 21Comments · Source: kubernetes-sigs/kind

I'm trying to deploy the new metallb release, and after following the steps:

kubectl apply -f https://raw.githubusercontent.com/google/metallb/v0.9.3/manifests/namespace.yaml
kubectl apply -f https://raw.githubusercontent.com/google/metallb/v0.9.3/manifests/metallb.yaml
# On first install only
kubectl create secret generic -n metallb-system memberlist --from-literal=secretkey="$(openssl rand -base64 128)"

$ kubectl get pods --all-namespaces

Returning the following error:

NAMESPACE            NAME                                          READY   STATUS                       RESTARTS   AGE
default              traefik-6588bbf6-dvd6z                        1/1     Running                      0          4m13s
kube-system          coredns-6955765f44-krc8w                      1/1     Running                      0          5m56s
kube-system          coredns-6955765f44-n48jn                      1/1     Running                      0          5m56s
kube-system          etcd-local-control-plane                      1/1     Running                      0          6m10s
kube-system          kindnet-cs8w9                                 1/1     Running                      0          5m56s
kube-system          kube-apiserver-local-control-plane            1/1     Running                      0          6m10s
kube-system          kube-controller-manager-local-control-plane   1/1     Running                      0          6m10s
kube-system          kube-proxy-xxfzm                              1/1     Running                      0          5m56s
kube-system          kube-scheduler-local-control-plane            1/1     Running                      0          6m10s
local-path-storage   local-path-provisioner-7745554f7f-jdf72       1/1     Running                      0          5m56s
metallb-system       controller-5c9894b5cd-mzx8q                   1/1     Running                      0          5m48s
metallb-system       speaker-dhbtb                                 0/1     CreateContainerConfigError   0          5m10s

The pod logs:

$ kubectl describe pods -n metallb-system

Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  speaker-token-zpcrv:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  speaker-token-zpcrv
    Optional:    false
QoS Class:       Guaranteed
Node-Selectors:  beta.kubernetes.io/os=linux
Tolerations:     node-role.kubernetes.io/master:NoSchedule
                 node.kubernetes.io/disk-pressure:NoSchedule
                 node.kubernetes.io/memory-pressure:NoSchedule
                 node.kubernetes.io/network-unavailable:NoSchedule
                 node.kubernetes.io/not-ready:NoExecute
                 node.kubernetes.io/pid-pressure:NoSchedule
                 node.kubernetes.io/unreachable:NoExecute
                 node.kubernetes.io/unschedulable:NoSchedule
Events:
  Type     Reason     Age                   From                          Message
  ----     ------     ----                  ----                          -------
  Normal   Scheduled  8m4s                  default-scheduler             Successfully assigned metallb-system/speaker-dhbtb to local-control-plane
  Normal   Pulled     6m8s (x8 over 7m54s)  kubelet, local-control-plane  Successfully pulled image "metallb/speaker:v0.9.3"
  Warning  Failed     6m8s (x8 over 7m54s)  kubelet, local-control-plane  Error: secret "memberlist" not found
  Normal   Pulling    3m1s (x20 over 8m3s)  kubelet, local-control-plane  Pulling image "metallb/speaker:v0.9.3"

kinsupport

Source

andersonkxiass

Most helpful comment

this should fix it:

kubectl create secret generic -n metallb-system memberlist --from-literal=secretkey="$(openssl rand -base64 128)"

ucef29 on 20 Apr 2020

👍18

All 21 comments

How long did you wait?
What kind version and cluster options?

BenTheElder on 28 Mar 2020

Hey @BenTheElder, I've tried some times and I waited some minutes, the host SO it's Ubuntu 19

andersonkxiass on 30 Mar 2020

How did you install kind? What version?
Are you running kind create cluster with any options?

BenTheElder on 30 Mar 2020

I've installed using de default installation, GO111MODULE="on" go get sigs.k8s.io/[email protected].
Kind version v0.7.0
It's cluster created with default command kind create cluster

andersonkxiass on 30 Mar 2020

@andersonkxiass I can't reproduce it

kubectl get pods -n metallb-system
NAME                          READY   STATUS    RESTARTS   AGE
controller-5c9894b5cd-bd6gx   1/1     Running   0          61s
speaker-cqkwf                 1/1     Running   0          61s
speaker-fmgq5                 1/1     Running   0          61s
speaker-r9zxt                 1/1     Running   0          61s

what's the output of kubectl get secret -n metallb-system?

Is KIND running on the OS host or do you have some kind in docker or in kubernetes?

aojea on 7 Apr 2020

I can't reproduce this either.

$ kubectl get po -A
NAMESPACE            NAME                                         READY   STATUS    RESTARTS   AGE
kube-system          coredns-6955765f44-glfdl                     1/1     Running   0          13m
kube-system          coredns-6955765f44-mswc9                     1/1     Running   0          13m
kube-system          etcd-kind-control-plane                      1/1     Running   0          13m
kube-system          kindnet-lrbwm                                1/1     Running   0          13m
kube-system          kube-apiserver-kind-control-plane            1/1     Running   0          13m
kube-system          kube-controller-manager-kind-control-plane   1/1     Running   0          13m
kube-system          kube-proxy-6rmq2                             1/1     Running   0          13m
kube-system          kube-scheduler-kind-control-plane            1/1     Running   0          13m
local-path-storage   local-path-provisioner-85445b74d4-f9ggg      1/1     Running   0          13m
metallb-system       controller-5c9894b5cd-v2z6q                  1/1     Running   0          11s
metallb-system       speaker-ln9md                                1/1     Running   0          11s

BenTheElder on 7 Apr 2020

👍1

Is KIND running on the OS host or do you have some kind in docker or in kubernetes?

It's running on the OS host (Ubuntu 19).

I'll try again today, so I'll update here.

andersonkxiass on 8 Apr 2020

hmm, I run on ubuntu 19.10 at home fwiw. I've installed docker via the apt packages from docker inc, and I tested with kind v0.7.0 and your instructions.

BenTheElder on 17 Apr 2020

@BenTheElder and it worked for you? I've installed docker using sh script

curl -fsSL https://get.docker.com -o get-docker.sh

andersonkxiass on 17 Apr 2020

yes, i'm unable to reproduce.

$ git checkout v0.7.0
<snip>
$ make install
<snip>
$ kind version
kind v0.7.0 go1.13.6 linux/amd64
$ kind create cluster
Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.17.0) 🖼 
 ✓ Preparing nodes 📦  
 ✓ Writing configuration 📜 
 ✓ Starting control-plane 🕹️ 
 ✓ Installing CNI 🔌 
 ✓ Installing StorageClass 💾 
Set kubectl context to "kind-kind"
You can now use your cluster with:

kubectl cluster-info --context kind-kind

Not sure what to do next? 😅 Check out https://kind.sigs.k8s.io/docs/user/quick-start/
$ kubectl apply -f https://raw.githubusercontent.com/google/metallb/v0.9.3/manifests/namespace.yaml
namespace/metallb-system created
$ kubectl apply -f https://raw.githubusercontent.com/google/metallb/v0.9.3/manifests/metallb.yaml
podsecuritypolicy.policy/controller created
podsecuritypolicy.policy/speaker created
serviceaccount/controller created
serviceaccount/speaker created
clusterrole.rbac.authorization.k8s.io/metallb-system:controller created
clusterrole.rbac.authorization.k8s.io/metallb-system:speaker created
role.rbac.authorization.k8s.io/config-watcher created
role.rbac.authorization.k8s.io/pod-lister created
clusterrolebinding.rbac.authorization.k8s.io/metallb-system:controller created
clusterrolebinding.rbac.authorization.k8s.io/metallb-system:speaker created
rolebinding.rbac.authorization.k8s.io/config-watcher created
rolebinding.rbac.authorization.k8s.io/pod-lister created
daemonset.apps/speaker created
deployment.apps/controller created
$ kubectl create secret generic -n metallb-system memberlist --from-literal=secretkey="$(openssl rand -base64 128)"
secret/memberlist created
$ kubectl get po -A
NAMESPACE            NAME                                         READY   STATUS    RESTARTS   AGE
kube-system          coredns-6955765f44-fzzrn                     1/1     Running   0          53s
kube-system          coredns-6955765f44-nkjvc                     1/1     Running   0          53s
kube-system          etcd-kind-control-plane                      1/1     Running   0          66s
kube-system          kindnet-txtdb                                1/1     Running   0          53s
kube-system          kube-apiserver-kind-control-plane            1/1     Running   0          66s
kube-system          kube-controller-manager-kind-control-plane   1/1     Running   0          66s
kube-system          kube-proxy-kxwqq                             1/1     Running   0          53s
kube-system          kube-scheduler-kind-control-plane            1/1     Running   0          65s
local-path-storage   local-path-provisioner-7745554f7f-64vvb      1/1     Running   0          53s
metallb-system       controller-5c9894b5cd-998nn                  1/1     Running   0          42s
metallb-system       speaker-kzfbn                                1/1     Running   0          42s

BenTheElder on 17 Apr 2020

stupid question: is your openssl failing? did creating the secret work?

BenTheElder on 17 Apr 2020

test host information:

$ openssl version
OpenSSL 1.1.1c  28 May 2019
$ uname -a
Linux gibson 5.3.0-46-generic #38-Ubuntu SMP Fri Mar 27 17:37:05 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
$ docker version
Client: Docker Engine - Community
 Version:           19.03.8
 API version:       1.40
 Go version:        go1.12.17
 Git commit:        afacb8b7f0
 Built:             Wed Mar 11 01:25:55 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.8
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.17
  Git commit:       afacb8b7f0
  Built:            Wed Mar 11 01:24:26 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.13
  GitCommit:        7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683
$ cat /etc/os-release
NAME="Ubuntu"
VERSION="19.10 (Eoan Ermine)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 19.10"
VERSION_ID="19.10"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=eoan
UBUNTU_CODENAME=eoan

BenTheElder on 17 Apr 2020

@BenTheElder Okay, I ran now and it worked as expected, it really seems like my mistake, but I don't know what went wrong before.

Anyway, sorry and thank you for your support.

andersonkxiass on 17 Apr 2020

Glad it works
/close

On Fri, Apr 17, 2020, 12:06 PM Anderson K notifications@github.com wrote:

@BenTheElder https://github.com/BenTheElder Okay, I ran now and it
worked as expected, it really seems like my mistake, but I don't know what
went wrong before.

Anyway, sorry and thank you for your support.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/kubernetes-sigs/kind/issues/1449#issuecomment-615414330,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAHADK3B2AWPM7OGHMR3IPDRNCSDTANCNFSM4LVT2ABA
.

BenTheElder on 17 Apr 2020

@BenTheElder: Closing this issue.

In response to this:

Glad it works
/close

On Fri, Apr 17, 2020, 12:06 PM Anderson K notifications@github.com wrote:

@BenTheElder https://github.com/BenTheElder Okay, I ran now and it
worked as expected, it really seems like my mistake, but I don't know what
went wrong before.

Anyway, sorry and thank you for your support.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/kubernetes-sigs/kind/issues/1449#issuecomment-615414330,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAHADK3B2AWPM7OGHMR3IPDRNCSDTANCNFSM4LVT2ABA
.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-ci-robot on 17 Apr 2020

this should fix it:

kubectl create secret generic -n metallb-system memberlist --from-literal=secretkey="$(openssl rand -base64 128)"

ucef29 on 20 Apr 2020

👍18

this should fix it:

kubectl create secret generic -n metallb-system memberlist --from-literal=secretkey="$(openssl rand -base64 128)"

Got the same issue here, this one does the trick, thank you very much man

ohmyhusky on 12 May 2020

the same problem on Ubuntu 16.04.6 LTS

i wanted to install metallb as loadbalancer. the speaker pods told me the same message: Error: secret "memberlist" not found

is it a bug or a not provided function from the onboard tools?

kubectl version:
Client Version: v1.11.1
Server Version: v1.17.4

OpenSSL 1.0.2g 1 Mar 2016

Client: Docker Engine - Community
Version: 19.03.12
API version: 1.40
Go version: go1.13.10
Git commit: 48a66213fe
Built: Mon Jun 22 15:45:49 2020
OS/Arch: linux/amd64
Experimental: false

Server: Docker Engine - Community
Engine:
Version: 19.03.12
API version: 1.40 (minimum version 1.12)
Go version: go1.13.10
Git commit: 48a66213fe
Built: Mon Jun 22 15:44:20 2020
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.2.13
GitCommit: 7ad184331fa3e55e52b890ea95e65ba581ae3429
runc:
Version: 1.0.0-rc10
GitCommit: dc9208a3303feef5b3839f4323d9beb36df0a9dd
docker-init:
Version: 0.18.0
GitCommit: fec3683

MoBlockbuster on 13 Aug 2020

"memberlist" in "metallb-system" is not the name of a standard object, normal clusters do not provide this, no.

It is not a kind bug.

kubectl version:
Client Version: v1.11.1
Server Version: v1.17.4

That's a huge version skew. Kubernetes doesn't support that. Your kubectl is too old for your cluster.

BenTheElder on 13 Aug 2020

this should fix it:
kubectl create secret generic -n metallb-system memberlist --from-literal=secretkey="$(openssl rand -base64 128)"

Unless I've missed it in the docs, this step is only alluded to in https://metallb.universe.tf/installation/ and I had to find this issue in order to understand what the contents of the secret should be. Could this be made more explicit?