etcdmain unable to find existing member certificate file

[cnkuyan]

Server version

 $ which etcd2
/usr/bin/etcd2
$ /usr/bin/etcd2 --version
etcd Version: 2.3.8
Git SHA: 7e4fc7e
Go Version: go1.7.6
Go OS/Arch: linux/amd64
core@coreos-512mb-fra1-01 ~ $ 
etcd2 --version
etcd Version: 2.3.8
Git SHA: 7e4fc7e
Go Version: go1.7.6
Go OS/Arch: linux/amd64

Important :
The etcdmain version shown by the image pulled from the git repo in the journalctl output is different from above though (etcdmain: etcd Version: 3.2.11 )

coreos version

$ cat  /etc/os-release
NAME="Container Linux by CoreOS"
ID=coreos
VERSION=1649.0.0
VERSION_ID=1649.0.0
BUILD_ID=2018-01-05-0906
PRETTY_NAME="Container Linux by CoreOS 1649.0.0 (Ladybug)"
ANSI_COLOR="38;5;75"
HOME_URL="https://coreos.com/"
BUG_REPORT_URL="https://issues.coreos.com"
COREOS_BOARD="amd64-usr"

Running etcd-member.service as system drop-in ( ignition script automatically generated by containerlinux transpiler . )

$ cat  /etc/systemd/system/etcd-member.service.d/20-clct-etcd-member.conf

[Unit]
Requires=coreos-metadata.service
After=coreos-metadata.service

[Service]
EnvironmentFile=/run/metadata/coreos
ExecStart=
ExecStart=/usr/lib/coreos/etcd-wrapper $ETCD_OPTS \
  --name="ocore-noup1" \
  --listen-peer-urls="https://${COREOS_DIGITALOCEAN_IPV4_PUBLIC_0}:2380, https://${COREOS_DIGITALOCEAN_IPV4_PUBLIC_0}:7001" \
  --listen-client-urls="https://0.0.0.0:2379, https://0.0.0.0:4001" \
  --initial-advertise-peer-urls="https://${COREOS_DIGITALOCEAN_IPV4_PUBLIC_0}:2380" \
  --advertise-client-urls="https://${COREOS_DIGITALOCEAN_IPV4_PUBLIC_0}:2379" \
  --discovery="https://discovery.etcd.io/b1f5b0c8c3ca950937f1ce3cc5557934" \
  --cert-file="/var/ssl/certs/ocore-noup1-server.pem" \
  --key-file="/var/ssl/keys/ocore-noup1-server-key.pem" \
  --client-cert-auth=true \
  --trusted-ca-file="/var/ssl/certs/ca.pem" \
  --peer-cert-file="/var/ssl/certs/ocore-noup1-member.pem" \
  --peer-key-file="/var/ssl/keys/ocore-noup1-member-key.pem" \
  --peer-client-cert-auth=true \
  --peer-trusted-ca-file="/var/ssl/certs/ca.pem"

error log :

$ journalctl -o cat -u etcd-member.service
++ id -u etcd
+ exec /usr/bin/rkt run --uuid-file-save=/var/lib/coreos/etcd-member-wrapper.uuid --trust-keys-from-https --mount volume=coreos-systemd-dir,target=/run/systemd/system --volume coreos-systemd-dir,kind=host
2018-01-15 09:55:28.194035 I | pkg/flags: recognized and used environment variable ETCD_DATA_DIR=/var/lib/etcd
2018-01-15 09:55:28.206543 I | pkg/flags: recognized environment variable ETCD_NAME, but unused: shadowed by corresponding flag
2018-01-15 09:55:28.206767 W | pkg/flags: unrecognized environment variable ETCD_USER=etcd
2018-01-15 09:55:28.206969 W | pkg/flags: unrecognized environment variable ETCD_IMAGE_TAG=v3.2.11
2018-01-15 09:55:28.207183 I | etcdmain: etcd Version: 3.2.11
2018-01-15 09:55:28.207359 I | etcdmain: Git SHA: 1e1dbb2
2018-01-15 09:55:28.207491 I | etcdmain: Go Version: go1.8.5
2018-01-15 09:55:28.207650 I | etcdmain: Go OS/Arch: linux/amd64
2018-01-15 09:55:28.207813 I | etcdmain: setting maximum number of CPUs to 1, total number of available CPUs is 1
2018-01-15 09:55:28.209886 I | embed: peerTLS: cert = /var/ssl/certs/ocore-noup1-member.pem, key = /var/ssl/keys/ocore-noup1-member-key.pem, ca = , trusted-ca = /var/ssl/certs/ca.pem, client-cert-auth = t
2018-01-15 09:55:28.217025 C | etcdmain: open /var/ssl/certs/ocore-noup1-member.pem: no such file or directory
etcd-member.service: Main process exited, code=exited, status=1/FAILURE
etcd-member.service: Failed with result 'exit-code'.
Failed to start etcd (System Application Container).
etcd-member.service: Service hold-off time over, scheduling restart.
etcd-member.service: Scheduled restart job, restart counter is at 7.
Stopped etcd (System Application Container).
Starting etcd (System Application Container)...
"cfbb1963-d71c-4453-bf53-fc11eb69b051"

file is there, owned by root, and readable :

$ stat /var/ssl/certs/ocore-noup1-member.pem
  File: '/var/ssl/certs/ocore-noup1-member.pem'
  Size: 1322            Blocks: 8          IO Block: 4096   regular file
Device: fd09h/64777d    Inode: 52          Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Context: system_u:object_r:unlabeled_t:s0
Access: 2018-01-14 21:29:20.000000000 +0000
Modify: 2018-01-14 21:21:57.000000000 +0000
Change: 2018-01-14 21:21:57.000000000 +0000
 Birth: -
coreos-512mb-fra1-01 etcd-member.service.d #

Any help or idea is appreciated ..
Thanks

[hexfusion]

Any help or idea is appreciated ..

@cnkuyan What is the perm of the containing directory? This still seems like a perm issue. Does the following succeed? A file can have read perms but not be readable if the containing directory perms are more strict.

sudo su etcd -c"cat /var/ssl/certs/ocore-noup1-member.pem"

ls -ld /var/ssl/certs/

[cnkuyan]

@hexfusion Good point. Here is the output of the cat command, run as user etcd

 $ sudo su etcd -c"cat /var/ssl/certs/ocore-noup1-member.pem"
This account is currently not available.
~ $

Looks like there is no such user. Neither have I created one with my ignition script .

and:

 $ ls -ld /var/ssl/certs/
drwxr-xr-x. 2 root root 4096 Jan 14 21:21 /var/ssl/certs/
 $

I was already suspicious about the following line the service log too ..:

2018-01-15 10:37:19.532269 W | pkg/flags: unrecognized environment variable ETCD_USER=etcd

Isn't the service running as root ?

[hexfusion]

Isn't the service running as root ?

@cnkuyan you need to follow the trail a bit further

ExecStart=/usr/lib/coreos/etcd-wrapper $ETCD_OPTS \

So you are passing params to etcd-wrapper which in turn fires rkt. Review that script /usr/lib/coreos/etcd-wrapper and it should become more clear re $ETCD_USER.

[cnkuyan]

@hexfusion First of all , I'm not doing anything special with $ETCD_OPTS env variable that is referenced by the etcd-wrapper script. It is empty.

By the way, I managed to get etcd service on this node up and running by passing --peer-auto-tls config flag at execution, and removing the manually provided peer cert and key files ..

But nothing stops me from experimenting , and passing an env variable by adding this to the drop-in service file , therefore changing the user:

Environment="ETCD_USER=root"

but still no joy, because , as the logs keep pointing out to me that, certain env variables are adamantly not recognized by the script - or the rkt image , I don't know the depths from which the error is being thrown from . ( I choose to overlook the lines saying " this server is already initialized as member before, starting as etcd member.. ", and 'invalid dir' . Because the server has been started already once before.. )

 $ sudo journalctl -f  -o cat  -u etcd-member.service
++ id -u root
+ exec /usr/bin/rkt run --uuid-file-save=/var/lib/coreos/etcd-member-wrapper.uuid --trust-keys-from-https --mount volume=coreos-systemd-dir,target=/run/systemd/system --volume coreos-systemd-dir,kind=host,source=/run/systemd/system,readOnly=true --mount volume=coreos-notify,target=/run/systemd/notify --volume coreos-notify,kind=host,source=/run/systemd/notify --set-env=NOTIFY_SOCKET=/run/systemd/notify --volume coreos-data-dir,kind=host,source=/var/lib/etcd,readOnly=false --volume coreos-etc-ssl-certs,kind=host,source=/etc/ssl/certs,readOnly=true --volume coreos-usr-share-certs,kind=host,source=/usr/share/ca-certificates,readOnly=true --volume coreos-etc-hosts,kind=host,source=/etc/hosts,readOnly=true --volume coreos-etc-resolv,kind=host,source=/etc/resolv.conf,readOnly=true --mount volume=coreos-data-dir,target=/var/lib/etcd --mount volume=coreos-etc-ssl-certs,target=/etc/ssl/certs --mount volume=coreos-usr-share-certs,target=/usr/share/ca-certificates --mount volume=coreos-etc-hosts,target=/etc/hosts --mount volume=coreos-etc-resolv,target=/etc/resolv.conf --inherit-env --stage1-from-dir=stage1-fly.aci quay.io/coreos/etcd:v3.2.11 --user=0 -- --name=ocore-noup1 '--listen-peer-urls=https://165.227.143.126:2380, https://165.227.143.126:7001' '--listen-client-urls=https://0.0.0.0:2379, https://0.0.0.0:4001' --initial-advertise-peer-urls=https://165.227.143.126:2380 --advertise-client-urls=https://165.227.143.126:2379 --discovery=https://discovery.etcd.io/b1f5b0c8c3ca950937f1ce3cc5557934 --cert-file=/var/ssl/certs/ocore-noup1-server.pem --key-file=/var/ssl/keys/ocore-noup1-server-key.pem --client-cert-auth=true --trusted-ca-file=/var/ssl/certs/ca.pem --peer-cert-file=/var/ssl/certs/ocore-noup1-member.pem --peer-key-file=/var/ssl/keys/ocore-noup1-member-key.pem --peer-client-cert-auth=true --peer-trusted-ca-file=/var/ssl/certs/ca.pem
2018-01-15 14:54:54.969303 I | pkg/flags: recognized and used environment variable ETCD_DATA_DIR=/var/lib/etcd
2018-01-15 14:54:54.969979 I | pkg/flags: recognized environment variable ETCD_NAME, but unused: shadowed by corresponding flag
2018-01-15 14:54:54.976456 W | pkg/flags: unrecognized environment variable ETCD_USER=root
2018-01-15 14:54:54.976557 W | pkg/flags: unrecognized environment variable ETCD_IMAGE_TAG=v3.2.11
2018-01-15 14:54:54.976691 I | etcdmain: etcd Version: 3.2.11
2018-01-15 14:54:54.976764 I | etcdmain: Git SHA: 1e1dbb2
2018-01-15 14:54:54.976839 I | etcdmain: Go Version: go1.8.5
2018-01-15 14:54:54.976919 I | etcdmain: Go OS/Arch: linux/amd64
2018-01-15 14:54:54.976986 I | etcdmain: setting maximum number of CPUs to 1, total number of available CPUs is 1
2018-01-15 14:54:54.977163 W | etcdmain: found invalid file/dir fixtures under data dir /var/lib/etcd (Ignore this if you are upgrading etcd)
2018-01-15 14:54:54.977370 N | etcdmain: the server is already initialized as member before, starting as etcd member...
2018-01-15 14:54:54.977470 I | embed: peerTLS: cert = /var/ssl/certs/ocore-noup1-member.pem, key = /var/ssl/keys/ocore-noup1-member-key.pem, ca = , trusted-ca = /var/ssl/certs/ca.pem, client-cert-auth = true
2018-01-15 14:54:54.977641 C | etcdmain: open /var/ssl/certs/ocore-noup1-member.pem: no such file or directory

In summary, what is going on with the automatically generated etcd-wrapper execution line ?

[hexfusion]

@hexfusion First of all , I'm not doing anything special with $ETCD_OPTS env variable that is referenced by the etcd-wrapper script. It is empty.

I just meant you are passing params to the script.

as the logs keep pointing out to me that, certain env variables are adamantly not recognized by the script

This is simply a notice etcd will log any var prefixed with ETCD_ that is not used internally by etcd. But this is not an error.

The main issue here is that rkt is running etcd container. rkt container can't find the file as it is on your host not in the container. Perhaps you need to pass --volume to rkt as well so that it can see the file in the container? auto-tls circumvents the issue but if you want to use the certs the container needs to have access to them.

[hexfusion]

@cnkuyan I am going to close this as it is a configuration problem vs etcd issue. Please feel free to continue the conversation here if you have more feedback or issues.

[hexfusion]

I did a little more research and it seems you can pass the ETCD_SSL_DIR var here.

https://github.com/coreos/coreos-overlay/blob/master/app-admin/etcd-wrapper/files/etcd-wrapper#L71

Which would resolve to /etc/ssl/certs in the container. I am assuming that the script is the same but perhaps this helps you.

[cnkuyan]

@hexfusion Thank you , that was actually what I needed . Or, I was going to map a volume manually in the ignition script.
Sure, let's go ahead and close this as a configuration problem .

At this point though, I'd like to know ask how to pass this environment variable in the 'etcd' block of the ContainerLinux source config, so that it gets transpiled into the final ignition config output by the ct tool ..

So far I have manually added the following line into my ignition script:

Environment="ETCD_SSL_DIR=/var/ssl/certs"

But having to manually put this, interferes with my automated workflow .

[cnkuyan]

@hexfusion I'd like to continue the conversation due to an issue .

I'm running etcd with the wrapper script, and passing the the host folder that holds my self-signed certificates, and the trusted ca cert. Here is the service drop in :

[Unit]
Requires=coreos-metadata.service
After=coreos-metadata.service

[Service]
EnvironmentFile=/run/metadata/coreos
Environment="ETCD_SSL_DIR=/var/ssl/certs"
ExecStart=
ExecStart=/usr/lib/coreos/etcd-wrapper $ETCD_OPTS \
  --name="ocore-noup1" \
  --listen-client-urls="https://0.0.0.0:2379, https://0.0.0.0:4001" \
  --initial-advertise-peer-urls="https://${COREOS_DIGITALOCEAN_IPV4_PUBLIC_0}:2380" \
  --advertise-client-urls="https://${COREOS_DIGITALOCEAN_IPV4_PUBLIC_0}:2379" \
  --discovery="https://discovery.etcd.io/b528cca6ebf432952ae5e520051df363" \
  --cert-file="/etc/ssl/certs/ocore-noup1-server.pem" \
  --key-file="/etc/ssl/certs/ocore-noup1-server-key.pem" \
  --client-cert-auth=true \
  --trusted-ca-file="/etc/ssl/certs/ca.pem"

The etcd-wrapper seemingly recognizes my flags, and folder locations , but issues a 'bad certificate error' , as follows:

2018-01-17 09:57:13.555667 W | pkg/flags: unrecognized environment variable ETCD_SSL_DIR=/var/ssl/certs
2018-01-17 09:57:13.555753 W | pkg/flags: unrecognized environment variable ETCD_USER=etcd
2018-01-17 09:57:13.556290 W | pkg/flags: unrecognized environment variable ETCD_IMAGE_TAG=v3.2.11
2018-01-17 09:57:13.556451 I | etcdmain: etcd Version: 3.2.11
2018-01-17 09:57:13.556549 I | etcdmain: Git SHA: 1e1dbb2
2018-01-17 09:57:13.556618 I | etcdmain: Go Version: go1.8.5
2018-01-17 09:57:13.556699 I | etcdmain: Go OS/Arch: linux/amd64
2018-01-17 09:57:13.556776 I | etcdmain: setting maximum number of CPUs to 1, total number of available CPUs is 1
2018-01-17 09:57:13.574173 W | etcdmain: found invalid file/dir fixtures under data dir /var/lib/etcd (Ignore this if you are upgrading etcd)
2018-01-17 09:57:13.574458 N | etcdmain: the server is already initialized as member before, starting as etcd member...
2018-01-17 09:57:13.575216 I | embed: listening for peers on http://localhost:2380
2018-01-17 09:57:13.575395 I | embed: listening for client requests on 0.0.0.0:2379
2018-01-17 09:57:13.575538 I | embed: listening for client requests on 0.0.0.0:4001
2018-01-17 09:57:13.577601 W | etcdserver: discovery token ignored since a cluster has already been initialized. Valid log found at "/var/lib/etcd/member/wal"
2018-01-17 09:57:13.577782 I | etcdserver: name = ocore-noup1
2018-01-17 09:57:13.577937 I | etcdserver: data dir = /var/lib/etcd
2018-01-17 09:57:13.578013 I | etcdserver: member dir = /var/lib/etcd/member
2018-01-17 09:57:13.578090 I | etcdserver: heartbeat = 100ms
2018-01-17 09:57:13.578176 I | etcdserver: election = 1000ms
2018-01-17 09:57:13.578253 I | etcdserver: snapshot count = 100000
2018-01-17 09:57:13.578328 I | etcdserver: discovery URL= https://discovery.etcd.io/b528cca6ebf432952ae5e520051df363
2018-01-17 09:57:13.578410 I | etcdserver: advertise client URLs = https://139.59.159.241:2379
2018-01-17 09:57:13.625716 I | etcdserver: restarting member 8e9e05c52164694d in cluster cdf818194e3a8c32 at commit index 3899
2018-01-17 09:57:13.626449 I | raft: 8e9e05c52164694d became follower at term 6
2018-01-17 09:57:13.626583 I | raft: newRaft 8e9e05c52164694d [peers: [], term: 6, commit: 3899, applied: 0, lastindex: 3899, lastterm: 6]
2018-01-17 09:57:13.638786 W | auth: simple token is not cryptographically signed
2018-01-17 09:57:13.640965 I | etcdserver: starting server... [version: 3.2.11, cluster version: to_be_decided]
2018-01-17 09:57:13.641198 I | embed: ClientTLS: cert = /etc/ssl/certs/ocore-noup1-server.pem, key = /etc/ssl/certs/ocore-noup1-server-key.pem, ca = , trusted-ca = /etc/ssl/certs/ca.pem, client-cert-auth = true
2018-01-17 09:57:13.671624 I | etcdserver/membership: added member 8e9e05c52164694d [http://localhost:2380] to cluster cdf818194e3a8c32
2018-01-17 09:57:13.672121 N | etcdserver/membership: set the initial cluster version to 3.2
2018-01-17 09:57:13.672300 I | etcdserver/api: enabled capabilities for version 3.2
2018-01-17 09:57:13.827257 I | raft: 8e9e05c52164694d is starting a new election at term 6
2018-01-17 09:57:13.827582 I | raft: 8e9e05c52164694d became candidate at term 7
2018-01-17 09:57:13.827725 I | raft: 8e9e05c52164694d received MsgVoteResp from 8e9e05c52164694d at term 7
2018-01-17 09:57:13.827901 I | raft: 8e9e05c52164694d became leader at term 7
2018-01-17 09:57:13.828024 I | raft: raft.node: 8e9e05c52164694d elected leader 8e9e05c52164694d at term 7
2018-01-17 09:57:13.829046 I | etcdserver: published {Name:ocore-noup1 ClientURLs:[https://139.59.159.241:2379]} to cluster cdf818194e3a8c32
Started etcd (System Application Container).
2018-01-17 09:57:13.833748 I | embed: ready to serve client requests
2018-01-17 09:57:13.834347 I | etcdserver/api/v3rpc: dialing to target with scheme: ""
2018-01-17 09:57:13.834472 I | etcdserver/api/v3rpc: could not get resolver for scheme: ""
2018-01-17 09:57:13.834972 I | embed: serving client requests on [::]:2379
2018-01-17 09:57:13.835126 I | embed: ready to serve client requests
2018-01-17 09:57:13.835344 I | etcdserver/api/v3rpc: dialing to target with scheme: ""
2018-01-17 09:57:13.835465 I | etcdserver/api/v3rpc: could not get resolver for scheme: ""
2018-01-17 09:57:13.835751 I | embed: serving client requests on [::]:4001
2018-01-17 09:57:13.912684 I | etcdserver/api/v3rpc: Failed to dial 0.0.0.0:4001: connection error: desc = "transport: authentication handshake failed: remote error: tls: bad certificate"; please retry.
2018-01-17 09:57:13.913488 I | etcdserver/api/v3rpc: Failed to dial 0.0.0.0:2379: connection error: desc = "transport: authentication handshake failed: remote error: tls: bad certificate"; please retry.

I can connect and interact with the etcd2 (2.3.7) , when executed direct from the command line on coreos host .
Before executing the daemond, I did the following on coreos host:

$ sudo cp /var/ssl/certs/ca.pem /etc/ssl/certs
$ update-ca-certificates
...

Will executing update-ca-certificates in rkt image based etcd2 ( as started by etcd-wrapper ) solve the issue, or is this another configuration problem on my side ?

Any help or idea is appreciated.

[cnkuyan]

Solved.

Copying of custom certificates over to folder /etc/ssl/certs and running update-ca-certificates on the host solves the issue.

I executed the commands manually, the update command should go as ExecPre statement in the service file.

[seh]

I'm facing the same problem as @cnkuyan: the "etcd" account is not mentioned in _/etc/passwd_, though its ID is known:

$ id -u etcd
232
$ sudo su etcd -c"echo 'Hi.'"
This account is currently not available.

Where is this "etcd" user defined? Am I supposed to "activate" it somehow? I haven't found any Container Linux- or _etcd_-related documentation that instructs users to create this account.

@hexfusion, I'm seeing this with the "stable" Container Linux channel version 1688.5.3.

[seh]

Adding detail:

$ getent passwd etcd
etcd:x:232:232::/dev/null:/sbin/nologin

[pietro]

I had he same issue and I found a fix so I'm writing it here for posterity.

CoreOS/Container Linux uses /usr/lib/coreos/etcd-wrapper to run etcd using rkt. That script creates a volume called coreos-etc-ssl-certs and mounts said volume at /etc/ssl/certs inside the etcd container.

The host directory used as source for the volume is controlled by the environment variable ETCD_SSL_DIR and the default value is /etc/ssl/certs.

If you are placing your certs somewhere else than /etc/ssl/certs on the host you need to do 2 things in order to get etcd to work:

1. Set ETCD_SSL_DIR to the host directory where you placed your certs. Add that to the etcd-member service environment.

2. Use /etc/ssl/certs as the base directory for the cert related options on etcd. Example: --trusted-ca-file=/etc/ssl/certs/ca.crt.

This is a drop in that will correctly start etcd with TLS when the certificates have been placed on /etc/ssl/etcd on the host:

[Unit]
Requires=coreos-metadata.service
After=coreos-metadata.service

[Service]
Environment="ETCD_SSL_DIR=/etc/ssl/etcd"
EnvironmentFile=/run/metadata/coreos
ExecStart=
ExecStart=/usr/lib/coreos/etcd-wrapper \
  --name=${name} \
  ---advertise-client-urlshttps://10.53.70.188:2379 \
  --cert-file=/etc/ssl/certs/server.crt \
  --client-cert-auth=true \
  --key-file=/etc/ssl/certs/server.key \
  --peer-cert-file=/etc/ssl/certs/peer.crt \
  --peer-key-file=/etc/ssl/certs/peer.key \
  --peer-trusted-ca-file=/etc/ssl/certs/ca.crt \
  --peer-client-cert-auth=true \
  --trusted-ca-file=/etc/ssl/certs/ca.crt \
  --initial-cluster=core0=https://10.53.70.188:2380 \
  --initial-advertise-peer-urls=https://10.53.70.188:2380 \
  --listen-client-urls=https://0.0.0.0:2379 \
  --listen-peer-urls=https://0.0.0.0:2380