Etcd: Can't use users with colon in names

[cristifalcas]

We can't enable authentication in etcd with kubernetes/openshift because the CN in the certs they use to connect to etcd contais a colon: "system:master".

Is there any way to bypass this and create users with colon in name?

And it seems we can have users with strange names:

# etcdctl user add 'system\\\:master'
# etcdctl user list
system
system\\\

[hexfusion]

Hi @cristifalcas would you mind posting full logs showing this failure, please? Also can you note what version of etcd you are using?

[cristifalcas]

I can give the logs from openshift after enabling authentication in etcd:

E0503 21:51:34.423824    3232 status.go:64] apiserver received an error that is not an metav1.Status: etcdserver: permission denied
E0503 21:51:34.424157    3232 available_controller.go:295] v1.project.openshift.io failed with: etcdserver: permission denied
2018-05-03 21:51:34.443950 E | auth: invalid user name system:master for permission checking
E0503 21:51:34.444216    3232 status.go:64] apiserver received an error that is not an metav1.Status: etcdserver: permission denied
E0503 21:51:34.444578    3232 available_controller.go:295] v1alpha1.admissionregistration.k8s.io failed with: etcdserver: permission denied
2018-05-03 21:51:34.463843 E | auth: invalid user name system:master for permission checking
E0503 21:51:34.464153    3232 status.go:64] apiserver received an error that is not an metav1.Status: etcdserver: permission denied
E0503 21:51:34.464504    3232 available_controller.go:295] v1.networking.k8s.io failed with: etcdserver: permission denied
2018-05-03 21:51:34.484000 E | auth: invalid user name system:master for permission checking
E0503 21:51:34.484284    3232 status.go:64] apiserver received an error that is not an metav1.Status: etcdserver: permission denied
E0503 21:51:34.484646    3232 available_controller.go:295] v1beta1.admissionregistration.k8s.io failed with: etcdserver: permission denied
2018-05-03 21:51:34.503821 E | auth: invalid user name system:master for permission checking
E0503 21:51:34.504113    3232 status.go:64] apiserver received an error that is not an metav1.Status: etcdserver: permission denied

And I have no way to add this user.

I will try to redo the client certs used by the kube apiserver

[hexfusion]

Is there any way to bypass this and create users with colon in name?

@cristifalcas I didn't test etcdctl but I had no issues adding a user with colons using gRPC gateway. So this is not a limitation of etcd. I will look into etcdctl further but it feel likes an escaping issue. have you tried "foo\:bar" ? .. oops, I didn't realize you basically did that with your test above.

2018-05-03 20:32:48.116839 N | auth: added a new user: system:master
[..]
2018-05-03 20:32:48.281289 D | auth: authorized system:master

[hexfusion]

I see so the user:pass usage is the issue so basically system:master = user system with pass master. Not sure how we best handle that.

[hexfusion]

Perhaps we could add a --password="" flag in which case the --user="" string is used verbatium if the password flag is also set. I am not sure if we want to go to that length to support a colon in the username with etcdctl.

/cc @mitake how do you feel?

Post Note: if it can be done with gRPC gateway it should probably be supported by etcdctl or disabled entirly.

[cristifalcas]

Can you help me with a grpc gateway example? How to add the user?

And I need it to work with tls certs.

[cristifalcas]

This is what I'm trying:

curl --cacert /etc/kubernetes/pki/ca.crt --key ./admin-key.pem --cert ./admin.pem  https://$HOSTNAME:2379/v3alpha/auth/user/add     -X POST -d '{ "name":"root", "password":"toor" }'
{"Error":"connection error: desc = \"transport: remote error: bad certificate\"","Code":13}

The same certificates work with etcdctl client

And the cert:

# openssl x509 -noout -text   -in ./admin.pem 
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            7a:b5:2f:ac:49:68:5c:48:b3:19:e8:b9:f3:29:48:ef:7d:1e:f8:08
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN = kubernetes
        Validity
            Not Before: May  4 11:48:00 2018 GMT
            Not After : May  1 11:48:00 2028 GMT
        Subject: C = USA, ST = California, L = San Francisco, O = etcd, OU = etcd Security, CN = kbm2-kube-management.company.net
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:c2:d3:e7:ae:7a:4b:f9:4b:f4:17:ac:ad:b8:47:
                    e4:ee:6b:37:a1:a9:88:74:25:d5:b4:5c:27:b8:9a:
                    97:09:2d:93:d3:2f:3d:01:8a:a9:08:9f:cb:8c:b2:
                    6d:1e:f1:89:24:f4:0b:a5:05:e2:86:38:a4:5c:65:
                    0d:5d:ea:b2:b1:d4:51:3d:f4:87:7c:ad:2d:01:2f:
                    74:65:73:61:b4:0f:6b:cc:5b:e0:68:36:5a:c9:9a:
                    3b:92:32:a0:0c:a7:0b:f8:e9:1e:c9:4e:48:3a:a5:
                    a6:07:2f:6b:78:1f:5a:2f:ab:51:84:51:65:fa:1a:
                    0f:cb:7a:ee:7f:70:48:f7:98:e6:c2:37:55:d5:a8:
                    4d:00:f7:c9:0e:fb:95:e9:4d:8d:83:b5:43:be:6e:
                    c0:57:e5:02:9a:a6:bc:3e:88:e8:a1:5d:e0:40:bb:
                    6e:cb:26:b5:20:e6:2a:7c:6c:30:e8:5b:5f:c8:0c:
                    60:40:dd:73:59:5b:25:c7:56:8d:b8:e9:01:42:46:
                    0a:7b:7c:8a:52:89:d2:33:87:70:66:97:d5:19:4a:
                    5a:9c:eb:9f:c8:b4:15:24:4a:7e:b2:ac:c2:b9:d4:
                    f9:cf:e1:30:0d:6c:dc:e9:4a:5d:da:96:bf:34:d2:
                    f6:d6:dd:99:64:c9:eb:3a:d9:cf:b0:38:5f:c1:0b:
                    28:5d
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage: 
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Subject Key Identifier: 
                4C:3B:E4:A5:33:C4:5B:71:33:B6:60:90:C4:E5:3C:4F:10:12:D9:E7
            X509v3 Subject Alternative Name: 
                DNS:kbm2-kube-management.company.net, DNS:localhost, IP Address:172.31.52.146, IP Address:127.0.0.1
    Signature Algorithm: sha256WithRSAEncryption
         aa:55:c5:98:59:86:f9:6c:eb:ca:73:ff:9f:c3:29:f5:89:bc:
         b2:b6:1d:1d:2e:73:5f:a0:b7:bd:dd:59:f2:4a:9a:03:a9:b5:
         07:30:e4:85:06:49:1b:6b:21:96:b0:16:c5:e7:bd:92:dc:b3:
         78:db:20:4a:90:f1:52:32:36:54:c7:b9:cc:40:cc:f8:f4:53:
         ed:0c:ec:aa:68:c1:ce:95:d8:d7:1e:9a:34:cd:5b:77:87:f2:
         19:1a:d4:3d:9b:79:8e:fd:9b:af:09:ee:cb:04:64:40:8e:dd:
         c6:ad:33:88:a8:9d:40:1a:f4:9b:00:cb:38:35:27:5b:05:fe:
         9a:db:b4:fc:67:c8:c2:ca:bb:35:29:a5:86:5f:a0:3e:26:ae:
         ff:41:65:d4:87:b0:fb:03:46:c0:98:7b:06:36:80:bb:5d:5d:
         70:12:00:7a:92:33:aa:e3:1c:02:79:c9:0c:fc:0a:4b:b9:38:
         7c:bc:d1:36:b9:b7:3a:94:6d:76:6a:09:fd:4c:33:32:a8:0d:
         7d:af:92:54:63:4f:6a:66:2c:4e:9b:2b:cf:c4:92:bb:ba:63:
         ab:e4:83:1a:ac:04:1e:7e:f3:e1:2d:c8:f3:d4:0b:31:5c:b3:
         91:39:fb:c4:fa:e5:1e:49:8b:6c:61:7f:9f:3a:cc:05:48:55:
         ce:a4:7e:8b

[cristifalcas]

Strange. I managed to create the user from inside a container with version 3.0.17:

SSL_OPTS="--cacert=/keys/ca.crt --cert=/keyslocal/root.pem --key=/keyslocal/root-key.pem --endpoints=127.0.0.1:4001"
docker run -it --rm --net=host --privileged -v $PWD:/keyslocal -v /var/lib/origin/openshift.local.config/master:/keys --name temp gcr.io/google_containers/etcd-amd64:3.0.17 sh -c "ETCDCTL_API=3 etcdctl $SSL_OPTS user add system:master"
# Password of system:master: 
# Type password of system:master again for confirmation: 

[hexfusion]

Can you help me with a grpc gateway example? How to add the user?

And I need it to work with tls certs.

@cristifalcas sure here is a bash example of auth and useradd using gRPC gateway.

https://github.com/coreos/etcd/blob/master/contrib/recipes/grpc_gateway/user_add.sh

usage : $ user_add.sh system:master

[cristifalcas]

Thank you

[hexfusion]

Just note for 3.0 you will need to use /v3alpha as with your above example. But you probably notice that by now :)

[hexfusion]

@cristifalcas is this solution going to work for you or do you want to explore further?

[cristifalcas]

It seems it worked, at least on 3.0.17. For me it will be better to do it with any version of etcdctl.

But I'm ok with the grpc solution also.

[mitake]

Sorry for my late reply. Probably adding a new flag --password would be nice. Current behaviour of separating the arg with : is just inherited from ctlv2.