Kops: Cert errors creating cluster

[chrislovecnm]

Thanks for submitting an issue! Please fill in as much of the template below as
you can.

------------- BUG REPORT TEMPLATE --------------------

1. What kops version are you running? The command kops version, will display
   this information.

on a feature branch

1. What Kubernetes version are you running? kubectl version will print the
   version if a cluster is running or provide the Kubernetes version specified as
   a kops flag.

1.8.x

1. What cloud provider are you using?
   aws

2. What commands did you run? What is the simplest way to reproduce this issue?

KOPS=kops

$KOPS create -f $KOPS_FILE
$KOPS create secret --name $CLUSTER_NAME sshpublickey admin -i ~/.ssh/id_rsa.pub
$KOPS update cluster --phase assets $CLUSTER_NAME --yes
$KOPS update cluster --phase network $CLUSTER_NAME --yes
$KOPS update cluster --phase security $CLUSTER_NAME --yes
$KOPS update cluster --phase cluster $CLUSTER_NAME --yes

1. What happened after the commands executed?

instances are created, but I am getting cert errors.

1. What did you expect to happen?

Cluster should be created

1. Please provide your cluster manifest. Execute

2. Please run the commands with most verbose logging by adding the -v 10 flag.
   Paste the logs into this report, or in a gist and provide the gist link here.

Dec 27 21:58:38 ip-172-60-0-196 nodeup[726]: I1227 21:58:38.478038     726 s3fs.go:213] Listing objects in S3 bucket "aws.k8spro.com" with prefix "us-west-2.aws.k8spro.com/pki/issued/ca/"
Dec 27 21:58:38 ip-172-60-0-196 nodeup[726]: I1227 21:58:38.574952     726 s3fs.go:239] Listed files in s3://aws.k8spro.com/us-west-2.aws.k8spro.com/pki/issued/ca: [s3://aws.k8spro.com/us-west-2.aws.k8spro.com/pki/issued/ca/6504343549136392374237039546.crt s3://aws.k8spro.com/us-west-2.aws.k8spro.com/pki/issued/ca/keyset.yaml]
Dec 27 21:58:38 ip-172-60-0-196 nodeup[726]: I1227 21:58:38.574981     726 s3fs.go:176] Reading file "s3://aws.k8spro.com/us-west-2.aws.k8spro.com/pki/issued/ca/6504343549136392374237039546.crt"
Dec 27 21:58:38 ip-172-60-0-196 nodeup[726]: I1227 21:58:38.663684     726 certificate.go:103] Parsing pem block: "CERTIFICATE"
Dec 27 21:58:38 ip-172-60-0-196 nodeup[726]: I1227 21:58:38.663901     726 s3fs.go:176] Reading file "s3://aws.k8spro.com/us-west-2.aws.k8spro.com/pki/issued/ca/keyset.yaml"
Dec 27 21:58:38 ip-172-60-0-196 nodeup[726]: W1227 21:58:38.751769     726 main.go:141] got error running nodeup (will retry in 30s): error building loader: error fetching CA certificate from keystore: error in 'FindCert' attempting to load cert "ca": error loading certificate "s3://aws.k8spro.com/us-west-2.aws.k8spro.com/pki/issued/ca/keyset.yaml": could not parse certificate

1. Anything else do we need to know?

I am trying to re-create.

[chrislovecnm]

I think we have broken using a kops master build, w/o current nodeup. I am testing to verify.

@justinsb can we cut an alpha version of 1.8.1 or another unreleased beta if so?

[chrislovecnm]

I just did

1. git co master
2. updated master
3. S3_BUCKET=s3://$BUCKET make kops upload
4. set the NODEUP_URL
5. kops create cluster --name $CLUSTER_NAME --zones us-west-2c --api-loadbalancer-type public --yes

And I am getting the cert errors :(

$ kops version
Version 1.8.0 (git-b04ec421c)

[chrislovecnm]

So on a whim I used go 1.9.2 to do the make upload, by changing the makefile. I also d/l'ed the version that was uploaded to s3. Doing all that has it working.

[KashifSaadat]

Got the same issue. Not sure why it's looking for keyset.yaml in S3, I haven't seen that before.

Edit: ^ Looks like that was introduced in the recent keypairs PRs, I'll do more debugging.

[chrislovecnm]

@KashifSaadat I got past this by using the same golang version for kops and nodeup.

[davidblum]

+1
Built from master branch today to get the new subnet functionality (hawtness), using go 1.9.2.

Running into the same issue with the certificate length when bringing a new cluster online:

got error running nodeup (will retry in 30s): error building loader: error fetching CA certificate from keystore: error in 'FindCert' attempting to load cert "ca": error loading certificate "s3://mahbucket/mahdomain/pki/issued/ca/keyset.yaml": could not parse certificate

[chrislovecnm]

You need to do a make upload and use the custom nodeup.

[chrislovecnm]

I have a PR in to fix this, but

1. Change makefile to use 1.9.2 or use golang 1.8.x locally
2. If you want to use 1.9 you need to upload nodeup and set the nodeup env variable

[chrislovecnm]

/assign @justinsb

He is going to add some unit tests to figure out what is going on

[chrislovecnm]

Well Justin put in https://github.com/kubernetes/kops/pull/4172 which shows the cert works work the supported versions of golang.

Any ideas folks?

[davidblum]

With the passing tests, it just looks like version conflict on go itself.

Once 1.9 is pushed up and used in the AMI's we should be fine?

I'll try a 1.8 version locally and see if the issue persists.

Thanks @chrislovecnm

[davidblum]

@chrislovecnm Still having the issues when building kops with golang 1.8 locally.

Same error as before with issues parsing the cert.

 > ./kops version
Version 1.8.0 (git-13baa65)

Jan  3 18:49:40 ip-x nodeup[694]: I0103 18:49:40.536288     694 certificate.go:103] Parsing pem block: "CERTIFICATE"
Jan  3 18:49:40 ip-x nodeup[694]: I0103 18:49:40.536378     694 s3fs.go:176] Reading file "s3://kube-state-store/kube.domain.com/pki/issued/ca/keyset.yaml"
Jan  3 18:49:40 ip-x nodeup[694]: W0103 18:49:40.559216     694 main.go:141] got error running nodeup (will retry in 30s): error building loader: error fetching CA certificate from keystore: error in 'FindCert' attempting to load cert "ca": error loading certificate "s3://kube-state-store/kube.domain.com/pki/issued/ca/keyset.yaml": could not parse certificate

# uname -a
Linux ip-x 4.4.102-k8s #1 SMP Sun Nov 26 23:32:43 UTC 2017 x86_64 GNU/Linux

Edit: Working. Had to grab the version of kops make upload dropped in the s3 bucket.

[justinsb]

I think this happens when running nodeup that doesn't match the kops version. I added keyset.yaml to the directory, and it isn't a PEM file, but nodeup 1.8 will still try to parse it as such.

The workaround (which is present in master) is this code:

        id := f.Base()
        if strings.HasSuffix(id, ".yaml") {
            // ignore bundle
            continue
        }

I suspect the problem is not go 1.8 vs 1.9, but rather just the act of building the code.

How are people building though? It sounds like we might have a problem with the makefile not building everything?

In any case, the easy workaround is that we publish a 1.9.0-alpha.1, so that the default nodeup handles the latest kops keyset.yaml files.

[davidblum]

@justinsb I agree with you on the build aspect (vs the go version).

Initially I was running the make independent of the make upload suggested by Chris. What's interesting is that those two should produce the same version of kops for OSX. However they seem to have a different git hash.

As I mentioned earlier, I was able to successfully build, and deploy from the master branch when I ran make upload and grabbed the OSX (darwin) version of kops from my s3 bucket.

Thanks for chiming in and confirming. Really appreciate your guys responsiveness and commitment to the code base.

[StephanX]

I can verify that on OSX, two different versions of kops are built when make kops upload is executed, apparently one version by the local go, and a different version by the containerized build. Both versions (when built against 1.9.2, and an updated Makefile) produce the original error:

error fetching CA certificate from keystore: error in 'FindCert'

As @chrislovecnm suggested, pushing the new version of nodeup to S3 works as expected.

Relevant details below, I hope they're helpful:

$ kops> uname -a
Darwin x.local 17.3.0 Darwin Kernel Version 17.3.0: Thu Nov  9 18:09:22 PST 2017; root:xnu-4570.31.3~1/RELEASE_X86_64 x86_64
$ kops> go version
go version go1.9.2 darwin/amd64
$ kops> grep "GOVERSION=" Makefile
GOVERSION=1.9.2
$ kops> git log | head -n 1
commit 21a372f6b2299933fe606402b86282aeba9f1edc
$ kops> S3_BUCKET=s3://my-kop-store make kops upload
<successful build outcomes snipped>
$ kops> find . -name kops -type file | grep -v linux | xargs shasum
fed2c72a15a231221562e79aad1e8a5850fb6e04  ./.build/dist/darwin/amd64/kops
fe00ed910c80f933e4d568535b796fb246df04cb  ./.build/local/kops
fed2c72a15a231221562e79aad1e8a5850fb6e04  ./.build/upload/kops/1.8.0/darwin/amd64/kops

[chrislovecnm]

So this seems to be an OSX go bug ... YAYAYAYAYA

[KashifSaadat]

@justinsb I was building via make version-dist and uploading all assets (including nodeup) to S3, however was still encountering this issue (I'm on OSX). Just re-testing now fresh off master (https://github.com/kubernetes/kops/commit/b8a701b2da72a4afde7cb0c69313cd2674c19022), will update shortly. :)

Edit: Same issue unfortunately (kops, nodeup, protokube, utils all built and pushed up and I've validated my built nodeup was pulled down on the hosts):

s3fs.go:187] Reading file "s3://${KOPS_STATE_BUCKET}/${KOPS_CLUSTER_NAME}/pki/issued/ca/keyset.yaml"
main.go:141] got error running nodeup (will retry in 30s): error building loader: CA certificate "ca" not found

So... keyset.yaml is created when building a new cluster, but not on updating an existing cluster. :( When I create a fresh cluster, the secret is created in S3 (along with all the other pki files) and everything works fine. I've temporarily set useBundle to false (vfs_castore.go) in my local kops build to get past this when upgrading an existing cluster.

[KashifSaadat]

Any pointers on how we can get kops to generate the keyset.yaml files in S3 (if it's detected as missing) during a kops update cluster.. command? I was trying to follow through the logic and got a little lost!

Doing so should cover the user journey on upgrading an existing cluster.

[nebril]

@chrislovecnm I don't think this is OSX go bug, I got hit by this on Linux with kops built from master with both go 1.9.2 and go 1.8.5.

[nebril]

@KashifSaadat can you specify what do you mean by pushing kops, nodeup, protokube, utils? Where are you pushing them for nodes to use them? I am asking, because I want to try it myself.

Also, I think OP bug is a bit different than your case, in OP there is a problem with parsing keyset.yaml, you seem to don't have it on s3.

[KashifSaadat]

Hey @nebril, just into S3 as normal (e.g. make upload) and then using them by setting PROTOKUBE_IMAGE, NODEUP_URL etc.

Yep you're right, looks like I've mixed up a separate issue here!

[nebril]

So building all binaries and pushing them to my s3 bucket as described in https://github.com/kubernetes/kops/blob/master/docs/development/adding_a_feature.md#testing works for me (I did it with 1.8 tag). I wonder if binaries in "default" s3 bucket are affected by this?

[StephanX]

Upstream master moved to go 1.9.2. I was able to build nodeup today and have it work just fine with kubernetes 1.9 (since I've been asked to use the new m5/c5 ec2 instance types.) . The standard nodeup from https://kubeupv2.s3.amazonaws.com/kops still fails for me, though.

[mmacfadden]

@StephanX Any tips on how to do that?

[francoran]

Hi we're having the same issue- all instances recognized as OutOfService in ELB. Im not sure if that's related to keyset.yaml as I don't see it's missing during kops update.
Issue started 2 days ago after rolling-update with latest stable kops version and k8s 1.8.5.
We tried to downgrade to 1.7.12 but issue still persist.

is there any workaround we can use?

[mmacfadden]

@francoran In terms of the keyset.yaml, I don't think you would see any error on the machine you are running KOPS from. The error happens when the nodes are bootstrapping after kops is done. I believe you would have to ssh in to a master node (for example) and take a look at the syslog to see the issue. At least that is my experience.

[StephanX]

@mmacfadden shortest answer I can give (for me on OSX):
Requirements:

• golang 1.9.2 from homebrew
• docker
• write access to an S3 bucket (export AWS_ACCESS_KEY= AWS_SECRET_KEY=)

go get k8s.io/kops
cd go/src/k8s.io/kops
git checkout master # (at least, that's what I'm building from)
make upload S3_BUCKET=s3://<your AWS S3 bucket> # (does not need to be a public bucket)
make upload S3_BUCKET=s3://<your AWS S3 bucket>
cat .build/upload/kops/1.8.0/linux/amd64/nodeup.sha1 # this sha will be needed later

Build time takes me 45 minutes (though, I comment out windows build targets in the Makefile to speed things along a bit.) Once the binaries are built, make will push the artifacts to your S3 bucket. During kops create you will need to then export the new S3 location and sha for nodeup:

export NODEUP_URL="https://s3.amazonaws.com/<your AWS S3 bucket>/kops/1.8.0/linux/amd64/nodeup"  NODEUP_HASH="<NODEUP.SHA1 value>"
kops create cluster --name=newcluster

@francoran mmacfadden is correct, to identify that it's a keyset.yaml bootstrap issue, you have to ssh to the node that isn't joining, and look at the /var/log/daemon.log for the events that KashifSaadat reported.

[mmacfadden]

@justinsb you mentioned publishing a 1.9.0-alpha.1 release to fix this. Is that a possibility?

[kaijparo]

I'm running into this too:

$ kops version
Version 1.8.0 (git-5e68a5a33)

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.4", GitCommit:"9befc2b8928a9426501d3bf62f72849d5cbcd5a3", GitTreeState:"clean", BuildDate:"2017-11-20T19:12:24Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"darwin/amd64"}

I had an existing cluster at version 1.8.7 and I was trying to add a new user token via this post:
https://github.com/kubernetes/kops/issues/2354

And one of the steps involved a rolling update.... boom :(

How do I save this cluster?

It's noted earlier that "keyset.yaml is created when building a new cluster, but not on updating an existing cluster".

Is there no way to create this keyset and update the existing cluster?

[chrislovecnm]

@kaijparo can I get detailed instructions of the steps you have taken? Also, can you please use “make upload” and use a custom version of nodeup?

[kaijparo]

@chrislovecnm I tried the steps @StephanX and it looks like i can get to the keyset.yaml file now... but now i'm getting:

"Jan 29 11:12:27 ip-10-203-27-231 nodeup[692]: W0129 11:12:27.654983 692 main.go:141] got error running nodeup (will retry in 30s): error building loader: CA certificate "ca" not found"

Although my versions have shifted some:

kops version
Version 1.8.0 (git-b48ed4055)

kubectl version
Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.2", GitCommit:"5fa2db2bd46ac79e5e00a4e6ed24191080aa463b", GitTreeState:"clean", BuildDate:"2018-01-18T21:11:08Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"darwin/amd64"}
Unable to connect to the server: EOF

[kaijparo]

@KashifSaadat did you ever resolve this?

[mmacfadden]

@kaijparo I think you are now at the bug the rest of us are seeing. It seems like this won't be resolved until we get an alpha release of kops 1.9.0...

[StephanX]

@kaijparo I agree with @mmacfadden , I couldn't effectively do an upgrade where the old nodes were built on 1.8.7 and new nodes on 1.9.2. The alternative (for me) was to ssh to the nodes, and change the download of the new nodeup to read from the old assets; basically, not upgrade at all. Building brand new clusters, using kubernetes version 1.9 and building nodeup worked for me, as of kops sha ec4a916f621 (Jan 22 HEAD), though I recognize this is a less-than-ideal solution.

[kaijparo]

@StephanX

how did you accomplish this?

The alternative (for me) was to ssh to the nodes, and change the download of the new nodeup to read from the old assets; basically, not upgrade at all.

Feeling pretty green here

[StephanX]

@kaijparo the answer gets a bit more in the weeds than I'd normally like to post in a bug report. Assuming you're on AWS, the most direct suggestion I can think of would be to look at your earlier AWS autoscaling launch configurations. They'll have a name similar to nodes.demo1.k8s.-20171124195225 . The 201711 in this case shows the configuration created on Nov 24, 2017. Click a configuration that you know worked, and click the user data link. Near the top, you'll see a line like:

NODEUP_URL=https://kubeupv2.s3.amazonaws.com/kops/1.8.0-beta.1/linux/amd64/nodeup NODEUP_HASH=

Take that NODUP_URL and hash (which may be blank), and export them:

export NODEUP_URL="https://kubeupv2.s3.amazonaws.com/kops/1.8.0-beta.1/linux/amd64/nodeup" NODEUP_HASH=""

Now you can update your cluster using the old nodeup (in the same terminal window you exported the NODUP_URL):

kops update cluster <your clustername> --yes && kops rolling-upgrade <yourclustername> --yes

I can't promise this will work, am just throwing out my best effort. A few others on this thread may have better advice. Best of luck!

[kaijparo]

@StephanX

Thanks for the help!

After doing those steps, i found myself back at:

Jan 29 23:26:54 ip-10-203-25-183 nodeup[706]: I0129 23:26:54.869405 706 s3fs.go:176] Reading file "s3:///pki/issued/master/keyset.yaml"
Jan 29 23:26:54 ip-10-203-25-183 nodeup[706]: W0129 23:26:54.879330 706 main.go:141] got error running nodeup (will retry in 30s): error building loader: error in 'FindCert' attempting to load cert "master": error loading certificate "s3:///pki/issued/master/keyset.yaml": could not parse certificate.

not sure what i'm missing... wonder if i have to revert my kops/kubectl versions back?

[StephanX]

"s3:///pki/issued/master/keyset.yaml"

Unless you deleted it for posting purposes, it appears your kops S3 store isn't being referenced. For example, my kops store is s3://<MY_KOPS_STORE_BUCKET>/<MY_CLUSTER_NAME>/pki/issued/ca/keyset.yaml

It might be helpful to know that the cloud-init script at /var/lib/cloud/instance/user-data.txt determines which nodeup you get, and writes config files at /var/cache/kubernetes-install/*.yaml. If the nodeup is correct, you can modify the yaml files to address the correct S3 buckets, and that may get things righted for you.

[chrislovecnm]

i am guessing you need a newer version of nodeup.

make upload with the s3 bucket name will help with that.

Let me know how it goes. I am concerned about upgrades with this as well.

[kaijparo]

@StephanX i did remove the bucket name for the post actually..
@chrislovecnm this right?

make upload S3_BUCKET=s3://<your AWS S3 bucket>/<Clustername>

[KashifSaadat]

Hi @kaijparo, The error about the missing keyset.yaml is different to what this issue was originally tracking. I've raised #4351 to track the missing keyset.yaml on cluster upgrades.

A temporary workaround (to get your cluster upgrade working) would be to modify the following lines, setting useBundle to false:

• https://github.com/kubernetes/kops/blob/2a5e136/upup/pkg/fi/vfs_castore.go#L360
• https://github.com/kubernetes/kops/blob/2a5e136/upup/pkg/fi/vfs_castore.go#L806

[kaijparo]

Just following up - I eventually had to rebuild the cluster with the new make upload binary and the new nodeup with hash.

Thanks all for your help!

[mmacfadden]

I also got a cluster up and running by uploading a new version of nodeup to S3. However, I think we will hold off on moving forward until this issue is resolve. It was suggested that an 1.9.0 alpha release might solve the problem by referencing the more recent nodeup. Do we still think this is the case? If so, is there any approximate timeline for a 1.9 alpha just for planning purposes? Thanks for the great work!

[isugimpy]

Echoing the questions above, any chance of that change making it to master or a 1.9.0 alpha coming out? I'm really itching to use the new --subnets functionality, and this is a blocker in its current state.

[chrislovecnm]

The 1.8.1 release and master have been fixed to use the new nodeup in 1.8.1. This was part of # 4375, and allows the keyset.yaml to be ignored.

[chrislovecnm]

Closing ...

[KIVagant]

I don't know how but after kops update cluster --name ... --state s3://... --yes now I see the same issue on two new master nodes:

kops version 1.9.0-alpha.1 (git-f799036a3)

Mar 19 18:25:41 ip-10-122-117-206 nodeup[729]: I0319 18:25:41.598204     729 task.go:98] task *nodetasks.File does not implement HasLifecycle
Mar 19 18:25:41 ip-10-122-117-206 nodeup[729]: I0319 18:25:41.598222     729 s3fs.go:198] Reading file "s3://.../.../pki/issued/ca/keyset.yaml"
Mar 19 18:25:41 ip-10-122-117-206 nodeup[729]: W0319 18:25:41.701521     729 main.go:141] got error running nodeup (will retry in 30s): error building loader: CA certificate "ca" not found

[KIVagant]

From the kops update cluster:

I0319 17:26:30.348983    1300 s3fs.go:144] Calling S3 PutObject Bucket="..." Key=".../pki/private/apiserver-aggregator-ca/keyset.yaml" SSE="AES256" ACL=""
I0319 17:26:30.442386    1300 s3fs.go:118] Writing file "s3://.../.../pki/private/apiserver-aggregator-ca/1234579941382152310448758482.key"
I0319 17:26:30.442536    1300 s3fs.go:144] Calling S3 PutObject Bucket="..." Key=".../pki/private/apiserver-aggregator-ca/1234579941382152310448758482.key" SSE="AES256" ACL=""
I0319 17:26:30.565496    1300 s3fs.go:235] Listing objects in S3 bucket "..." with prefix ".../pki/issued/apiserver-aggregator-ca/"
I0319 17:26:30.684268    1300 s3fs.go:261] Listed files in s3://.../.../pki/issued/apiserver-aggregator-ca: []
I0319 17:26:30.684795    1300 s3fs.go:118] Writing file "s3://.../.../pki/issued/apiserver-aggregator-ca/keyset.yaml"
I0319 17:26:30.684899    1300 s3fs.go:144] Calling S3 PutObject Bucket="..." Key=".../pki/issued/apiserver-aggregator-ca/keyset.yaml" SSE="AES256" ACL=""
I0319 17:26:30.813716    1300 s3fs.go:118] Writing file "s3://.../.../pki/issued/apiserver-aggregator-ca/6534708517641585138170839523.crt"
I0319 17:26:30.813878    1300 s3fs.go:144] Calling S3 PutObject Bucket="..." Key=".../pki/issued/apiserver-aggregator-ca/6534708517641585138170839523.crt" SSE="AES256" ACL=""
I0319 17:26:30.915206    1300 s3fs.go:198] Reading file "s3://.../.../pki/issued/apiserver-aggregator-ca/6534708517641585138170839523.crt"
I0319 17:26:31.004421    1300 certificate.go:103] Parsing pem block: "CERTIFICATE"
I0319 17:26:31.004651    1300 keypair.go:215] created certificate &{{[] [] [] [] [] [] []  apiserver-aggregator-ca [{2.5.4.3 apiserver-aggregator-ca}] []} true 0xc4213aca00 0xc421485930}
...
...
...
I0319 17:28:38.772816    1300 certificate.go:103] Parsing pem block: "CERTIFICATE"
I0319 17:28:38.773091    1300 s3fs.go:198] Reading file "s3://.../.../pki/private/kubecfg/keyset.yaml"
I0319 17:28:38.852858    1300 vfs_castore.go:817] no private key bundle "s3://.../.../pki/private/kubecfg/keyset.yaml", falling back to directory-list method
I0319 17:28:38.852995    1300 s3fs.go:235] Listing objects in S3 bucket "..." with prefix ".../pki/private/kubecfg/"
I0319 17:28:38.940809    1300 s3fs.go:261] Listed files in s3://.../.../pki/private/kubecfg: [s3://.../.../pki/private/kubecfg/1234579941382152310448758482.key]
I0319 17:28:38.940933    1300 s3fs.go:198] Reading file "s3://.../.../pki/private/kubecfg/1234579941382152310448758482.key"
I0319 17:28:39.118518    1300 privatekey.go:169] Parsing pem block: "RSA PRIVATE KEY"
I0319 17:28:39.119071    1300 s3fs.go:198] Reading file "s3://.../.../secrets/kube"
I0319 17:28:39.231333    1300 loader.go:357] Config loaded from file /home/.../.kube/config
I0319 17:28:39.232116    1300 loader.go:357] Config loaded from file /home/.../.kube/config
kops has set your kubectl context to _cluster_name_

Cluster changes have been applied to the cloud.


Changes may require instances to restart: kops rolling-update cluster

Make upload failed first time:

➔ make upload S3_BUCKET=s3://kops-nodeup-1.9
mkdir -p /Users/kivagant/go/src/k8s.io/kops/.build/dist
docker pull golang:1.9.3 # Keep golang image up to date
1.9.3: Pulling from library/golang
723254a2c089: Pull complete
abe15a44e12f: Pull complete
409a28e3cc3d: Pull complete
503166935590: Pull complete
abe52c89597f: Pull complete
1ef5b28c171b: Pull complete
e564f1d18399: Pull complete
2e76f3e2e9fa: Pull complete
Digest: sha256:3983f9b1f0b11509e21ab72591f5677cfca0eabbe3a863df01b29e46550ae84b
Status: Downloaded newer image for golang:1.9.3
docker run --name=nodeup-build-1521542833 -e STATIC_BUILD=yes -e VERSION=1.9.0-alpha.1 -v /Users/kivagant/go/src/k8s.io/kops:/go/src/k8s.io/kops golang:1.9.3 make -C /go/src/k8s.io/kops/ nodeup
make: Entering directory '/go/src/k8s.io/kops'
mkdir -p /go/src/k8s.io/kops/.build/local
go build -installsuffix cgo -ldflags="-s -w" -o /go/src/k8s.io/kops/.build/local/go-bindata k8s.io/kops/vendor/github.com/jteeuwen/go-bindata/go-bindata
cd /go/src/k8s.io/kops; /go/src/k8s.io/kops/.build/local/go-bindata -o upup/models/bindata.go -pkg models -ignore="\\.DS_Store" -ignore="bindata\\.go" -ignore="vfs\\.go" -prefix upup/models/ upup/models/...
go build -installsuffix cgo -ldflags="-s -w -X k8s.io/kops.Version=1.9.0-alpha.1 -X k8s.io/kops.GitVersion=4ab61d924" -o /go/src/k8s.io/kops/.build/local/nodeup k8s.io/kops/cmd/nodeup
make: Leaving directory '/go/src/k8s.io/kops'
docker start nodeup-build-1521542833
nodeup-build-1521542833
docker exec nodeup-build-1521542833 chown -R 501:20 /go/src/k8s.io/kops/.build
docker cp nodeup-build-1521542833:/go/src/k8s.io/kops/.build/local/nodeup .build/dist/
(/usr/bin/shasum .build/dist/nodeup | cut -d' ' -f1) > .build/dist/nodeup.sha1
docker pull golang:1.9.3 # Keep golang image up to date
1.9.3: Pulling from library/golang
Digest: sha256:3983f9b1f0b11509e21ab72591f5677cfca0eabbe3a863df01b29e46550ae84b
Status: Image is up to date for golang:1.9.3
docker run --name=kops-build-1521542833 -e STATIC_BUILD=yes -e VERSION=1.9.0-alpha.1 -v /Users/kivagant/go/src/k8s.io/kops:/go/src/k8s.io/kops golang:1.9.3 make -C /go/src/k8s.io/kops/ crossbuild
make: Entering directory '/go/src/k8s.io/kops'
mkdir -p /go/src/k8s.io/kops/.build/dist
GOOS=windows GOARCH=amd64 go build -a -installsuffix cgo -o /go/src/k8s.io/kops/.build/dist/windows/amd64/kops.exe -ldflags="-s -w -X k8s.io/kops.Version=1.9.0-alpha.1 -X k8s.io/kops.GitVersion=4ab61d924" k8s.io/kops/cmd/kops
mkdir -p /go/src/k8s.io/kops/.build/dist
GOOS=darwin GOARCH=amd64 go build -a -installsuffix cgo -o /go/src/k8s.io/kops/.build/dist/darwin/amd64/kops -ldflags="-s -w -X k8s.io/kops.Version=1.9.0-alpha.1 -X k8s.io/kops.GitVersion=4ab61d924" k8s.io/kops/cmd/kops
vendor/github.com/aws/aws-sdk-go/service/ec2/api.go:13:2: readdirent: input/output error
Makefile:237: recipe for target '/go/src/k8s.io/kops/.build/dist/darwin/amd64/kops' failed
make: *** [/go/src/k8s.io/kops/.build/dist/darwin/amd64/kops] Error 1
make: Leaving directory '/go/src/k8s.io/kops'
make: *** [crossbuild-in-docker] Error 2

But second pass is ok. But there was other moment:

aws s3 sync --acl public-read ~/go/src/k8s.io/kops/.build/upload/ s3://kops-nodeup-1.9
fatal error: An error occurred (AuthorizationHeaderMalformed) when calling the ListObjects operation: The authorization header is malformed; a non-empty Access Key (AKID) must be provided in the credential.

And to fix that I have exported AWS_PROFILE and AWS_DEFAULT_PROFILE instead of AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.

But it still doesn't work:

# Here is important to use correct URL for different regions:
export NODEUP_URL="https://s3-us-west-2.amazonaws.com/kops-nodeup-1.9/kops/1.9.0-alpha.1/linux/amd64/nodeup"
export NODEUP_HASH="1ee505714d22d0589c04f4644127d18347d30236"
kops update cluster --name ... --state s3://... -v 10 --yes
...
I0320 11:35:32.526654   11287 s3fs.go:198] Reading file "s3://.../.../pki/issued/ca/keyset.yaml"
I0320 11:35:32.610833   11287 vfs_castore.go:370] no certificate bundle "s3://.../.../pki/issued/ca/keyset.yaml", falling back to directory-list method
...

And the same error is still in /var/log/syslog:

Mar 20 11:39:56 ip-10-122-65-229 nodeup[727]: I0320 11:39:56.402411     727 s3fs.go:198] Reading file "s3://.../.../pki/issued/ca/keyset.yaml"
Mar 20 11:39:56 ip-10-122-65-229 nodeup[727]: W0320 11:39:56.482484     727 main.go:141] got error running nodeup (will retry in 30s): error building loader: CA certificate "ca" not found

[KIVagant]

Finally solved it with:

$ export NODEUP_URL=https://kubeupv2.s3.amazonaws.com/kops/1.8.1/linux/amd64/nodeup
$ export NODEUP_HASH=""
$ kops version
Version 1.9.0-alpha.1 (git-f799036a3)
$ kops update cluster --name ... --state s3://... -v 10 --yes

Recreated master nodes now have other error, but it's a different story:

root@ip-10-122-76-172:/home/admin# tail -f /var/log/kube-controller-manager.log
E0320 11:59:15.462652       5 leaderelection.go:229] error retrieving resource lock kube-system/kube-controller-manager: Get https://127.0.0.1/api/v1/namespaces/kube-system/endpoints/kube-controller-manager: dial tcp 127.0.0.1:443: getsockopt: connection refused