Kops: Invalid metadata in OpenStack instance

[zetaab]

I am trying 1.18.0 alpha 3 release and creating instances:

Bad request with: [POST https://exxx.13774/v2.1/servers], error message: {"badRequest": {"message": "Invalid input for field/attribute metadata. Value: {u'kopsGroupName': u'master-zone-3-1', u'k8s': u'clusterospr-0502de.k8s.local', u'KopsRole': u'Master', u'k8s.io/cluster-autoscaler/node-template/label/kops.k8s.io/instancegroup': u'master-zone-3-1', u'cluster_generation': u'0', u'k8s.io/role/master': u'1', u'ig_generation': u'0', u'kops.k8s.io/instancegroup': u'master-zone-3-1', u'KubernetesCluster': u'clusterospr-0502de.k8s.local', u'KopsInstanceGroup': u'master-zone-3-1', u'KopsNetwork': u'clusterospr-0502de.k8s.local'}. u'k8s.io/cluster-autoscaler/node-template/label/kops.k8s.io/instancegroup', u'k8s.io/role/master', u'kops.k8s.io/instancegroup' do not match any of the regexes: '^[a-zA-Z0-9-_:. ]{1,255}$'", "code": 400}}

cc @mitch000001 that PR that you made broke this..

and this is now backported to all releases

problem PRs
https://github.com/kubernetes/kops/pull/8999
https://github.com/kubernetes/kops/pull/9001
https://github.com/kubernetes/kops/pull/9013

It is impossible to use any latest kops version with openstack

versions affected:
v1.18.0-alpha.3
v1.17.0-beta.2
v1.16.2

/kind bug
/priority critical-urgent

[zetaab]

/kind bug
/priority critical-urgent

[hakman]

Really sad to see this...

[zetaab]

I am thinking could we have e2e OpenStack tests also, but we need someone to deliver access

[hakman]

Someone could also manually run such tests before releases until such cluster would exist.
https://kops.sigs.k8s.io/development/testing/#kubernetes-e2e-testing

[mitch000001]

I am sorry. Unfortunately it sneaked through. And it is in release 1.17 and 1.16. ~So, we should create PRs for those releases as well~. But @zetaab patched it. Thank you.

[mitch000001]

@hakman @zetaab What do we need in order to add an Openstack env to the e2e tests of kops?

[rifelpet]

I think its worth following what cluster-api-provider-openstack did here: https://github.com/kubernetes-sigs/cluster-api-provider-openstack/issues/484

They use a project called OpenLab to run their tests: https://github.com/theopenlab/openlab/issues/141

And it looks like those tests are in the k8s testgrid: https://testgrid.k8s.io/sig-cluster-lifecycle-cluster-api-provider-openstack#capo-conformance-stable-k8s-master

If we could setup something similar that would be great. I'm not familiar enough with the OpenStack side of things to take the next steps but I can provide assistance in getting the jobs themselves setup.

[zetaab]

we do actually have https://github.com/theopenlab/openlab/issues/181 openstack project in some openstack installation. However, the problem is that the openstack version is pretty old in that one. We cannot really test loadbalancers etc there

[rifelpet]

Hm do we have any tests setup there at all? I wonder if "something is better than nothing" at this point. If we can get any sort of periodic testing on there and connected to our test grid, it could at least serve as some basic smoke testing until they eventually upgrade their openstack version and add functionality. Would a test there have caught an issue like this?

[zetaab]

we do not have anything currently there. I am not sure how much the quota can handle there, maybe not much. So I do not know can that really be used for PR tests

[rifelpet]

yea, perhaps not PR tests but a single job that runs periodically? we could make sure even the most basic smoke test is green before releasing new kops versions. It'd also help us trace issues back to specific PRs.

[johanssone]

Just a comment here for people that are running into this, but wants to try out 1.18.0-alpha for Openstack deployments until there is a new release with fix, the below steps worked for me (golang required ofc):

mkdir ~/kops
cd ~/kops
export GOPATH=`pwd`

go get -d k8s.io/kops
cd ${GOPATH}/src/k8s.io/kops/
git checkout master # (commit b80edfdd2526164de9b0cd3e6cffac8fa8c7cca1 as of writing)
make

kops binary can then be found under ${GOPATH}/src/k8s.io/kops/.build/local/kops

cd ${GOPATH}/src/k8s.io/kops/.build/local
chmod +x kops
./kops version
Version 1.18.0-alpha.3 (git-b80edfdd2)

That binary works out OK for deploying clusters on OpenStack it seems. Just keep in mind that it's a build directly from master development branch.

[hakman]

Even easier, the build including just this fix from the CI tests:
https://storage.googleapis.com/kops-ci/bin/1.18.0-alpha.4+3df14f2be/linux/amd64/kops

[johanssone]

Even easier, the build including just this from the CI tests:
https://storage.googleapis.com/kops-ci/bin/1.18.0-alpha.4+3df14f2be/linux/amd64/kops

Or that :D

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale