Kops: Kubernetes 1.9 still with k8s-1.8 AMI

[rekcah78]

Thanks for submitting an issue! Please fill in as much of the template below as
you can.

------------- BUG REPORT TEMPLATE --------------------

1. What kops version are you running? The command kops version, will display
   this information.

kops 1.9.0
k8s 1.9.3

1. What Kubernetes version are you running? kubectl version will print the
   version if a cluster is running or provide the Kubernetes version specified as
   a kops flag.

Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.6", GitCommit:"9f8ebd171479bec0ada837d7ee641dec2f8c6dd1", GitTreeState:"clean", BuildDate:"2018-03-21T15:21:50Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.3", GitCommit:"d2835416544f298c919e2ead3be3d0864b52323b", GitTreeState:"clean", BuildDate:"2018-02-07T11:55:20Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}

1. What cloud provider are you using?

AWS

1. What commands did you run? What is the simplest way to reproduce this issue?

kops get ig nodes -o yaml

1. What happened after the commands executed?

The result for spec.image is a AMI k8s-1.8

spec:
image: kope.io/k8s-1.8-debian-jessie-amd64-hvm-ebs-2018-02-08

1. What did you expect to happen?

I'm expect find an AMI for k8s 1.9 like this AMI:
k8s-1.9-debian-stretch-amd64-hvm-ebs-2018-04-11

https://github.com/kubernetes/kops/blob/master/channels/stable seem to not have been updated before release of kops 1.9.0

1. Please provide your cluster manifest. Execute
   kops get --name my.example.com -oyaml to display your cluster manifest.
   You may want to remove your cluster name and other sensitive information.

apiVersion: kops/v1alpha2
kind: Cluster
metadata:
creationTimestamp: 2018-04-13T09:51:10Z
name: k8s-init.lab-aws.domdom.com
spec:
api:
loadBalancer:
type: Public
authorization:
rbac: {}
channel: stable
cloudLabels:
Environment: lab
cloudProvider: aws
configBase: s3://cgasmi-kops/k8s-init.lab-aws.domdom.com
etcdClusters:

• enableEtcdTLS: true
  etcdMembers:
  
  
  
  • instanceGroup: master-eu-central-1a
    
    name: a
  
  
  • instanceGroup: master-eu-central-1b
    
    name: b
  
  
  • instanceGroup: master-eu-central-1c
    
    name: c
    
    name: main
    
    version: 3.0.17
  
  
• enableEtcdTLS: true
  etcdMembers:
  
  
  
  • instanceGroup: master-eu-central-1a
    
    name: a
  
  
  • instanceGroup: master-eu-central-1b
    
    name: b
  
  
  • instanceGroup: master-eu-central-1c
    
    name: c
    
    name: events
    
    version: 3.0.17
    
    iam:
    
    allowContainerRegistry: true
    
    legacy: false
    
    kubeAPIServer:
    
    runtimeConfig:
    
    autoscaling/v2beta1: "true"
    
    kubeControllerManager:
    
    horizontalPodAutoscalerUseRestClients: true
    
    kubelet:
    
    enableCustomMetrics: true
    
    kubernetesApiAccess:
  
  
• 10.99.0.0/22
• 0.0.0.0/0
  kubernetesVersion: 1.9.3
  masterPublicName: api.k8s-init.lab-aws.domdom.com
  networkCIDR: 10.99.0.0/22
  networking:
  weave:
  mtu: 8912
  nonMasqueradeCIDR: 100.64.0.0/10
  sshAccess:
• 0.0.0.0/0
  subnets:
• cidr: 10.99.0.128/25
  name: eu-central-1a
  type: Private
  zone: eu-central-1a
• cidr: 10.99.1.0/25
  name: eu-central-1b
  type: Private
  zone: eu-central-1b
• cidr: 10.99.1.128/25
  name: eu-central-1c
  type: Private
  zone: eu-central-1c
• cidr: 10.99.0.0/28
  name: utility-eu-central-1a
  type: Utility
  zone: eu-central-1a
• cidr: 10.99.0.16/28
  name: utility-eu-central-1b
  type: Utility
  zone: eu-central-1b
• cidr: 10.99.0.32/28
  name: utility-eu-central-1c
  type: Utility
  zone: eu-central-1c
  topology:
  dns:
  type: Public
  masters: private
  nodes: private

---

apiVersion: kops/v1alpha2
kind: InstanceGroup
metadata:
creationTimestamp: 2018-04-13T09:51:12Z
labels:
kops.k8s.io/cluster: k8s-init.lab-aws.domdom.com
name: master-eu-central-1a
spec:
image: kope.io/k8s-1.8-debian-jessie-amd64-hvm-ebs-2018-02-08
machineType: m4.large
maxSize: 1
minSize: 1
nodeLabels:
kops.k8s.io/instancegroup: master-eu-central-1a
role: Master
rootVolumeSize: 30
subnets:

• eu-central-1a

---

apiVersion: kops/v1alpha2
kind: InstanceGroup
metadata:
creationTimestamp: 2018-04-13T09:51:12Z
labels:
kops.k8s.io/cluster: k8s-init.lab-aws.domdom.com
name: master-eu-central-1b
spec:
image: kope.io/k8s-1.8-debian-jessie-amd64-hvm-ebs-2018-02-08
machineType: m4.large
maxSize: 1
minSize: 1
nodeLabels:
kops.k8s.io/instancegroup: master-eu-central-1b
role: Master
rootVolumeSize: 30
subnets:

• eu-central-1b

---

apiVersion: kops/v1alpha2
kind: InstanceGroup
metadata:
creationTimestamp: 2018-04-13T09:51:12Z
labels:
kops.k8s.io/cluster: k8s-init.lab-aws.domdom.com
name: master-eu-central-1c
spec:
image: kope.io/k8s-1.8-debian-jessie-amd64-hvm-ebs-2018-02-08
machineType: m4.large
maxSize: 1
minSize: 1
nodeLabels:
kops.k8s.io/instancegroup: master-eu-central-1c
role: Master
rootVolumeSize: 30
subnets:

• eu-central-1c

---

apiVersion: kops/v1alpha2
kind: InstanceGroup
metadata:
creationTimestamp: 2018-04-13T09:51:13Z
labels:
kops.k8s.io/cluster: k8s-init.lab-aws.domdom.com
name: nodes
spec:
image: kope.io/k8s-1.8-debian-jessie-amd64-hvm-ebs-2018-02-08
machineType: m4.large
maxSize: 3
minSize: 3
nodeLabels:
kops.k8s.io/instancegroup: nodes
role: Node
rootVolumeSize: 50
subnets:

• eu-central-1a
• eu-central-1b
• eu-central-1c

[azman0101]

As stated in https://github.com/kubernetes/kops/blob/master/docs/images.md,

kope.io => 383156758163

The image k8s-1.9-debian-stretch-amd64-hvm-ebs-2018-04-11 you mentioned is owned by 487596255802.

Is this an official account ?

Regards

[thomasjungblut]

The base image doesn't matter so much for the actual installation, all the relevant k8s pieces are installed through nodeup and they come up just fine and report 1.9.3. But I agree, it's confusing for people that don't know that implementation detail.

[mattparkes]

This image doesn't support M5s (nvme) though, so it'd be great if it got updated.

[cg-rsands]

Would be really helpful for kops to look at the instance type if an AMI is not compatible or at least document clearly on the images read me. Kops upgrade brought down our cluster due to this issue

[viniciusnz]

It looks like kope.io/k8s-1.9-debian-stretch-amd64-hvm-ebs-2018-05-27 does work for m5 / c5 now, just fyi...

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[philtweir]

Just to add another data point to @cg-rsands comment - we got caught out by kubernetes/kubernetes#56850 which requires a change to the systemd kubelet definitions when upgrading to k8s 1.10

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.