kops fails when using c5 instances on aws

[dharmisha]

I am running kops version 1.7.1 in aws, i used the amazon/amzn-ami-hvm-2017.09.1.20171103-x86_64-gp2 image, which has ena support. However, when I specify the --node-size c5.large, it fails.

c5 instances are launched recently, however is there a way kops can launch c5 instances?

[pieterza]

+1; perhaps kops should not care what size we specify, and let aws tell us if we were wrong? Else it's always catchup

[chrislovecnm]

It is fixed in master and the current beta. Also the Debian base image that kops uses out of the box does not support ena driver. C5 requires that.

So use the beta and you need a different ami

[justinsb]

If we can't get this supported in the default image, this at least has to be a warning.

[adrianchifor]

I agree, please put a warning up, was banging my head against the table all day of why pods were failing randomly on c5.4xlarge nodes.

[coderanger]

Copying the specific error here to aid others in googling:

Launching a new EC2 instance. Status Reason: Enhanced networking with the Elastic Network Adapter (ENA) is required for the 'c5.large' instance type. Ensure that you are using an AMI that is enabled for ENA. Launching EC2 instance failed.

[chrislovecnm]

Here is the issue for ENA driver with the base image https://github.com/kubernetes/kube-deploy/pull/292

[dharmisha]

It is fixed in master and the current beta. Also the Debian base image that kops uses out of the box does not support ena driver. C5 requires that.So use the beta and you need a different ami

I used the beta Version 1.8.0-beta.1 (git-9b71713) binary with amazon/amzn-ami-hvm-2017.09.1.20171103-x86_64-gp2 ami which has support for ena. It still gives an error saying

nodes.spec.machineType: Invalid value: "c5.2xlarge": machine type specified is invalid

Please let me know what needs to be done. This solution is urgently required

[chrislovecnm]

https://github.com/kubernetes/kube-deploy/pull/292

Thanks needs to be tested and completed

[bcorijn]

Having ENA support by default would be great for all instances, not just C5. I've been running the modified image from my PR for several months now without any issues and was hoping it could get merged before the 1.8 stable release.

[chrislovecnm]

@bcorjin the image and the kops releases are not always tied together

[chrislovecnm]

1558 is related, and https://github.com/kubernetes/kube-deploy/pull/292 needs to be fixed in the base image

[justinsb]

There are three aspects:

1) recognize c5 instances (done)
2) build a new image supporting ENA (I've merged the supporting fixes to the image and am validating them, so more or less done)
3) support NVME volume mounting in protokube, if we want to run a master on c5. not done Kubernetes won't support nvme for 1.8, but we're debating whether we can get it into 1.9. Likely we'll punt this aspect out of the 1.8 milestone, but we'll see...

[chrislovecnm]

In regards to number 3. This means that k8s will not support mount pv as well, I assume.

[justinsb]

Exactly - so I have two PRs (with much the same code), one that adds nvme mounting to protokube, one that adds it to k8s.

[dharmisha]

@justinsb @chrislovecnm
Hi, No new binary has been released which support c5 instances. Is there a specific image that I need to use other than amazon/amzn-ami-hvm-2017.09.1.20171103-x86_64-gp2 to have kops recognize c5? If not what needs to be done for having it resolved.

[thuandt]

kops 1.8.1 has support c5 instances but EC2 instance will failed. We need AMI update also

[dharmisha]

Any idea about the timeline?

On Tue, Dec 5, 2017 at 8:07 AM, Thuan Duong [e-mail]
wrote:

kops 1.8.1 has support c5 instances but EC2 instance will failed. We need
AMI update also

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/kubernetes/kops/issues/3868#issuecomment-349351985,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AH99ckkq4qjO7SGX8bcL9tAxZlNFsrlcks5s9Wo6gaJpZM4Qf0-g
.

[onyxraven]

@justinsb mind linking those PRs for easy tracking from this ticket?

[chrislovecnm]

This is fixed in 1.8. Please see release notes, let us know if you need more details!!

[chrislovecnm]

kops 1.8 which is backwards compatible with k8s

[thuandt]

FYI, if you want use c5 please ensure you didn't use EBS volume. Because kubernetes 1.8 can't mount it as devices path.

Waiting for 1.9 you want to use c5 with EBS

[chrislovecnm]

To be clear pvc volumes are not supported in k8s 1.8 - will be supported in 1.9

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[diegocn]

/remove-lifecycle stale

[diegocn]

Is c5 instances supposed to work on 1.9.x now? I've started a cluster with c5.large nodes and the root volumes just got 8GB on / partition (but the volume starts with 128GB) and I got errors with PODs trying to attach PVCs.

[thuandt]

@diegocn PVCs is only works with stretch image not jessie image.

current stable still use jessie as i see

[diegocn]

Hello @thuandt, thank you.
I'll try to use supported AWS Instances, such as t2.large instead of c5.large for now.

[chenrui333]

+1

[diegocn]

Running a few POCs with other workloads (not K8S), c5 instances are far better than t2. It would be really nice to have support for these instance type.

[rtrive]

Today i try to launch a cluster with c5 instances but the cluster doesn't work...

[faheem-nadeem]

C5 instances have been working for us, using the debian stretch AMI for about two months. Haven't had any problems relating to mounting volumes or any instability.

[thuandt]

@faheem-cliqz problem is upgrade. when do kops upgrade it will replaced stretch with jessie :)

[chenrui333]

@thuandt do you have any code reference for that?

[faheem-nadeem]

@thuandt yup the upgrade part is a problem. We still get the upgrade messages. Just ignoring them for now, and try to upgrade as much as we can with cluster and instance group specs :)

[bcorijn]

@chenrui333 upgrade pulls it's information from the channels, which have jessie images. https://github.com/kubernetes/kops/blob/master/channels/stable

[geojaz]

closed in #5660