Kops: Managing cluster using terraform after diverging from kops

[MirrorZ]

Using terraform output from kops with AWS enables one to provision a kubernetes cluster. Terraform can then be used to add specific IAM roles, security groups etc and the state diverges from what was the intially set up by kops. So how can kops be used to carry out more granular updates with terraform such as enabling batch jobs for the current kubernetes cluster.

kops : 1.6
kubectl client: 1.7
cloud provider: AWS

[chrislovecnm]

So first, please use kops 1.7.1 or the beta of 1.8.x. We are backward compatible.

So how can kops be used to carry out more granular updates with terraform such as enabling batch jobs for the current kubernetes cluster.

So there are multiple things to consider. And this is a complicated question :)

First, you need to understand that there is kops, nodeup, and protokube in the kops project. A couple of other toys as well. We use TF to build infrastructure only. nodeup handles node creation and configuration.

enabling batch jobs

Is handled by nodeup. nodeup accesses the cluster manifest, which is in the state store. Based on that it configures such components as the k8s api controller.

I have some issues open around doing a deep dive documentation about this. If you want to talk for a few minutes on a video call, I am happy to offer some time. Maybe we can figure out a good way to explain this in the documentation.

[macropin]

It would be great to see this documented.

[notmaxx]

@chrislovecnm I have similar question, I would like to use my tf configuration mgmt system but really love how kops does k8s setup & startup. So if it is possible to manage/create/modify configuration for nodeup, protokube (what else?) by terraform and its providers/templates that would be great

[simoneroselli]

@notmaxx I had the same idea, but that would be an Hashicorp Terraform feature request. Most probably Terraform devs will work on AWS EKS (_AWS promise of kubernetes as a service_), but it would be super useful for my point of view to have a kops provider in Terraform. In this way (if it's feasible) we could benefit from the granularity of kops setup, without overlapping with VPC/subnets or other resources created by TF.

[tahoward]

I'm running into a similar situation as well. Right now I'm using Terraform to generate contents of the state store via scripts executed in null resources. I lose the advantages of kops but get to leverage nodeup for bootstrap ping the cluster. A Terraform provider for kops would be interesting.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close