Kops: Azure Support

[chrislovecnm]

Problem Statement

Support the running a kops cluster on Azure. This implementation would model AWS and GCE, and would be awesome!

How

Azure includes full API support for the creation of virtual machines and their scaling groups. There are multiple examples of the 'hard way' but we first can model there managed version.

Why

_Why not_ ? Having AWS, GCE, Azure is the trifecta

Design

TBD

[robinpercy]

+1 Cloud-agnosticism is a consistent priority I hear from customers deploying K8s. Being able to use Kops across all three would solve a lot of pain.

[chrislovecnm]

@robinpercy I have gotten asks from my clients that they want to run on the big three. We got two down, one to go

[vendrov]

Great idea! I believe that our first step should be, adding windows nodes support for kops :)

[chrislovecnm]

Yes we do need windows node support as well. Will open a separate ticket. I have talked with Microsoft shops who would love that,

[vendrov]

No need to open a new one, https://github.com/kubernetes/kops/issues/2422

[bgeesaman]

I've looked at over a dozen K8s installation tools and managed K8s services in the past few months on all major providers. What was painfully clear was the level of fragmentation in the tools and offerings as well as what features are supported. It required a ton of mental energy to translate terms and patterns to get similar output.

We are at a point where focused efforts like this can provide that next level of true abstraction so that administrators/operators can see and manage Kubernetes uniformly--regardless of cloud provider. I think bringing all the advanced features and capabilities (HA, rolling-upgrades, etc) battle-tested in AWS and GCE to Azure is a no-brainer.

[chrislovecnm]

@bgeesaman well put - fortunately the coming machine api should help with this, but the installers do have different use cases

[geojaz]

We should be building out azure support regardless of windows nodes. We can definitely consider building out windows node support in kops, but I would start with the basics. That is to say, let's build out linux support first because that's what we support on AWS and GCE. That will bring some level of feature parity to the 3 biggies via kops.

I too have clients that need 1 tool to manage all the clusters across these providers and kops is a great way to go. We now have much of the background work done to build a new provider so I think it's quite doable. I'm personally not that up to speed on Azure, but I'd like to see a good "azure-native" implementation to complement aws and gcp. Excited to see this get attention and I'm happy to help.

[robertojrojas]

IMHO, Azure support is a must have! I agree with @geojaz windows support would be nice, but Linux should come first. I'll be interesting to see what changes would be needed in the CLI and API to accomplish this. I'm happy to help as well.

[brandoncole]

This would be great. There are definitely some companies who prefer Azure to AWS in foreign countries which is where this comes into play a lot. In the U.S.A. there's indifference to some degree in the cloud vendor, but there's certain places globally where the choice has been made for you which we have to respect.

[chrislovecnm]

/assign

[justinsb]

Yes, I'd love to see us work on this. Azure is doing some very cool stuff, and it would be great to help people to try out Kubernetes on Azure as easily as they can try out AWS & now GCE.

[hubt]

This would be excellent. Windows support isn't important at least initially. People need this for cloud portability(usually because of cost), not OS portability.

[chrislovecnm]

@hubt we have had one request for windows which mean x100 have wanted it. But KISS right? And that work can be completed in ||.

[perryao]

@chrislovecnm Is this work active or still awaiting the machine api?

[chrislovecnm]

@perryao nobody has started on this, would love someone to pick this up!

[perryao]

I'd like to try my hand at this. I'll work on a rough version over the coming weeks and hopefully come back to discuss design with you @chrislovecnm

[chrislovecnm]

@perryao let me know, if you have questions happy to chat with you.

[rajeevb2]

@perryao any news on kops for azure?

[perryao]

@rajeevb2 unfortunately not too much. I have started and have a pretty good feel for what needs to be done, but haven't had enough time to make meaningful progress on it yet. But still a priority for me.

At this point I've just stubbed out the tasks I'll need in cloudup and starting implementing a few interfaces.

[chrislovecnm]

@perryao if you want to add a feature flag, please start pushing code. Love to have more than one person working on this if you feel it would help.

Sometimes more people don't help 😆

[zarakay]

Happy to help with testing if you need it

[marc-sensenich]

@perryao If you would like to have some help on this from the testing and development front, I'd be willing to help out

[perryao]

@chrislovecnm @marc-sensenich thanks guys. I just pushed the initial stuff I did to layout the changes I'd need to make. Here's the diff of where I'm at now: https://github.com/kubernetes/kops/compare/master...perryao:feature/%233957azure-support?expand=1. I'm wrapping up a deliverable at work this week, but I think I should be back to working on this early next week.

[chezbut]

Is there any plan for being able to run this on local Hyper-V hosts, with multiple VM's?

[Cittarasu]

@chrislovecnm @perryao -- great start on Azure front. I can help in testing/development front. Let me know how we can sync-up on this front. We have been using KOPS for AWS extensively but we are exploring KOPS for vsphere and Azure.

[luisdavim]

Any updates on this?

[thangchung]

+1

[surenm]

Great start @perryao @chrislovecnm. Happy to help with the testing effort!

[wandergeek]

+10

[rajivchodisetti]

+10

[billpratt]

Hello everyone. I have started to work on this during free time. You can track progress here: https://github.com/billpratt/kops/commits/azure

So far I have basic state-store functionality working with Azure blob storage.

[dazdaz]

Awesome to see the beginning of the work to implement Azure support with kops.
May I ask what the current status is.
Have you considered creating a roadmap of features/comaptibility to highlight where we're at and what requires developer effort.

[billpratt]

@dazdaz I havent had as much time lately to work on this. I plan to start up again soon. I originally created this project on my fork. It needs a little bit of cleanup. I also ran into a blocker where a downstream dependency of kops requires a specific version of the azure-sdk-for-go. I will need to update to use a newer version. I need to also evaluate what version of k8s to start supporting this in Azure. Kubernetes 1.12 released support for Azure VM Scale Sets, which gives similar functionality as AWS scale sets. If kops is going to support < 1.12 for Azure, I'll have to use Azure Availability Sets which does not give you automatic cluster scaling.

[argiris-mat]

@billpratt hello there. Can we do anything to support this? Is there anything I could test or document?

[billpratt]

Not yet. I have some free time coming up and I plan to continue to work on this. I will let you know when something is ready.

[igormusinov]

any updates?

[innlouvate]

Any chance this might be coming in the next 6 months or so? Would happily support if I can.

[bzuelke]

i need this in my life so bad. I hope in the near future this can become a reality. I hate having to force myself to have a snowflake in Azure from our AWS clusters.

[DG4227]

Would love to see this come to fruition!

[bzuelke]

@billpratt I've looked over your fork and looks like you've done a ton of leg work. Is there anything that anyone here could do to try and get this in a PR? Didn't want to reinvent the wheel if I didn't have to. I'd love to have something in place for this before end of year. Would be incredibly useful.

[bryaneburr]

MS loves to give away free time on azure, and higher-ups love to take it. This would make devops' lives like a billion times easier 😀

[billpratt]

Hi everyone,

This was strictly a personal project of mine that I worked on at night and no work was done as a Microsoft employee. Unfortunately I have had no time finish. @bzuelke, if you or anyone else on this thread would like to fork what I've done and take over, please do as I would also love to see this feature added 😃

[bzuelke]

Thanks for getting back to the thread @billpratt I totally get where you're at. I'm kind of in the same boat time wise which is frustrating naturally. I know you have a decent amount of leg work put into the code and if the universe is willing to grant me some time to check it out I'll 100% fork it and muck with it. Or if someone else jumps on it I'm totally open to collab to work this out.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[pierluigilenoci]

@billpratt any news?

[bzuelke]

Chances are there won't be any support for Azure unless someone really jumps on it. But I think more people use other forms of cluster management until Cluster API goes GA or simply uses AKS. I would have like to take a stab at this but simply don't have resources available to follow up on Bills work :/

[christianh814]

Chances are there won't be any support for Azure unless someone really jumps on it. But I think more people use other forms of cluster management until Cluster API goes GA or simply uses AKS. I would have like to take a stab at this but simply don't have resources available to follow up on Bills work :/

+1 to this comment and the ":/" ...realistically; when Cluster API matures, KOPS would/should use that as the core of the tool.

[prksu]

/remove-lifecycle stale

[rendhalver]

Just catching up on this. I see cluster-api has a non pre-release release in 0.28.0.
Is that what people were waiting for?
I am also interested in helping out where I can.
I have been looking for reason to learn more go and we use kops to deploy our clusters and having the ability to manage all of them with kops would be grand.

[romantsai-github]

any updated? like to know how kops support on Azure if possible?

[Cittarasu]

Wondering why KOPS support require - as each cloud provider now providing
k8s engine via API or CLI support.

On Mon, 10 Feb 2020 at 07:38, romantsai-github [e-mail]
wrote:

any updated? like to know how kops support on Azure if possible?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/kubernetes/kops/issues/3957?email_source=notifications&email_token=AD4I5YL3OKAUHM45DAHANNDRCCZINA5CNFSM4EFZ27GKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOELHAIXY#issuecomment-583926879,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AD4I5YIXO6XLEUYIAUHOZKLRCCZINANCNFSM4EFZ27GA
.

[lsca]

Any updates on this?

[bzuelke]

Mentioned before this will probably not be a thing unless someone jumps on it. I don't believe anyone is currently working on this or have initiative. IIRC cluster API will be a defacto for Azure. I know Kops will begin utilizing CPAI based on the talk they gave at kubecon 2019 but still isn't a thing yet and will be in phases. Also at said talk they mentioned efforts on other aspects of kops so thinking main contributors aren't focusing efforts on Azure and might never. Someone will most likely need to hack through this or use another tooling like AKS-Engine or CAPI. I mention this considering the previous conversations in this thread and the age of it.

[luisdavim]

Yeah, focusing on cluster API seems like a better idea that would eventually lead to solving this issue as well.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[Bessonov]

/remove-lifecycle stale

[hakman]

/kind feature
/lifecycle frozen

[kenji-cloudnatix]

Just wanted to update this issue as we have a fork that supports Azure.

It's not fully done, but major features have been implemented, including:

• management of Azure resources such as Resource Group, Virtual Network, VM Scale Sets and Managed Disks.
• etcd cluster setup with Azure Managed Disks in etcd-manager
• Azure Cloud config injection
• VFS backed by Azure Blob storage service

We can keep the changes in the fork, but we would like to upstream it if possible. Is there any interest in the community? Here are the change diffs:

• kops: https://github.com/cloudnatix/kops-public/compare/base...cloudnatix:azure
• etcd-manager: https://github.com/cloudnatix/etcd-manager-public/compare/base...cloudnatix:azure

[bzuelke]

personally id love to see support for this even if there is a shift toward cluster-api. this is amazing and can be added onto if needed. haven't checked the code base yet but this would be great to have 1 tool for multi cloud provider if we can.

[rifelpet]

Great work @kenji-cloudnatix ! Would you or someone from your team be able to join the next Kops office hours? Friday Oct 23rd at 12pm US/Eastern. We can discuss a high level plan with other maintainers there and then hash out any details here or in PRs.

[hakman]

Would be really cool to get this into Kops. Thanks @kenji-cloudnatix !

[kenji-cloudnatix]

Thanks all! Sure, will join the office hours on Oct 23rd.

[rifelpet]

@kenji-cloudnatix as discussed today, I think our next steps will be:

1. Open the big kops PR with all the changes. we'll likely end up breaking it up into smaller PRs but it helps to see the full scope of changes.
2. Open a kops PR with just the VFS changes (util/pkg/vfs/*) + necessary dependencies
3. Once the VFS PR is merged, open the etcd-manager PR with the updated kops dependency

In the big PR we can further discuss how to break down changes as necessary.

[kenji-cloudnatix]

Sure, @rifelpet ! Just opened https://github.com/kubernetes/kops/pull/10114 for the first item.