Envoy: Cloud developer flow for Windows builds

[htuch]

In order to debug Windows builds (e.g. with CI failures), it's helpful to allow arbitrary Envoy developers the ability to perform Windows builds. I think there are two options here:

1. We have some pool of Windows VMs that we can share access to which are ready to go for Windows debugging.
2. We document, e.g. in README.md in https://github.com/envoyproxy/envoy/tree/master/bazel, how to setup a Windows build from scratch on AWS/GCP. This documentation should be detailed enough that following it will result in a valid Windows build environment, even for non-experienced Windows devs.

I think (2) is most likely to be viable, but (1) would be awesome if there was some way to achieve it.

CC @wrowe @sunjayBhatia @stedsome

[sunjayBhatia]

I was hoping something like the existing Linux dev container workflow would work, but looks like VSCode doesn't yet support windows containers: https://code.visualstudio.com/docs/remote/containers#_known-limitations

Setting up a remote docker host and having developers connect to that could be an option as well: https://docs.microsoft.com/en-us/virtualization/windowscontainers/management/manage_remotehost

[sunjayBhatia]

I can't see anywhere yet that Github codespaces supports windows, but that may be a future option

[wrowe]

We could potentially set this up with Azure, but the risk is that many tests including the ginormous mock tests still fail to compile in the small-ish azure containers. If this were restricted to only senior maintainers, it might be possible to rely on the gcp rbe environment for the heavy lifting. Knowing how large the resulting symbol tables are, I'm not sure if the Azure instances we've stood up for CI in the past are really sufficient. Agreed with your goals as stated.

[wrowe]

cc @envoyproxy/windows-dev to bring you all into this dialog

[sunjayBhatia]

Spent some time on this and it seems option (1) above is achievable with relatively low investment/moving parts. A first iteration would be a pool of Windows VMs maintainers can SSH to for rudimentary debugging.

• Works with the existing setup we have in ci-infra whereas VMs (currently for CI workers) are registered and deregistered
  
  
  
  • Maintain a pool of at least 1 claimable Windows workstation in AWS ASG
  
  
  • Maintainers get SSH key and can SSH to workstation, on SSH connection automate the VM detaching itself from ASG, ASG starts another standby VM available to claim
  
  
  • Likely the flow would be to clone the Envoy codebase, start up the dev container interactively (./ci/run_envoy_docker_windows.sh bash), use VSCode remote editing over ssh (or something similar) to edit files on the remote workstation, and iterate to debug
  
  
  • When user is done with the VM, terminate VM to clean it up
  
  
• Should be pretty much 1 VM running large majority of the time, relatively inexpensive
• Cloud providers often provide Windows VM images with the containers feature and Docker pre-installed, regardless, Docker is easy to install
• Only other dependencies we need to install are our lifecycle scripts, Git/bash, enable SSH/public key authentication

This first pass would enable maintainers to spin up a Docker container with the required tools to build on Windows. If RBE credentials are available, they will be able to take advantage of the remote cache/execution for builds. For debugging, initially this will enable simple log/trace debugging and help people fix compilation issues. We can work on attaching a remote debugger/debug server/hacking VSCode to provide remote devcontainer support (not currently supported on Windows) in the future if needed.

Links:

• https://docs.microsoft.com/en-us/windows-server/administration/openssh/openssh_install_firstuse
• https://www.concurrency.com/blog/may-2019/key-based-authentication-for-openssh-on-windows
• https://github.com/envoyproxy/ci-infra#self-hosted-azp-agents
• http://man.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/man5/sshd_config.5?query=sshd_config#ForceCommand

[sunjayBhatia]

https://github.com/envoyproxy/envoy/pull/13282 is a direct example where we could have used this, should bump the priority of getting this done