Vscode-remote-release: Kubernetes container

[bhack]

Support for containers running in kubernetes (pod, deployment etc..)

[Chuxel]

@bhack Where are you running Kubernetes? If it's local, you should be able to attach to it after you spin it up. F1 > Remote-Containers: Attach to Running Container...

Would that work for you?

[bhack]

Often it is remote specially when I need special (hardware/data) resources.
I need always to use kubectl cp when I edit a file in vscode.

[bhack]

Have you collaboration with https://github.com/Azure/vscode-kubernetes-tools Azure team?

[Chuxel]

@bhack Definatley something we've been thinking about which is partly why I asked about the scenario. We actually have a dev container defintion that allows you to use a Docker sandbox container with all the needed tools (kubctl, helm docker, etc) so you can containerize your build toolchain and still use Kubernetes extension and deploy to a Kubernetes cluster from there: https://github.com/Microsoft/vscode-dev-containers/tree/master/containers/kubernetes-helm

F1 > Create Container Configuration File... > Kubernetes & Helm

[bhack]

F1 > Remote-Containers: Attach to Running Container... for remote K8s could be a good initial feature here.

[bhack]

Just to clarify I meant, as first useful feature, the same use case at (but for K8s):
https://code.visualstudio.com/docs/remote/containers#_attaching-to-running-containers

While using VS Code to spin up a new container can be useful in many situations, it may not match your workflow and you may prefer to "attach" VS Code to an already running container.

Cause my K8S remote workflow "may not match" VS Code spin up.

[Chuxel]

@bhack Yep - totally makes sense. Use kubectl to get to the pod instead of the Docker CLI. K8s configs are more complicated to manage in general so starting with an attach flow makes sense.

[bhack]

I've a question about the server side/extensions. Cause the container/pod nature it is ephemeral how do you handle server side extensions over containers? Do you need to re-configure the remote container extensions every time?

[bhack]

Also I don't understand what could be the case between remote vs container for remote-container K8s case:


We need to suppose that the running code is in the container (i.e. from init container, persitent volume, remote mount, etc) but remote/local edits need to be in sync with the local/remote files.
So there in not a volume mount of a local path like in your current "local" container case.
We need to have an auto-sync over the "source code block" in the diagram.

[bhack]

About source code block sync check https://github.com/ernoaapa/kubectl-warp/issues/10.
As mentioned there I don't know if a sidecar injector could be a good strategy.

[chrmarti]

Extensions running in the container need to be installed in each container. That happens automatically for the extensions listed in the devcontainer.json.

The files don't have to be local for VS Code to work on them. You could just check out the files in the container.

[bhack]

No the problem is that you work on remote files but generally you git commit changes locally. So we need a sync.

[chrmarti]

If you commit / push through the Git viewlet in VS Code, you do it on the remote side currently.

[bhack]

Generally you don't have git push credentials on remote containers.

[chrmarti]

We currently register a credentials helper for Git in the container (if git is installed) that forwards requests to the local Git to lift that limitation.

[bhack]

@chrmarti I still think that a sync is optionally required over the source code about volume block in the diagram.
A very frequent use case is that you test a change on a small scale in a local container (i.e. Docker container) and then you want to test on an active developing Pod (container in k8s).
Generally you don't commit until changes are tested locally and on the pod resources.
So if the change works locally then you want to sync "code block" on the remote existing developing pod/container. If all works also remotely probably you can commit also with your mentioned helper.
So a local to existing pod sync it is required IMHO.

[kdvolder]

@Chuxel said:

Where are you running Kubernetes? If it's local, you should be able to attach to it after you spin it up. F1 > Remote-Containers: Attach to Running Container...

Would that work for you?

If it'd allow me to somehow connect to pods inside my local k8s cluster... yes it would (at least for my current local experimenations). But I don't think that this actually works right now (is it supposed to work already?).

When I use the "Remote-Containers: Attach to Running Container" it shows a list of things that seems to correspond to the stuff you get running docker ps command locally. But the pods running in my k8s cluster (i.e. the things that you get running kubdctl get pods) is not amongst the things it lets me choose from.

So what we need really is what you suggested here:

Yep - totally makes sense. Use kubectl to get to the pod instead of the Docker CLI.

That, by the way, would then also work with any k8s cluster be it local or remote.

[bhack]

Cause this Is still not going in the montly roadmap there Is a boiloperate workaround in the meantime:
https://okteto.com/blog/vs-code-remote-development-in-kubernetes/

[mlgill]

Cause this Is still not going in the montly roadmap there Is a boiloperate workaround in the meantime:
https://okteto.com/blog/vs-code-remote-development-in-kubernetes/

To clarify, it looks like Oketo has to be installed locally and on Kubernetes and does the equivalent of running syncthing (bi-directional sync) locally and remotely as a sidecar container, correct?

Assuming this is correct, VS Code development works through file sync and it's not possible to use any of the debugging features within the remote container.

I do hope this feature gets put into the roadmap so that it will work similar to remote development over SSH.

[bhack]

@mlgill Yes but It Is better than nothing cause there Is anything planned in the next iteraton https://github.com/microsoft/vscode-remote-release/issues/1390

[mlgill]

I get that. Just want to be clear about the limitations.

[bhack]

@mlgill Cause It Is closed source we cannot do anything more that upvote this. So, if you can, please try to help to collect upvotes here in your GitHub network of vscode users.

[kdvolder]

I experimented a bit with using the ssh support to connect to a k8s container. Was able to get this working but the main drawback of the approach is that you have to explicitly install ssh daemon into your 'development contiainer'. It's also a fairly involved process to set it all up. Anyhow... just in case anyone is interested. I put this up here: https://github.com/kdvolder/vscode-remote-java-dev-env

[kdvolder]

@bhack said

Cause It Is closed source

Is that true? If so, I am very surprised! I assumed otherwise given that we are here, filing issues and discussing on github (which is a platform for open source software development). But perhaps it is true after all since when I go and try to find the source code, indeed I can't seem to find any.

I've filed this ticket https://github.com/microsoft/vscode-remote-release/issues/1402 as I think the github readme and docs for vscode-remote are really not clear on that point.

[bhack]

@kdvolder Yes, see this old thread https://github.com/microsoft/vscode-remote-release/issues/179

[chrmarti]

There are a few FAQs about the license.

[rberrelleza]

Cause this Is still not going in the montly roadmap there Is a boiloperate workaround in the meantime:
https://okteto.com/blog/vs-code-remote-development-in-kubernetes/

To clarify, it looks like Oketo has to be installed locally and on Kubernetes and does the equivalent of running syncthing (bi-directional sync) locally and remotely as a sidecar container, correct?

Assuming this is correct, VS Code development works through file sync and it's not possible to use any of the debugging features within the remote container.

I do hope this feature gets put into the roadmap so that it will work similar to remote development over SSH.

@mlgill if you follow the instructions on the blog post that @bhack linked, you'll be able to use all the features of VS Code Remote-SSH, including debugging. You'll also have your development environment running in Kubernetes and with bidirectional file synchronization (this helps if you want to commit from your local machine, instead of forwarding your credentials).

I'm one of the maintainers of github.com/okteto/okteto. We're working on a smother integration with VS Code Remote, your feedback on the scenario and approach would be great.

[mlgill]

Thanks for the reply. Using Oketo requires admin access to the Kubernetes cluster to install a package, correct?

If so, this is not an option for me.

On Thu, Sep 12, 2019, at 01:56, Ramiro Berrelleza wrote:

Cause this Is still not going in the montly roadmap there Is a boiloperate workaround in the meantime:
https://okteto.com/blog/vs-code-remote-development-in-kubernetes/

To clarify, it looks like Oketo has to be installed locally and on Kubernetes and does the equivalent of running syncthing (bi-directional sync) locally and remotely as a sidecar container, correct?

Assuming this is correct, VS Code development works through file sync and it's not possible to use any of the debugging features within the remote container.

I do hope this feature gets put into the roadmap so that it will work similar to remote development over SSH.

@mlgill https://github.com/mlgill if you follow the instructions on the blog post https://okteto.com/blog/vs-code-remote-development-in-kubernetes/ that @bhack https://github.com/bhack linked, you'll be able to use all the features of VS Code Remote-SSH, including debugging. You'll also have your development environment running in Kubernetes and with bidirectional file synchronization (this helps if you want to commit from your local machine, instead of forwarding your credentials).

I'm one of the maintainers of github.com/okteto/okteto. We're working on a smother integration with VS Code Remote, your feedback on the scenario and approach would be great.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/microsoft/vscode-remote-release/issues/12?email_source=notifications&email_token=AANNAOK3CQVO5O7O24IR2ALQJHK2VA5CNFSM4HKE3FZKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD6QX4AY#issuecomment-530677251, or mute the thread https://github.com/notifications/unsubscribe-auth/AANNAOMERQZ6R6XLTHUZDLDQJHK2VANCNFSM4HKE3FZA.

[rberrelleza]

Thanks for the reply. Using Oketo requires admin access to the Kubernetes cluster to install a package, correct? If so, this is not an option for me.
…
On Thu, Sep 12, 2019, at 01:56, Ramiro Berrelleza wrote: >> Cause this Is still not going in the montly roadmap there Is a boiloperate workaround in the meantime: >> https://okteto.com/blog/vs-code-remote-development-in-kubernetes/ > To clarify, it looks like Oketo has to be installed locally and on Kubernetes and does the equivalent of running syncthing (bi-directional sync) locally and remotely as a sidecar container, correct? > Assuming this is correct, VS Code development works through file sync and it's not possible to use any of the debugging features within the remote container. > I do hope this feature gets put into the roadmap so that it will work similar to remote development over SSH. @mlgill https://github.com/mlgill if you follow the instructions on the blog post https://okteto.com/blog/vs-code-remote-development-in-kubernetes/ that @bhack https://github.com/bhack linked, you'll be able to use all the features of VS Code Remote-SSH, including debugging. You'll also have your development environment running in Kubernetes and with bidirectional file synchronization (this helps if you want to commit from your local machine, instead of forwarding your credentials). I'm one of the maintainers of github.com/okteto/okteto. We're working on a smother integration with VS Code Remote, your feedback on the scenario and approach would be great. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#12?email_source=notifications&email_token=AANNAOK3CQVO5O7O24IR2ALQJHK2VA5CNFSM4HKE3FZKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD6QX4AY#issuecomment-530677251>, or mute the thread https://github.com/notifications/unsubscribe-auth/AANNAOMERQZ6R6XLTHUZDLDQJHK2VANCNFSM4HKE3FZA.

@mlgill You don't need admin permissions to the cluster, just deploy privileges on a namespace. If you can deploy an application to your cluster, you can use Okteto. We have an okteto channel in the Kubernetes slack, we can continue the conversation there 😄 .

[bhack]

@rberrelleza

The dev image includes an SSH server

It Is still quite unmpratical to hack every image but we haven't any other solution currently.

[kwlzn]

is there some reason that the plugin couldn't use e.g. kubectl exec the same way e.g. Remote SSH uses ssh to install and start up the remote vscode process group?

[chrmarti]

There are a few other Docker-specific things we use. (E.g., an NPM package to inspect containers / images and to listen for events.)

[bhack]

There are many NPMs also for Kubernetes like https://www.npmjs.com/package/kubernetes-client

[bhack]

I think that also some code could be reused from the other Microsoft plugin https://marketplace.visualstudio.com/items?itemName=ms-kubernetes-tools.vscode-kubernetes-tools

[bhack]

This issue now is the 2nd most upvoted feature request in the container sub-component/label.
I hope it will start to join in the imminent October roadmap cycle ticket.

[vanbroup]

We created a proof of concept ssh proxy for visual studio code that deploys a pod in your selected namespace with a custom docker image (make sure you have wget or curl in your image) on Kubernetes for the authenticated user.

https://github.com/digitorus/remote-vsc-k8s

Please check the repository of you want to give it a try or want to help with contributions. Feedback is welcome!

[bhack]

October iteration plan was just opened:
https://github.com/microsoft/vscode-remote-release/issues/1661

I hope we will see something about this in the container section.

[kdvolder]

I hope we will see something about this in the container section.

Agreed. I hope so too.

In the mean time it might be interesting for people who are looking to connect / run a development environment to something deployed in a k8s cluster to consider using Theia instead of vscode. Interesting fact, Theia is fully compatible with vscode extension api. You can drop a vscode .vsix file into Theia and it usually just works (tried it with several of our own vscode extensions with great success in the gitpod version of Theia). Theia also provides a very similar look and feel based on the same Monacco editor as vscode, and even down to the level of keyboard shortcuts it looks and feel like you are using vscode inside a web browser.

So if vscode remote support for k8s does not seem to be coming soon, it is reassuring to know that perhaps Theia might be an alternative way for us to provide a k8s based IDE experience. Our team will start exploring this a bit soon.

[davidwindell]

I also strongly recommend looking at Eclipse Che which provides Their based K8's workspaces.

[pchico83]

We have released a VS Code plugin for remote development in Kubernetes:
Release blog post: https://okteto.com/blog/remote-kubernetes-development
Marketplace: https://marketplace.visualstudio.com/items?itemName=okteto.remote-kubernetes
@bhack it does not require to modify your image to include an ssh server any more

[ggaaooppeenngg]

Suppose I want to multiplex clients requests through a proxy, ssh can not help, I would like other protocol (Websocket HTTP) than SSH to remotely control my pod.

[pchico83]

@ggaaooppeenngg what is the use case for having multi clients?

[ggaaooppeenngg]

@pchico83
I am searching for a tool like vscode to provide coding environment in cloud to customers.
For ssh, I need to provide a direct external tcp service to user, in cloud environment it could be ip address:port or a loadbalancer which costs time to be established, while if through HTTP protocol I can dynamically config a route from proxy to to a pod like Jupyter Notebook.
BTW, we don't provide K8S access control to users.

[bhack]

Kubectl operatios like exec use API:
https://erkanerol.github.io/post/how-kubectl-exec-works/

I suppose we are talking about a solution without SSH for Kubernetes.

[pchico83]

@ggaaooppeenngg We can support your use case, ping me at twitter (@pchico83) if you want to connect.
@bhack why do you like it better via kubectl exec? Our ssh connection is done via port-forward to the pod. The ssh server is injected in the pod itself. ssh is more standard and supports things like reverse proxying. We can adapt our solution to work via kubectl exec + kubectl port-forward but we would need to understand the benefits better because it requires much more work.

[bhack]

@pchico83 Cause I've seen that in the last two years this upstream proposal didn't got momentum:
https://github.com/kubernetes/kubernetes/issues/42950

[bhack]

If you follow that upstream ticket It Is not clear why they have not adopted ssh protocol or kubectl ssh subcommand. So I am not sure what are the upstream drawback.

[ggaaooppeenngg]

@pchico83

I found http CONNECT method can tunnel ssh.

package main

import (
    "fmt"
    "log"
    "net/http"

    "github.com/elazarl/goproxy"
)

func httpsHandler(host string, ctx *goproxy.ProxyCtx) (*goproxy.ConnectAction, string) {
        // apply some rules to fake-host and proxy to real host
    return goproxy.OkConnect, "your-real-ssh-host"
}

func main() {
    proxy := goproxy.NewProxyHttpServer()
    proxy.Verbose = true
    proxy.OnRequest().HandleConnectFunc(httpsHandler)
    log.Fatal(http.ListenAndServe(":8892", proxy))
}

ssh user@fake-host -o "ProxyCommand=nc -X connect -x 127.0.0.1:8892 %h %p" could connect to the real-ssh-host

[pchico83]

@ggaaooppeenngg thanks for the code snippet :blush:
Yes, we could leverage the ssh connection through https very easily.

[bhack]

Seems that this Is not in the November Plan:

https://github.com/microsoft/vscode-remote-release/issues/1775

[bhack]

November plan was finalized. No Kubernetes initial support this month.

[bhack]

Any Kubernetes support again in the Jan 2020 dev iteration also if this ticket is in the top 2 community priorities for containers section: https://github.com/microsoft/vscode-remote-release/issues/2075

[bhack]

Explore support to attach to Kubernetes

is in the Feb plan https://github.com/microsoft/vscode-remote-release/issues/2329

[chrmarti]

The latest Remote-Containers (0.107.0) for VS Code Insiders now supports attaching to a container in Kubernetes. There is a context menu action Attach Visual Studio Code on pods in the Kubernetes Explorer:

Gif is too large to post here: https://twitter.com/christof_marti/status/1237395371241791489?s=20

[lumaxis]

I had pinged @chrmarti on Twitter about this already bug adding a note here too:
Currently, the extension requires the Kubernetes container to be running as root which many services will likely not, due to the security concerns that come with that. Would be great if it also worked with a non-root user.

[chrmarti]

@lumaxis I have a fix for running as non-root queued for the next release. Should be out for VS Code Insiders sometime next week.

[lumaxis]

Great! I'll keep an eye out for the change and give it another spin then 👌🏼

[bhack]

@chrmarti What is the next step? Do you plan to support run?

[chrmarti]

@bhack That could be a next step. I imagine that you would usually have several containers deployed and the DevContainer would be one of them. In that case run might only work well when the DevContainer only has dependencies on the other containers and not the other way around.

[bhack]

@chrmarti Probably Yes. For more complex scenario we could support apply. I don't know if we could control kubectl apply -k so that Kustomize could handle more complex configurations.

[TripleEmcoder]

Judging from the error message I get when trying to attach to a Windows container, this isn't supported yet, is it?

[chrmarti]

@TripleEmcoder Correct, Windows containers are not supported currently.

[erihanse]

Nice job! Can we also support the scenario where read-only root filesystems are enabled?

[itssimon]

I can't get remote workspaces to work. I can open a single remote folder, but adding more folders to the workspace breaks it. Folders are showing in yellow with an exclamation mark next to it and no content.

[chrmarti]

Created new issues for the above mentioned request (#2755) and bug (#2756). Thanks.

[kwlzn]

does Remote-Containers support attaching to remote kubernetes pods via kubectl? when I attempted to use it, it seemed to require a local docker daemon.

we'd like to connect to pods in large compute clusters like e.g. Google's GKE - wasn't able to figure out how to get that working. any pointers?

[pchico83]

@kwlzn do you know. this extension?
https://marketplace.visualstudio.com/items?itemName=okteto.remote-kubernetes

[chrmarti]

@kwlzn You need Microsoft's Kubernetes extension and the Remote-Containers extension installed to attach to a container in Kubernetes with the work being done here.

See the VS Code 1.44 Remote Release Notes for a screenshot: https://github.com/microsoft/vscode-docs/blob/master/remote-release-notes/v1_44.md#kubernetes-support

[bhack]

@chrmarti Any idea for April? Is there any proposal for attach/run?
Cause run still doesn't support Manifest https://github.com/kubernetes/kubernetes/issues/63214.
Instead apply It work also on complex config cases cause it sopport also kustomize parameter.

[bhack]

Sorry I meant apply. It was a typo.

[chrmarti]

@bhack Nothing is planned for April.

I wonder if it makes sense for us to favor one way of creating a container over another or if we should try to find a way to generically support user-defined commands to create containers.

Also: Where does that information on how to create a container come from? Would that be in user settings or would there be a locally checked out repository with the information on how to get to a container and that container could then contain (or have mounted) the source repository (which can be different)?

With Docker we set things up in a way that works well locally where the same devcontainer.json contains information on how to create a container and how to configure it once connected.

[bhack]

I think that we need to maintain all the Docker features related to devcontainer.json but we could add a push step cause often the remote container need to (optionally) pull the image from a remote registry in Kubernetes in the case we manipulate the Dockerfile (currently we monitor Dockerfile but we propose only a build step).

Then I think that we could expand devcontainer.json with a the deployment command in as an initial feature. So that user can users could choice if the want to deploy with helm, kubectl, etc.. command.

Then later we could support of kind of image builds step that are not strictly related to docker build (i.e. remote kokoro etc...)

[PavelSosin-320]

@bhack There is no "remote" registry. I deployed the registry image to my Docker engine in 2 minutes and use it for docker service deployment. All used images can be replicated from the central repository to local and back manually at the cost of 5 minutes per image. docker-compose push and docker stack deploy are unaware which repository is used - it is simple ARG. It makes the Development environment fast - no need to publish tag for every build and testing & production environment stable - local registry tags can reference vscode version and back.

[bhack]

E.g. But how you handle that case when your Docker engine infrastrcture It Is running under a remote kubernetes cluster where there is the hardware that you want to develop on like ML/Deep learning development where you want to develop GPU code on a remote GPU container/node provisioned by Kubernetes.

[wstrange]

Not sure if we are talking about the same thing, but we are experimenting with using kaniko in conjunction with vscode remote - so the pod does not need access to the docker socket.

[bhack]

@wstrange We have a Dockerfile monitoring in this extension that propose you to rebuild the image and restart the container if needed. I cannot see the internals cause the extension is closed source but how do you propose to handle the interaction of this machinery with Kaniko?

[wstrange]

I think you are talking about the Dockerfile that defines the pod where vscode is running itself? I dont think Kaniko will solve that problem.

We bundle up all the tools we need for development (skaffold, kustomize, etc.) into the pod, and run skaffold inside the pod (we install the google cloud code plugin in the remote side). Skaffold uses the kaniko builder - so we do not need access to the docker socket.

[bhack]

Yes I am talking about the Dockerfile that is referenced by the .devcontainer.json on which this extension is interacting on.

[PavelSosin-320]

I don't understand something; dickerfile which defines a pod? Dockerfile dependent on hardware? In Kubernetes pod which contains hardware-dependent images can be scheduled to the nodes with such hardware by the label. Hardware drivers can be installed on the node because they are hardware-dependent. How Docker container, i.e. subprocess can communicate with the hardware directly bypassing pod and node isolation? Only Windows containers on the Windows machine can ignore isolation.
Do you mean something like NVIDIA container which doesn't use any Kaniko?

[bhack]

Currently the extension is supporting only attach so it doesn't care how you have deployed the pod. Here we are discussing the next iteration steps.

[bhack]

@chrmarti Do you think that we could start to control the deployment in this month iteration?

[bhack]

I think that what we mentioned at https://github.com/microsoft/vscode-remote-release/issues/12#issuecomment-613359085 could be the next step.

[chrmarti]

@bhack We don't have this planed for the June milestone. The main challenge from my point of view is that there are several commands for deployment, each with its own options to affect the outcome. It is not clear to me that picking one command and a subset of options will get this to work for most users. Offering all deployment commands and all their options on the other hand seems overly complex to implement and use.

What if you could configure the commands used to set up and connect to a DevContainer? The commands can use kubectl or any other locally available CLI.

A simple way of doing this would be to have a single command that sets up the container (if not set up already) and ends with a /bin/sh session that Remote-Containers can use to install the VS Code Server.

[bhack]

@chrmarti So what do you think we will have?
As we are monitoring the Dockerfile do you think that we need an user pre-deployment command (i.e. for docker build and push, remote build and push, etc..) and an user deployment command?
Do you want to pilot generic executables or do you want to interact with the cluster using the JavaScript client library or a mixed approach?

[chrmarti]

Trying to come up with a simple example. You could also run separate script files for each of these:

{
    "start": "kubectl create deployment --image=nginx nginx-dev ; kubectl wait --for=condition=Available deployment.apps/nginx-dev",
    "attach": "kubectl exec -i deployment.apps/nginx-dev -- /bin/sh",
    "rebuild": "kubectl delete deployment nginx-dev",
    "stop": "kubectl delete deployment nginx-dev"
}

Only "attach" would be mandatory, the other commands would be optional. This configuration would have to be available locally either as user data or checked out from SCM.

Detecting changes in a Dockerfile or other files would require additional configuration.

[bhack]

I think this command could be ok but are you sure that for the kubectl case you don't want to use the client library instead of interacting with the exec?

[chrmarti]

Using the CLI has the advantage that we can leave the choice of deployment and other details to the user. We might provide examples of how you can set things up, but you could then still tune them to your specific needs. I haven't explored this in much depth yet though, does that sound like it might work for what you plan on doing?

[bhack]

This really depend on our use case scenario. I meant if we think that 80% of the targets commands Is kubectl we Cloud integrated with the library for a small subset of commands. Then we could let the user to override it like in your example. For that case we will not have so much control on command outputs/returns but It could be ok for custom cases to let the user to just check the log if something goes wrong.

[zywillc]

any plan or progress on this request?