Dashboard: Exec into Pod

[ianlewis]

I am planning on working on "exec into Pod" functionality. I'm planning on gathering requirements via this issue and creating a proof of concept.

• [x] Gather requirements
• [x] Create POC
• [x] Create UX design
• [ ] Implementation

My general thought is that I will search for and use an existing library on the client side to display the terminal and take input. I'll use something like websockets or whatever transport the client side library supports for terminal events. The server side will use the exec endpoint from the Kubernetes API for connecting to the actual pod.

Related issues: #21 #550

[ianlewis]

I may be getting ahead of myself, but a couple of thoughts re: requirements:

1. kubectl exec supports the -t option to enable tty. Do we need to have the ability to turn tty on/off? I can think of cases where it isn't needed but are there cases where the user would need to turn these off? My first thought is that we can leave this option out and just enable tty all the time for simplicity.
2. kubectl exec supports the -i option to enable stdin. Is there a need for some kind of stdin option? i.e. some kind of text box where the user can put in text that can be sent on stdin? My first thought is no, but I want to make sure.
3. kubectl attach is very similar to kubectl exec and my feeling is that the machinery to make exec and attach work would be pretty much the same. They could probably be developed at the same time or attach could be added quickly after exec into pod.

[ianlewis]

Here are a few browser based terminal libraries I'm looking at:

• http://xtermjs.org/
• http://terminal.js.org/
• http://terminal.jcubic.pl/
• http://cb.vu/

xterm.js seems like the best based on a cursory look. It's a fairly active project and is used as part of Microsoft Visual Studio Code for their integrated terminal.

[ianlewis]

In terms of requirements it looks like debugging is the main use case for exec into pod.

This might affect the UI and how users discover the feature, but for now it doesn't seem like it's going to affect base functionality from what I described above.

[colemickens]

No hterm (https://chromium.googlesource.com/apps/libapps/+/HEAD/hterm)? This is what Google's Secure Shell extension for Chrome uses, IIRC.

[ianlewis]

@colemickens Cool. I didn't know about that one.

[bryk]

kubectl exec supports the -t option to enable tty. Do we need to have the ability to turn tty on/off? I can think of cases where it isn't needed but are there cases where the user would need to turn these off? My first thought is that we can leave this option out and just enable tty all the time for simplicity.

Yep, I also think so.

kubectl exec supports the -i option to enable stdin. Is there a need for some kind of stdin option? i.e. some kind of text box where the user can put in text that can be sent on stdin? My first thought is no, but I want to make sure.

Do you suggest to enable or disable this option by default? AFAIU this should be always enabled to make interactivity work? Or am I missing something?

kubectl attach is very similar to kubectl exec and my feeling is that the machinery to make exec and attach work would be pretty much the same. They could probably be developed at the same time or attach could be added quickly after exec into pod.

Yes, exactly :)

[bryk]

In terms of requirements it looks like debugging is the main use case for exec into pod.
This might affect the UI and how users discover the feature, but for now it doesn't seem like it's going to affect base functionality from what I described above.

Do you have any proposals for interaction flow of this feature? My rough understanding is that users look at a pod and then there is a button or already opened card to start the interaction. But then there are problems. What's the interaction with the exec functionality? I sketched a rough mock of how I see it:

It has some problems, though. E.g., it is hard to use when all you want to do is to open a bash inside your pod. But on the other hand it allows you to do everything that exec allows for, and people do use the other features.

I'm interested what do you folks think. @romlein @rf232 ?

[rf232]

kubectl exec supports the -i option to enable stdin. Is there a need for some kind of stdin option? i.e. some kind of text box where the user can put in text that can be sent on stdin? My first thought is no, but I want to make sure.

Do you suggest to enable or disable this option by default? AFAIU this should be always enabled to make interactivity work? Or am I missing something?

You indeed need the -i (and the -t) if you want something like bash to work. I don't know how high the overhead would be for something like kubectl -p my-pod ps aux to add the -i and -t, so don't know if we should worry about making that configurable.

About the mocks. I think it looks good if you really only want to do two/three things in the shell, but if you want to do more you might want to be able to 'popout' the terminal window and move it around yourself.

[bryk]

About the mocks. I think it looks good if you really only want to do two/three things in the shell, but if you want to do more you might want to be able to 'popout' the terminal window and move it around yourself.

Yeah, this may be pop-out-able card or a window by default. Like chrome debugging tools.

[ianlewis]

Just to be clear, I think having a popup dialog for the terminal interface is best and have it able to pop out.

I also think we need a way to "force quit" the terminal. Simply closing the window is the easiest and most natural way I think.

Something like this:

[bryk]

Agreed with all stated above.

@IanLewis Can you propose a complete user journey for a use case? E.g., "I want to debug a pod. First I start with ps aux. I notice bad stuff. Then I open bash and type some more commands. Then I finish the session".

[ianlewis]

@bryk Sure I'll write something up here.

I think pop-out functionality isn't core. What do you think about doing it as a separate PR?

[ianlewis]

@bryk can you assign this to me?

[bryk]

I think pop-out functionality isn't core. What do you think about doing it as a separate PR?

Yes, definitely. If you can I'd even suggest doing this in several incremental PRs. Like: start with backend, then do frontend prototype that is still hidden, then finalize look and feel, then productionize.

[djsly]

Here's a typical user journey on our side.

I'm a developer, I'm testing out my new container and related manifest. I realized that the behavior isn't correct so I log in the Kubebernetes-Dashboard, looks at the deployment tab, identify my deployment, open up the pod page. I would then like to have a TAB where I could get access to a terminal for all of my containers running in my pod. I always like to run a bash command to really get a nice terminal functionality. I look at the ENV configured in my container and realized that some of them are wrongly configured. I also perform cross container communication tests. Damn I don't have telnet installed, so I install it for this single time usage. The webserver can be reached using its Cluster IP nice, but I still can't access it using the service name. Let's install nslookup and see if I have a potential issue with the dns. It is not working :( Could be affecting all containers ? I copy my long command from one terminal and paste it in another terminal to see if I have the same issue. I also need to test the same connectivity from the same POD but a different replica on a different node. So I pop out the terminal to keep the history handy and navigate to a different replica's Terminal TAB. After a few copy pasta, I'm able to see that all tests are working good!

Let me destroy the previous POD and have k8s recreate it to see if that helps. Ohhh my previous poped out Terminal just closed. I guess that the shutdown is now completed.

[colemickens]

Random thoughts since this popped in my Inbox again:

I'm never just running ps ax randomly and thinking that's going to fix my problem or illuminate the real problem. Has anyone, ever, just run ps ax alone thinking that would solve their problem? I'd bet they're always running exec with bash to get into an interactive session.

I guess where I'm going with this is... it seems really awkward, to me, to have the input textbox _above_ the output area. In fact, why is there a separate box at all? hterm and the like usually present a UX that very closely resembles an actual terminal session... seems like we'd want the same here...

[colemickens]

(And then of course after sending this comment, I realize that bash is not available in all containers, and making such assumptions has it's own significant downsides...) I guess in this model typing bash or the like, somehow puts you into the active terminal session?

[ianlewis]

@colemickens I think we want to support both scenarios. I don't think anyone is saying ps aux will solve anything by itself, but just using it as an example. For instance, If one doesn't have bash available in the container, they might run the ps aux command and then other commands after that to follow up on what they learned leading to a UI that allows you run multiple commands easily. And then maybe the user wants to be able to see the output of the ps command after they run the next command (rather than clearing it from the terminal first) etc. etc.

[ianlewis]

@colemickens I think it could be possible to have a "exec into pod" prompt that is part of the terminal itself where you can type commands.

Something like this?

[bryk]

I guess where I'm going with this is... it seems really awkward, to me, to have the input textbox above the output area. In fact, why is there a separate box at all? hterm and the like usually present a UX that very closely resembles an actual terminal session... seems like we'd want the same here...

Yes, that was my first proposal to kick off the discussion. What we can do is to start a terminal-like window where you type exec commands. When you type something bogus, we tell you this and advice to run "bash".

PS. in the middle of me writing this post I noticed @IanLewis with the same idea... :) That's good sign.

[ianlewis]

For reference here is the Openshift UI. Openshift allows you to run a "debug" container with a modified entrypoint.

[ianlewis]

@bryk I'm curious how you would figure something is bogus or not. I think we can just try to run the command given and give an error like "command not found" as returned by the API or container. BTW AFAIK ls is an actual binary you can run on most systems.

Thoughts:

• I think some things like exit codes will be important.
• We can start the terminal with some "welcome" instructions on what kind of terminal it is and what to do.
• How can we help users differentiate easily between the exec prompt and a bash prompt? :-/
• We need to be careful about having a powerful terminal thing, and throwing users into it without enough guidance on how to use it.

[bryk]

For reference here is the Openshift UI. Openshift allows you to run a "debug" container with a modified entrypoint.

Yes, but they modify the pod, which I'd like us to avoid if possible.

I'm curious how you would figure something is bogus or not. I think we can just try to run the command given and give an error like "command not found" as returned by the API or container.

Yes, exactly. And then augment this error with info that one should typically run "bash" or "sh" to do interactive sessions.

BTW AFAIK ls is an actual binary you can run on most systems.

Yes, my example in the mock is unfortunate.

I think some things like exit codes will be important.

Yes, for exec commands.

We can start the terminal with some "welcome" instructions on what kind of terminal it is and what to do.

Awesome idea. We'd say there that one can exec or start a terminal (if it is installed in the container).

How can we help users differentiate easily between the exec prompt and a bash prompt? :-/

That's a hard problem... I look forward to any ideas here.

[ianlewis]

Yes, but they modify the pod, which I'd like us to avoid if possible.

Agreed.

Yes, exactly. And then augment this error with info that one should typically run "bash" or "sh" to do interactive sessions.

My thought is that we could just say this as part of the welcome message and be done with it.

[parisk]

Hello everyone, while I have not much of personal experience with kubernetes:

1. we are experimenting with kubernetes in our company in various ways
2. I am one of the core maintainers of xterm.js which was proposed for usage at https://github.com/kubernetes/dashboard/issues/1345#issuecomment-254679978

I just want to point out that we would be really happy to help moving this feature forward in the following ways:

1. help integrating xterm.js
2. perform user testing on our own use cases

I already subscribed to stay up to date with the progress and please feel free to mention, if there is anything you believe we could help.

[bryk]

@parisk That's cool. Can you provide us consulting and code reviews when integrating this? It'd help a lot :)

@IanLewis Do you need any help from the rest of the team to move this forward?

[ianlewis]

I think I'm ok. I'm close to having a prototype working. I hit some snags
related to the dev environment and supporting websockets but it's mostly
resolved (It was just easier to use the existing dev environ rather than
setting up something else).

In terms of xterm.js I don't have much of a problem yet though having the
main directive in xterm.js' bower.json would help.

I hope to have a couple prototypes with different libraries so others can
try them out but xterm is the easiest to integrate with websockets AFAICT.

On Fri, Nov 4, 2016, 22:05 Piotr Bryk [email protected] wrote:

@parisk https://github.com/parisk That's cool. Can you provide us
consulting and code reviews when integrating this? It'd help a lot :)

@IanLewis https://github.com/IanLewis Do you need any help from the
rest of the team to move this forward?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/kubernetes/dashboard/issues/1345#issuecomment-258481581,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AADAiaYkI3YsnqiBf7or3S6qxhBF-cIeks5q617GgaJpZM4KZcQ-
.

[ianlewis]

I thought I was close but the tooling just keeps getting in my way.

1. proxy-middleware doesn't support websockets so http-proxy-middleware has to be used instead. But that's mute because...
2. It looks like websockets + browsersync will just not work at all (https://github.com/chimurai/http-proxy-middleware/issues/15, https://github.com/BrowserSync/browser-sync/issues/780)
3. go-restful's Response doesn't implement the http.Hijacker interface so a new handler for websockets with an separate URL parsing library (like gorrilla/mux) will be needed.

I would rather not but I guess I could try and implement something based on long-polling?

For the prototype I am trying to make it work without the proxy by requiring that it be run by gulp serve:prod but that means I have to wrestle with the closure compiler, and making sure the 3rd party dependencies (xterm.js or hterm etc) get bundled properly.

So much yak shaving... Any pointers or insights into how I could save some time writing the prototype would be helpful...

[rf232]

For the prototype I am trying to make it work without the proxy by requiring that it be run by gulp serve:prod but that means I have to wrestle with the closure compiler, and making sure the 3rd party dependencies (xterm.js or hterm etc) get bundled properly.

I think this is the most important thing to see and get to work, since this is the actual usecase we want to support. The non :prod option is for developer experience, and while important the most important thing is that the code actually works on a 'real' installation :)

So much yak shaving... Any pointers or insights into how I could save some time writing the prototype would be helpful...

Basically what I said above, first make sure the feature will work, and focus on dev experience only when we have a working prototype in a :prod environment. We can even do that after we already merged the code in master.

Also feel free to open a [WIP] pull request where we can take a look at what you got so far and can perhaps help you out a bit.

[ianlewis]

Yah, the problem is that :prod has it's own annoyances which make a quick and dirty prototype loose the quick part since I have to either hack up the gulp build scripts to disable to closure compiler or make it happy by defining all the interfaces I happen to implement.
It's not always obvious how to make it happy either. :-/

[parisk]

@parisk That's cool. Can you provide us consulting and code reviews when integrating this? It'd help a lot :)

@bryk definitely. Let me know when the PR is open or just mention me in this issue to jump into the discussion.

I have to wrestle with the closure compiler, and making sure the 3rd party dependencies (xterm.js or hterm etc) get bundled properly

@IanLewis if anything breaks with compiling xterm.js with closure compiler, please open an upstream issue. We are happy to help.

[shamil]

I've used weave-scope console feature. With the kubernetes integration enabled, works really great. I can select pods and open terminal for its containers. Scope can automatically detect installed shell.
The terminal opens as modal popup. The reason I stopped using it is that it does more than just console to containers. I never needed those all features it has. So having console feature integrated in kubernetes dashboard will be really great!

Maybe it worth looking at theirs implementation...

[ianlewis]

It's not much but I created a PR for the simple prototype I'm working on -> #1455

It's still only an echo server that echos what you type back to the terminal on the frontend but I plan to get the backend part working soon. Currently struggling with how to create a remotecommand client since it seems to be different from the normal API client.

[ianlewis]

BTW The exec API is not usable through kubectl proxy due to it not being upgrade aware -> https://github.com/kubernetes/kubernetes/issues/32026

In order to make it work you currently will have to build the dashboard into a binary and run the binary with a local kubeconfig file for auth so that it connects to the API directly.

[bryk]

BTW The exec API is not usable through kubectl proxy due to it not being upgrade aware -> kubernetes/kubernetes#32026

That's sad.

In order to make it work you currently will have to build the dashboard into a binary and run the binary with a local kubeconfig file for auth so that it connects to the API directly.

How about we make gulp serve default to use kubeconfig file, as opposed to hard-coded kubeproxy URL? We already have support for this, so it is a matter of changing our build scripts. @rf232 @cheld Opinions?

[ianlewis]

How about we make gulp serve default to use kubeconfig file, as opposed to hard-coded kubeproxy URL? We already have support for this, so it is a matter of changing our build scripts. @rf232 @cheld Opinions?

I think we can leave these issues until a future time and just live with the inconvenience when doing work on the exec/attach feature until the issue is fixed in 1.6 (hopefully).

[bryk]

All right, SGTM

[ianlewis]

I've pushed a version that's working end-to-end to #1455. Currently it doesn't have a way to specify the command; it just uses "bash" and the command for testing purposes. I hope to resolve how to specify the command as part of UX discussions.

For now the terminal works (doesn't support color yet). As I mentioned though, it requires that you run gulp build and run the dashboard binary out of the dist/amd64 directory with a local kubeconfig for auth with the API because kubectl proxy doesn't work with websockets. I hope this can suffice as a prototype.

[ianlewis]

@bryk I think we can start UX discussions. I'm thinking of creating an Invision mock and having folks review it and add comments.

Any other ideas to get started? Maybe set up a meeting?

[bryk]

@IanLewis From what we've done in the past, the most effective strategy was to craft a rough design doc and follow it up with a meeting with eng + UX. The design doc served as the base for discussion.

[ianlewis]

Ok. I'll write a markdown design doc and create a PR or is there a better
medium for the design doc?

On Fri, Dec 2, 2016, 19:14 Piotr Bryk [e-mail] wrote:

@IanLewis https://github.com/IanLewis From what we've done in the past,
the most effective strategy was to craft a rough design doc and follow it
up with a meeting with eng + UX. The design doc served as the base for
discussion.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/kubernetes/dashboard/issues/1345#issuecomment-264420056,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AADAibj4cVWyCear4p3FJh5ACYgXwsl3ks5rD-9ygaJpZM4KZcQ-
.

[bryk]

Ok. I'll write a markdown design doc and create a PR or is there a better
medium for the design doc?

That's the best way.

[ianlewis]

PR with design doc is up at #1585. PTAL

[Dmitry1987]

Hi @IanLewis , this is an awesome feature, can't wait to see it included in stable release of K8s :)

just want to share, that a simple shell implemented in Rancher too, and maybe you'd like to have a look at their implementation, who knows maybe you can get some new idea :) .

in Rancher it's very basic, but works good and we use it for all troubleshooting needs (the scheduler we use is Kubernetes, but Rancher shows all containers on all hosts in a nice fashion so we use it too).

btw my 2 cents (as K8s user) for "why would people need that" discussion, is that ALL our developers in department like the quick Rancher terminal, for experiments/debugging, it's available on their Staging/Dev Rancher cluster web UI, and everyone thinks it's supercool :)
(I can't imagine them running "kubectl get pods ..." then "kubectl exec -it pod12345678 container123...." for a quick glance at some container internals, that'd be cruel from our DevOps side :D )

huge thanks to Ian for proposing this and working on it! (if I can help somehow with testing, let me know)

[danielromlein]

@IanLewis any progress updates on this?

[ianlewis]

It's been difficult to get time to work on dashboard due to work responsibilities/travel etc. Not likely to let up anytime soon either :-/ We might want to look at getting someone else to take a look at it since I'm solidly booked into late summer.

[lenartj]

The other day I too have started playing with this idea and I have created a separate prototype. I haven't looked at the code in #1455 yet. I think this can be finished with a couple of days work. I will post some code later.

[danielromlein]

How should we handle the existence of both this issue and #1939? Should we close this issue in favor of the other? I've added #1939 to the 2017 milestone.
/ @lenartj @floreks @maciaszczykm

[ianlewis]

1939 is a PR that potentially solves this issue. We can close the issue if/when the PR is merged.

Comments on the overall issue go here. #1939 will have comments related to the actual PR implementation/code.

[floreks]

Closing as #1939 was merged. Moving discussion to #2029.

[kachkaev]

Hi guys! I've just discovered _Exec into Pod_ feature, thanks to https://github.com/kubernetes/dashboard/pull/2155. Looks pretty useful!

After trying the new button for a few containers, I noticed that the shell does not launch and the error says the following:

rpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:247: starting container process caused "exec: \"bash\": executable file not found in $PATH"

/ # 

This occurs for the lightweight alpine-based images, which do not have bash by design. So I'm wondering if it is anyhow possible to annotate the containers within the pod so that the dashboard knew if it needs to use something else instead of just bash. It looks like the containers do not have annotations in the k8s API, but maybe there could be some environment variable that serves as a hint?

Shall I open a new issue to discuss the customisation of bash/shell path? What do you think of this?

[floreks]

We have actually a fallback mechanism that tries to access another supported shell if some is unavailable. It's first bash then sh. Only if all of them are unavailable then you won't be able to do anything.

[kachkaev]

Thanks for your comment @floreks – the fact that bash falls back to shell explains why I see / # in the web terminal after the error message. However, it is still not clear why shell does not enter the interactive mode and exits immediately:

Any ideas what could be causing this? My containers with this behaviour are based on the alpine image (e.g. nginx:alpine). The same web terminal for other containers (with bash installed) works fine.

[floreks]

This is a bigger problem with session handling. In fact if first error occurs dashboard crashes. This is a known issue and we are working on improving exec feature. Right now it was added only for testing purposes so we can catch all issues. It is not considered stable.

[kachkaev]

OK, I see. Just found another issue where you mention the problem I just described – will subscribed to it now. https://github.com/kubernetes/dashboard/issues/2029