Moby: Docker Exec does not resize terminal

[thaJeztah]

I noticed this on both master (and docker 17.11-rc2, both on Ubuntu and Docker for Mac);

Client:
 Version:      17.11.0-ce-rc2
 API version:  1.34
 Go version:   go1.8.3
 Git commit:   d7062e5
 Built:        Wed Nov  1 22:11:59 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.11.0-ce-rc2
 API version:  1.34 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   d7062e5
 Built:        Wed Nov  1 22:10:29 2017
 OS/Arch:      linux/amd64
 Experimental: false

When running docker exec, the terminal doesn't resize:

$ docker run -d --name foo nginx:alpine
$ docker exec -it foo sh
/ # 0123456789012345678901234567890123456789012345678901234567890123456789012345
6789

Checking the size with stty size _does_ show the correct dimensions:

/ # stty size
93 410

This only happens on docker exec; doing docker run -it works correct:

$ docker run -it --rm nginx:alpine sh
/ # 01234567890123456789012345678901234567890123456789012345678901234567890123456789

(and shows the same output for stty size):

/ # stty size
93 410

Checking the daemon logs, the resize parameters are passed to the API for both docker run and docker exec;

time="2017-11-04T00:17:19.801321056Z" level=debug msg="Calling POST /v1.34/containers/378120448a6af3b289c12574445dde6f7a5a8f6fb09a311a46d1ffc735fab010/resize?h=93&w=410"


time="2017-11-04T00:16:37.697677138Z" level=debug msg="Calling POST /v1.34/exec/c57124724228e4af259281af94206ce9f02c64e472408198f665b342c1347f03/resize?h=93&w=410"

ping @vieux @mlaventure @crosbymichael

[thaJeztah]

I'm not consistently getting this; investigating a bit more, and checking if the resize is actually called;

watch 'curl -sL localhost:1338/v1/metrics | grep ResizePty'

[thaJeztah]

So, the metrics are not enabled by default. From discussion on slack;

in /var/run/docker/containerd/containerd.toml;

Add:

[metrics]
    address = "..."

(the code assume it'll be tcp socket)

After updating the config, kill containerd to have it restart and reread the config. Do _not_ restart docker, as it will override the changes.

[crosbymichael]

you are not running the original nginx container with a tty so execs are not going to have one either. it has to be setup when the container is created so a correct /dev/console and /dev/pty are created that later exec's fork off new ptys from

[thaJeztah]

Oh! That does make sense, and didn't think of that; let me try;

that does seem to work indeed; it's a bit confusing because the docker exec accepts a -t flag as well 🤔

[thaJeztah]

So, thanks! Looks like this is a PEBCAK - let me close

[thaJeztah]

Still not sure if this could be related to my setup, but there does appear to be a weird issue.

I'm testing a pull-request, so started an interactive development container;

$ make BIND_DIR=. binary shell

Inside the container, installing, no wrapping occurs;

root@5f5c4dc29878:/go/src/github.com/docker/docker# ds;kljads;lkajsd;lkadsj;lasdkjads;lkjasdl;kasdj;laksdjad;slkjads;lkadsj;aldksjad;lkadjs;ladksjad;lkjads;lakdja;dlkjads;lkajd;adlskjads;lkadjs;ladksjasd;lkjad;lkadsj;ladksjasd;lkjads;lkdasj;adlskjads;lkadj;ladkjads;lkadj;ladskjads;lkj^C

I then start the daemon; and output from the daemon also doesn't wrap:

root@5f5c4dc29878:/go/src/github.com/docker/docker# make install
root@5f5c4dc29878:/go/src/github.com/docker/docker# dockerd --debug
DEBU[2017-11-15T01:17:23.552063465Z] Listener created for HTTP on unix (/var/run/docker.sock)
INFO[2017-11-15T01:17:23.553092593Z] libcontainerd: started new docker-containerd process  pid=2915
INFO[0000] starting containerd                           module=containerd revision=2b8ed96d2a422bf205adcdfa05272dc12c81613b version=v1.0.0-beta.3
INFO[0000] changing OOM score to -500                    module=containerd
INFO[0000] loading plugin "io.containerd.content.v1.content"...  module=containerd type=io.containerd.content.v1
INFO[0000] loading plugin "io.containerd.snapshotter.v1.btrfs"...  module=containerd type=io.containerd.snapshotter.v1
WARN[0000] failed to load plugin io.containerd.snapshotter.v1.btrfs  error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter" module=containerd
INFO[0000] loading plugin "io.containerd.snapshotter.v1.overlayfs"...  module=containerd type=io.containerd.snapshotter.v1
INFO[0000] loading plugin "io.containerd.metadata.v1.bolt"...  module=containerd type=io.containerd.metadata.v1
WARN[0000] could not use snapshotter btrfs in metadata plugin  error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter" module="containerd/io.containerd.metadata.v1.bolt"
INFO[0000] loading plugin "io.containerd.differ.v1.walking"...  module=containerd type=io.containerd.differ.v1

In a different shell, verify that a TTY is enabled for the container:

$ docker inspect $(docker ps -q -n1) | grep Tty
            "Tty": true,

And open a second session (docker exec) into the container:

$ docker exec -it $(docker ps -q -n1) bash

Output produced by docker correctly fits to the size, but anything that I'm _typing_ is wrapped to 80-characters

root@5f5c4dc29878:/go/src/github.com/docker/docker# docker run --help

Usage:  docker run [OPTIONS] IMAGE [COMMAND] [ARG...]

Run a command in a new container

Options:
      --add-host list                  Add a custom host-to-IP mapping (host:ip)
  -a, --attach list                    Attach to STDIN, STDOUT or STDERR
      --blkio-weight uint16            Block IO (relative weight), between 10 and 1000, or 0 to disable (default 0)

# (snip)

      --volumes-from list              Mount volumes from the specified container(s)
  -w, --workdir string                 Working directory inside the container
terminal wraps878:/go/src/github.com/docker/docker# but when I start typing the

See the last line in the output above (I typed "but when I start typing the terminal wraps");

terminal wraps878:/go/src/github.com/docker/docker# but when I start typing the

[mpepping]

This indeed seems an issues since the early 17.11 releases. It's quite easy to reproduce:

• make sure your terminal is > 80x24 chars.
• use docker engine 17.11 (eg. 17.11.0-ce)
• start a container /w tty +interactive: docker run -ti --name shelltest ubuntu bash
• start top and verify it's in full screen/terminal
• open a new terminal and make sure your terminal is > 80x24 chars.
• run docker exec -ti shelltest bash
• start top and see it starts in a 80x24 boundary

No issue in =< 17.10.

[mpepping]

So, this still seems to be an issue. My workaround for now is setting the COLUMNS and LINES environment variables in the Exec.

docker exec -ti --env COLUMNS=`tput cols` --env LINES=`tput lines` shelltest bash

[thaJeztah]

@mempepping are you on Docker for Mac, or something else?

[mpepping]

@thaJeztah Same outcome on Docker for Mac and with Docker Engine on a Fedora host.

[thaJeztah]

Thanks! Was wondering, because others were reporting not seeing the issue, and only saw it when using Docker for Mac

[lgfausak]

I am on docker/mac, I am seeing term size issues. For me they started when i upgraded to 17.12 (i think i was on 17.09 before). I've tried setting LINES/COLUMNS/TERM in both the run and exec docker calls. When I exec in, and immediately do a stty size, I see 0 0 as the answer. If I do a reset, or if I resize my window, stty will then report the correct term size and I can carry on inside the docker exec.

[mpepping]

The suggestion of resizing the window is an interesting one. It makes it seem that the SIGWINCH signal propagates the terminal size correctly. A small test ..

Start a TTY via docker exec:

docker exec -ti shelltest bash
$ stty size
0 0

On the host, send a SIGWINCH to the docker exec process:

kill -SIGWINCH $(ps -ef | grep 'docker exec -ti shelltest' | grep -v grep | awk '{ print $2}')

Re-run stty in the same TTY as the first step:

$ stty size
68 212

It seems that an initial SIGWINCH on the start of the TTY is missed or isn't sent (correctly).

[mpepping]

Two issues that describe the same behavior: #10341, #33794

[brianbruggeman]

My office is on a mac; we all see the terminal issue using docker exec. Easy to test with tput lines, tput cols or stty size.

Work-arounds:

* A resize of the terminal fixes the issue.
* docker exec -it $container -e COLUMNS=`tput cols` -e LINES=`tput lines` /bin/bash -l -i

[mpepping]

The docker exec -ti performs a "Calling POST /v1.35/exec/${EXEC-ID}/resize" during it's start:

time="2018-01-12T21:15:41.725755283Z" level=debug msg="Calling GET /_ping"
time="2018-01-12T21:15:41.726438011Z" level=debug msg="Calling GET /v1.35/containers/youthful_fermi/json"
time="2018-01-12T21:15:41.728114278Z" level=debug msg="Calling POST /v1.35/containers/youthful_fermi/exec"
time="2018-01-12T21:15:41.728217094Z" level=debug msg="form data: {\"AttachStderr\":true,\"AttachStdin\":true,\"AttachStdout\":true,\"Cmd\":[\"bash\"],\"Detach\":false,\"DetachKeys\":\"\",\"Env\":null,\"Privileged\":false,\"Tty\":true,\"User\":\"\",\"WorkingDir\":\"\"}"
time="2018-01-12T21:15:41.736287790Z" level=debug msg="Calling POST /v1.35/exec/f1382509b171dfbd9c5d8c8fa2f1a6f3acf4309dfa491196ef98bd40f450c18f/start"
time="2018-01-12T21:15:41.736377670Z" level=debug msg="form data: {\"Detach\":false,\"Tty\":true}"
time="2018-01-12T21:15:41.738118385Z" level=debug msg="starting exec command f1382509b171dfbd9c5d8c8fa2f1a6f3acf4309dfa491196ef98bd40f450c18f in container 1f0da635b423808b3438e2d338fe11ed6049c428e2768451bc937f507dcde1ac"
time="2018-01-12T21:15:41.740337256Z" level=debug msg="Calling POST /v1.35/exec/f1382509b171dfbd9c5d8c8fa2f1a6f3acf4309dfa491196ef98bd40f450c18f/resize?h=61&w=212"
time="2018-01-12T21:15:41.739369481Z" level=debug msg="attach: stdin: begin"
time="2018-01-12T21:15:41.741206281Z" level=debug msg="attach: stdout: begin"
time="2018-01-12T21:15:41Z" level=debug msg="event published" module="containerd/tasks" ns=moby topic="/tasks/exec-started" type=containerd.events.TaskExecStarted 
time="2018-01-12T21:15:41.806260976Z" level=debug msg=event module=libcontainerd namespace=moby topic=/tasks/exec-started
time="2018-01-12T21:15:41.806261138Z" level=debug msg=event module=libcontainerd namespace=plugins.moby topic=/tasks/exec-started
time="2018-01-12T21:15:41.806302812Z" level=warning msg="unknown container" container=1f0da635b423808b3438e2d338fe11ed6049c428e2768451bc937f507dcde1ac module=libcontainerd namespace=plugins.moby

Although it seems that this POST happens somewhat too early, since stty size shows:

root@1f0da635b423:/# stty size
0 0

Now, if the exact same "Calling POST /v1.35/exec/${EXEC-ID}/resize" call is performed via curl, right after the TTY session is started, it corrects the stty size:

On the Docker host:

curl --unix-socket /var/run/docker.sock -H "Content-Type: application/json" -X POST \
"http:/v1.35/exec/f1382509b171dfbd9c5d8c8fa2f1a6f3acf4309dfa491196ef98bd40f450c18f/resize?h=61&w=212"

And in the container:

# stty size
61 212

This makes this tty issue seem to be a race condition, or timing issue, during the start of the exec. The correct "Calling POST /v1.35/exec/${EXEC-ID}/resize" call is performed during the start, but the STTY size is set incorrectly. Re-running the same call a somewhat later sets the correct STTY size.

[thaJeztah]

Yes, it looks like a race condition, and likely the initial resize is triggered to early; cli/command/container/tty.go#L48. Moving the resize to the end of that function (or adding another resize after monitoring is started; cli/command/container/exec.go#L156-L160) seems to help slightly, but it's still racy.

So, this may actually be the same issue as https://github.com/moby/moby/issues/31223

ping @mlaventure

[mlaventure]

Yes, sorry this is still in my list. Unfortunately, I still haven't gotten around setting up a proper working env for docker-ce (the mono-repo doesn't make it easy :sweat_smile:). I'll try to dedicate some time to it in the next couple weeks.

[mkurz]

Any updates here?
@mlaventure Three months since your comment, did you find time to fix this?

[mlaventure]

@mkurz yes and no. I did make a PR to fix it (https://github.com/moby/moby/pull/36147) but the actual solution doesn't have consensus. And I haven't had much time lately to work on things Docker unfortunately.

Anyone is welcome to carry on/address the comment on that PR :)

[grigoryosifov]

The SIGWINCH workaround, suggested by @mpepping is much better than setting COLUMNS and LINES envs because it also supports resizing of the terminal window, without the need to restart Docker exec every time your terminal size changes. However the issue was that it had to be manually run every time after docker exec, so here's my workaround:

If you use Makefile, you can add:

.PHONY: sh
sh:
#   Fix Docker terminal to resize properly:
#   https://github.com/moby/moby/issues/35407#issuecomment-357046159
    @./fix-docker-terminal.bash &
    @docker exec -it -u www-data container zsh

and create fix-docker-terminal.bash in the same directory:

#!/bin/bash
sleep 1
ps -ef | grep 'docker exec -it' | grep -v grep | awk '{ print $2}' | xargs kill -SIGWINCH

Do not forget to chmod +x fix-docker-terminal.bash.

If you don't use Makefile, you can go with this one line:
./fix-docker-terminal.bash & docker exec -it -u www-data container zsh

[mpepping]

Nice workaround, @grigoryosifov! Luckily it seems that de definitive fix for this issue is getting posture: https://github.com/moby/moby/pull/37172/

[thaJeztah]

Fixed on master through https://github.com/moby/moby/pull/37172

[mpepping]

@thaJeztah Thanks! This will probably make a lot of people happy! 👍

[uptycs-shivurh]

So, this still seems to be an issue. My workaround for now is setting the COLUMNS and LINES environment variables in the Exec.

docker exec -ti --env COLUMNS=`tput cols` --env LINES=`tput lines` shelltest bash

This helped me on mac -

$ tput lines
39

$ tput cols
178

$ docker exec -e COLUMNS="178" -e LINES="39" -it CONTAINER_NAME /bin/bash

[thaJeztah]

@uptycs-shivurh what version of docker are you on? I think this issue should be fixed on current versions of the docker cli

[a4orce84]

I am getting this issue again on a fresh docker installation:

pi@rpi-dev:~ $ docker run -ti hello-world
failed to resize tty, using default size

pi@rpi-dev:~ $ docker --version
Docker version 19.03.12, build 48a6621

This is a fresh docker installation on a raspberry pi (raspbian). Any help would be greatly appreciated. TIA.

[jacobdanovitch]

Also getting the exact same output as above with same version on a fresh install. Have tried on the pi itself and over SSH via Mac and Ubuntu.

[mkny]

Could it be a problem with raspberry pi zero w ?
I have two models (pi zero w and pi 3b).

On the 3b it runs normal, as expected.

When I try to run anything interactively on pi zero w, it fails with the message like @a4orce84 posted.

[thaJeztah]

The failed to resize tty, using default size error will be shown if resizing failed after several attempts, and is part of a follow-up fix for this issue (implemented in https://github.com/moby/moby/pull/37172). The original issue was that resizing did not happen due to a race-condition (https://github.com/moby/moby/pull/37172), and the later fix was to retry the resize a couple of times (in case the container is slow to start to accept resizes); https://github.com/docker/cli/blob/7f612bfca67119c61461b61dd37b998668937f07/cli/command/container/tty.go#L57-L58

With that retry, 5 attempts are made, with a 10ms pause in between. Perhaps the container is slower to start/handle the resize signal? Not sure if there's still a bug, but if you're consistently able to reproduce, have more details, and suspect there's a bug feel free to open a new ticket

[mkny]

@thaJeztah I don't think it is a bug related to docker at all.

Could it maybe be a configuration on the pi zero w?