Python: `subprocess.check_call('env', env={'need.env.flag.with.dot': 'true'}, shell=True)` behaves incorrect in newer python:3.6.5 python:3.6.6 python:3.7.0

[Aaron2Ti]

On 3.6.4 the env variables are passed correctly to subprocess

↪ docker run -it --rm python:3.6.4
Python 3.6.4 (default, Mar 14 2018, 17:49:05)
[GCC 4.9.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess
>>> subprocess.check_call('env', env={'need.env.flag.with.dot': 'true'}, shell=True)
need.env.flag.with.dot=true     # <----  this is correct behavior to us
PWD=/
0
>>>

↪ docker run -it --rm python:3.6.4-alpine
Python 3.6.4 (default, Jan 10 2018, 05:26:33)
[GCC 5.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess
>>> subprocess.check_call('env', env={'need.env.flag.with.dot': 'true'}, shell=True)
SHLVL=1
need.env.flag.with.dot=true
PWD=/
0
>>>

↪ docker run -it --rm python:3.6.6-alpine
Python 3.6.6 (default, Aug 22 2018, 20:48:31)
[GCC 6.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess
>>> subprocess.check_call('env', env={'need.env.flag.with.dot': 'true'}, shell=True)
need.env.flag.with.dot=true
SHLVL=1
PWD=/
0
>>>

↪ docker run -it --rm python:3.7.0-alpine
Unable to find image 'python:3.7.0-alpine' locally
3.7.0-alpine: Pulling from library/python
Digest: sha256:2b0c0bfceea1bc00e87f03b8b1b8adda89dfe9e9cceb03fd1adfca3a511a5340
Status: Downloaded newer image for python:3.7.0-alpine
Python 3.7.0 (default, Aug 22 2018, 20:39:59)
[GCC 6.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess
>>> subprocess.check_call('env', env={'need.env.flag.with.dot': 'true'}, shell=True)
need.env.flag.with.dot=true
SHLVL=1
PWD=/
0
>>>

however for newer python:3.6.5 and upon, the behavior was wrong

↪ docker run -it --rm python:3.6.5
Python 3.6.5 (default, Jun 27 2018, 08:15:56)
[GCC 6.3.0 20170516] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess
>>> subprocess.check_call('env', env={'need.env.flag.with.dot': 'true'}, shell=True)
PWD=/
0
>>>

↪ docker run -it --rm python:3.7.0
Python 3.7.0 (default, Aug  4 2018, 02:33:39)
[GCC 6.3.0 20170516] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess
>>> subprocess.check_call('env', env={'need.env.flag.with.dot': 'true'}, shell=True)
PWD=/
0
>>>

↪ docker run -it --rm python:3.6.5-alpine
Python 3.6.5 (default, Jun  6 2018, 23:08:29)
[GCC 6.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess
>>> subprocess.check_call('env', env={'need.env.flag.with.dot': 'true'}, shell=True)
SHLVL=1
PWD=/
0
>>>

Regards

[tao12345666333]

The subprocess.check_call function run command with arguments. Wait for command to complete. If the return code was zero then return, otherwise raise CalledProcessError.

It's behaves only return zero or raise error.

If you want to know the output, you can use subprocess.check_output.

[Aaron2Ti]

what we want is not the outpout, we need passing env.flag.which.with.dot=True to the forked subprocess

and the python:3.7.0 omit those env variables silently when Popen the subprocess

[Aaron2Ti]

but python:3.7.0-alpine behaves correctly, so wondering if this is the docker image's problem

Regards

[Aaron2Ti]

found a fix

RUN apt-get update && apt-get install -y bash
RUN ln -nfs /bin/bash /bin/sh

[JayH5]

This is due to differences in shells due to differences in underlying operating systems. See here about the POSIX standards: environment variable names can have dots in them, but shells aren't required to support them.

The Debian Jessie /bin/sh supported (inherited) env vars with dots in the name:

❯ docker run --rm -it -e need.env.flag.with.dot=true debian:jessie-slim /bin/sh -c env
HOSTNAME=20a59998dab6
need.env.flag.with.dot=true
HOME=/root
TERM=xterm
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PWD=/

For whatever reason Stretch's /bin/sh does not:

❯ docker run --rm -it -e need.env.flag.with.dot=true debian:stretch-slim /bin/sh -c env
HOSTNAME=c60a8b9086ab
HOME=/root
TERM=xterm
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PWD=/

With Alpine Linux, it looks like it works in Alpine 3.8, but not 3.7.

In general it's a bad idea from a security perspective to use shell=True when using the subprocess module (the documentation details the security considerations). Also, if you want to use Bash you can do something like this after installing Bash:

subprocess.check_call(['/bin/bash', '-c', 'env'], env={'need.env.flag.with.dot': 'true'})

...I would not recommend anybody overwrite /bin/sh as suggested above.

I think this issue can be closed as it is not an issue with the Python image itself.

[tianon]

Excellent, detailed overview; thanks @JayH5 :heart:

[tao12345666333]

Apologies for the delay, here is my tests.

(Tao) ➜  /tmp docker run -it --name 37 --rm python:3.7.0
Python 3.7.0 (default, Aug  4 2018, 02:33:39)
[GCC 6.3.0 20170516] on linux
Type "help", "copyright", "credits" or "license" for more information.                                                
>>> import subprocess
>>> subprocess.check_call('env;sleep 60;', env={'need.env.flag.with.dot': 'true'}, shell=True)
PWD=/
0
>>> 

(Tao) ➜  /tmp sudo docker exec -it 37 /bin/bash
root@a22d79508015:/# ps -ef 
UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 02:31 pts/0    00:00:00 python3
root         8     1  0 02:32 pts/0    00:00:00 /bin/sh -c env;sleep 60;
root        10     8  0 02:32 pts/0    00:00:00 sleep 60
root        11     0  1 02:32 pts/1    00:00:00 /bin/bash
root        18    11  0 02:32 pts/1    00:00:00 ps -ef
root@a22d79508015:/# cat /proc/8/environ 
need.env.flag.with.dot=trueroot@a22d79508015:/# 
root@a22d79508015:/# cat /proc/10/environ 
PWD=/root@a22d79508015:/# 
root@a22d79508015:/# cat /proc/1/environ 
PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/binHOSTNAME=a22d79508015TERM=xtermLANG=C.UTF-8GPG_KEY=0D96DF4D4110E5C43FBFB17F2D347EA6AA65421DPYTHON_VERSION=3.7.0PYTHON_PIP_VERSION=18.0HOME=/rootroot@a22d79508015:/# 
root@a22d79508015:/# 

We can see the subprocess's process has set the environment variable correctly. The reason for the difference between process's environ and code's output is probably as @JayH5 said.

[liudonghua123]

@JayH5 ,Thanks 😃

ldh@ldh:~/docker/fc-api$ docker run --rm -it -e need.env.flag.with.dot=true debian:stretch-slim /bin/sh -c env
HOSTNAME=950457a5efb8
HOME=/root
TERM=xterm
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PWD=/
ldh@ldh:~/docker/fc-api$ docker run --rm -it -e need.env.flag.with.dot=true debian:stretch-slim /bin/bash -c env
HOSTNAME=926fd72c7ba4
PWD=/
HOME=/root
TERM=xterm
SHLVL=1
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
need.env.flag.with.dot=true
_=/usr/bin/env
ldh@ldh:~/docker/fc-api$