Compose: Error when using docker-compose exec command after updating to Docker Desktop 2.2.0.0 on macOS

[yeclense]

Description of the issue

Same error while executing docker-compose exec command no matter in with config or command to execute in container.

Traceback (most recent call last):
  File "docker-compose", line 6, in <module>
  File "compose/cli/main.py", line 72, in main
  File "compose/cli/main.py", line 128, in perform_command
  File "compose/cli/main.py", line 491, in exec_command
  File "compose/cli/main.py", line 1469, in call_docker
  File "subprocess.py", line 339, in call
  File "subprocess.py", line 800, in __init__
  File "subprocess.py", line 1462, in _execute_child
  File "os.py", line 810, in fsencode
TypeError: expected str, bytes or os.PathLike object, not NoneType
[24719] Failed to execute script docker-compose

Context information (for bug reports)

Output of docker-compose version

docker-compose version 1.25.2, build 698e2846
docker-py version: 4.1.0
CPython version: 3.7.5
OpenSSL version: OpenSSL 1.1.1d  10 Sep 2019

Output of docker version

Client: Docker Engine - Community
 Version:           19.03.5
 API version:       1.40
 Go version:        go1.12.12
 Git commit:        633a0ea
 Built:             Wed Nov 13 07:22:34 2019
 OS/Arch:           darwin/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.5
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.12
  Git commit:       633a0ea
  Built:            Wed Nov 13 07:29:19 2019
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.2.10
  GitCommit:        b34a5c8af56e510852c35414db4c1f4fa6172339
 runc:
  Version:          1.0.0-rc8+dev
  GitCommit:        3e425f80a8c931f88e6d94a8c831b9d5aa481657
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Observed result

Traceback (most recent call last):
  File "docker-compose", line 6, in <module>
  File "compose/cli/main.py", line 72, in main
  File "compose/cli/main.py", line 128, in perform_command
  File "compose/cli/main.py", line 491, in exec_command
  File "compose/cli/main.py", line 1469, in call_docker
  File "subprocess.py", line 339, in call
  File "subprocess.py", line 800, in __init__
  File "subprocess.py", line 1462, in _execute_child
  File "os.py", line 810, in fsencode
TypeError: expected str, bytes or os.PathLike object, not NoneType
[24719] Failed to execute script docker-compose

Expected result

Execute the desired command in the selected container and output the result

Stacktrace / full error message

Tried with any Python verson I could install via python installer or via brew. Result keeps being the exact same one.

Here a stacktrace I could get with more detailed info:

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.8/bin/docker-compose", line 8, in <module>
    sys.exit(main())
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/compose/cli/main.py", line 72, in main
    command()
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/compose/cli/main.py", line 128, in perform_command
    handler(command, command_options)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/compose/cli/main.py", line 489, in exec_command
    sys.exit(call_docker(
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/compose/cli/main.py", line 1469, in call_docker
    return subprocess.call(args, env=environment)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/subprocess.py", line 340, in call
    with Popen(*popenargs, **kwargs) as p:
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/subprocess.py", line 854, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/subprocess.py", line 1624, in _execute_child
    env_list.append(k + b'=' + os.fsencode(v))
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/os.py", line 804, in fsencode
    filename = fspath(filename)  # Does type-checking of `filename`.
TypeError: expected str, bytes or os.PathLike object, not NoneType

Additional information

OS version / distribution, docker-compose install method, etc.
macOS Catalina (10.15.2), Docker Descktop Community 2.2.0.0

[mat007]

Thanks, this looks like the same as #7170 ?

[danielmartincmp]

It throws a different error on different file/line, so I decided to open a new issue.

[ulyssessouza]

Thank you for reporting @danielmartincmp !
Could you please, attach the docker-compose.yml and .env (if existent) ?
Also add steps to reproduce the issue, please.

[yeclense]

I've just tried in another macbook with same versions and same projects and I cannot reproduce it.
In the failing one (work laptop) I get the same error on different projects with different docker-compose.yml files.

I won't share any private project compose files here due to sensitive data, but I'll try to find a public one where I can reproduce the error.

[yeclense]

Tried to reproduce the error in my working laptop with other project but couldn't. It seems that the docker-compose config is causing the issue.

Here the config causing the failure, I just modified it a bit to not expose sensitive data:
```networks:
backend:
driver: bridge
ipam:
config:
- gateway: 10.5.0.1
subnet: 10.5.0.0/16
frontend: {}
services:
app:
depends_on:
mariadb:
condition: service_started
environment:
COMPOSER_MEMORY_LIMIT: 3G
DEV_EMAIL: [email protected]
LARAVEL_ENV: local
PHINX_ENVIRONMENT: devel
PHP_IDE_CONFIG: serverName=adminix
QUAY_PASSWORD: password
QUAY_USER: username
STAGING_DB_IP: 1.2.3.4
VIRTUAL_HOST: some.list.of.domains.com
VR_DB_IP: 1.2.3.4
extra_hosts:
sudomain.domain.com: 1.2.3.4
image: quay.io/company/imagename:build
networks:
backend: {}
frontend:
aliases:
- alias.domain.com
restart: on-failure:5
volumes:
- /Some/path:/path:rw
blackfire:
depends_on:
app:
condition: service_started
environment:
SOME_ENV_VAR: some_value
image: blackfire/blackfire
networks:
backend: {}
ports:
- 8707:8707/tcp
dd_agent:
environment:
API_KEY: some_api_key
HOSTNAME: some.hostname
TAGS: some_tag
hostname: some.hostname
image: datadog/docker-dd-agent
networks:
backend: {}
privileged: true
volumes:
- /Some/path:/path:ro
- /sys/fs/cgroup:/host/sys/fs/cgroup:ro
- /var/run/docker.sock:/var/run/docker.sock:rw
eetkak:
depends_on:
dd_agent:
condition: service_started
environment:
SOME_ENV_VAR: some_value
image: quay.io/company/imagename:build
networks:
backend: {}
ports:
- 32:32/tcp
elasticsearch:
container_name: elasticsearch
environment:
ES_JAVA_OPTS: -Xms2g -Xmx2g
bootstrap.memory_lock: "true"
http.host: 0.0.0.0
transport.host: 127.0.0.1
xpack.security.enabled: "false"
image: docker.elastic.co/elasticsearch/elasticsearch:6.4.2
networks:
backend: {}
ports:
- 127.0.0.1:9200:9200/tcp
ulimits:
memlock:
hard: -1
soft: -1
nofile:
hard: 65536
soft: 65536
kibana:
environment:
SOME_ENV_VAR: some_value
image: williamyeh/dummy
networks:
backend: {}
ports:
- 5601:5601/tcp
logstash:
container_name: logstash
image: williamyeh/dummy
networks:
backend: {}
ports:
- 5514:5044/tcp
volumes:
- /Some/path:/path:rw
mariadb:
build:
context: /Some/path/mariadb
environment:
MYSQL_DATABASE: database
MYSQL_ROOT_PASSWORD: password
healthcheck:
interval: 5s
retries: 3
start_period: 45s
test:
- show databases
timeout: 2s
networks:
backend: {}
ports:
- 3307:3306/tcp
volumes:
- /Some/path:/path:rw
memcached:
image: memcached:1.4
networks:
backend: {}
mockcontainer:
image: quay.io/company/imagename:build
networks:
backend: {}
frontend:
aliases:
- dev.mockcontainer.com
ports:
- 8882:8082/tcp
proxy:
build:
context: /Some/path/proxy
networks:
frontend: {}
ports:
- 80:80/tcp
volumes:
- /var/run/docker.sock:/tmp/docker.sock:ro
redis:
build:
context: /Some/path/redis
networks:
backend: {}
ports:
- 6379:6379/tcp
requestbin:
build:
context: /Some/path/requestbin
environment:
REALM: prod
REDIS_URL: //redis:6379
links:
- redis
networks:
backend: {}
frontend: {}
ports:
- 8000:8000/tcp
selenium:
image: selenium/standalone-chrome:3.141.59-oxygen
networks:
frontend: {}
shm_size: 2g
syslog:
container_name: syslog
image: balabit/syslog-ng
networks:
backend: {}
ports:
- 514:514/tcp
- 601:601/tcp
volumes:
- /Some/path/syslog-ng/syslog-ng.conf:/etc/syslog-ng/syslog-ng.conf:rw
vr-nginx:
environment:
VIRTUAL_HOST: 'some.list.of.subdomains.com

    '
  VIRTUAL_PORT: '82'
image: nginx:latest
networks:
  backend: {}
  frontend:
    aliases:
    - subdomain.domain.com
ports:
- 82:82/tcp
volumes:
- /Some/path/nginx/siteconfig.conf:/etc/nginx/conf.d/default.conf:rw
- /Some/path/www:/www:rw

vr-php:
environment:
SOME_ENV_VAR: some_value
image: quay.io/company/imagename:build
networks:
backend: {}
frontend: {}
volumes:
- /Some/path/php/30-xdebug.ini:/usr/local/etc/php/conf.d/docker-php-ext-xdebug.ini:rw
- /Some/path/www:/www:rw
vr-redis:
build:
context: /Some/path/redis
networks:
backend: {}
ports:
- 6380:6379/tcp
wiremock:
image: quay.io/company/imagename:build
networks:
backend: {}
ports:
- 8885:8080/tcp
- 8886:8443/tcp
version: '2.3'
```

[Ovsyanka]

I ran into this bug too.

Arch Linux,
python 3.8,
docker-compose versions 1.25.1, 1.25.2, 1.25.3.

It doesn't appear in 1.25.0.

There is the same Traceback, just line numbers are a bit different :

Traceback

Traceback (most recent call last):
  File "/usr/bin/docker-compose", line 11, in <module>
    load_entry_point('docker-compose==1.25.2', 'console_scripts', 'docker-compose')()
  File "/usr/lib/python3.8/site-packages/compose/cli/main.py", line 72, in main
    command()
  File "/usr/lib/python3.8/site-packages/compose/cli/main.py", line 128, in perform_command
    handler(command, command_options)
  File "/usr/lib/python3.8/site-packages/compose/cli/main.py", line 489, in exec_command
    sys.exit(call_docker(
  File "/usr/lib/python3.8/site-packages/compose/cli/main.py", line 1469, in call_docker
    return subprocess.call(args, env=environment)
  File "/usr/lib/python3.8/subprocess.py", line 340, in call
    with Popen(*popenargs, **kwargs) as p:
  File "/usr/lib/python3.8/subprocess.py", line 854, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib/python3.8/subprocess.py", line 1624, in _execute_child
    env_list.append(k + b'=' + os.fsencode(v))
  File "/usr/lib/python3.8/os.py", line 804, in fsencode
    filename = fspath(filename)  # Does type-checking of `filename`.
TypeError: expected str, bytes or os.PathLike object, not NoneType

[Ovsyanka]

It seems like I've found what caused the error in my case. It was the \ symbol at the end of the .env file. I put it there by mistake.

@ulyssessouza I assume that it would be nice if the user would get clear message about such type of errors in the .env file. It took a lot of time to figure out what cause a problem =(

@yeclense you didn't provide your .env file. Don't you have one? If you do - could you check it for a similar mistake?
And I noticed something odd in your .yml. What is that new line in the VIRTUAL_HOST value?

   vr-nginx:
    environment:
      VIRTUAL_HOST: 'some.list.of.subdomains.com

        '

[hrayr-artunyan]

I had the same error message as @yeclense and after looking at my .env file thanks to @Ovsyanka I discovered I had env variables without their values set that were also missing the = at the end

TEST

vs

TEST=

Adding = or removing the variables solved the problem for me.

Thanks!

[yeclense]

Hi, quite useful feedbacks the last ones.
@Ovsyanka I removed that new line, or at least reformatted that to not show that empty line, but still got same result.

@hrayr-artunyan your message made me search in our project and yes, we have a .env file in the root folder of the repo where the docker-compose is launched (it consist on a bunch of repos). This file is totally unrelated to docker-compose but to the config of the project itself (it sets the Symfony environment to work with)

So after removing this file or setting there a = at the end of the string value, docker-compose exec runs properly without errors.

I would say that there should be some kind of checking on this so that we can still work as we where doing before the last update, which means not forcing to have a specific format on this .env on the path where docker-compose is run, or at least, ignore whatever wrong format is set in there (instead of just fail)>
But not on me to decide.

Thanks guys for the help!

[derwentx]

Was able to reproduce this with this minimalist file structure. https://github.com/derwentx/docker-compose-test

The issues is from having the following lines under your service in docker-compose:

    env_file: .env
    entrypoint: []

and an .env file with ANY envs that don't have a value assigned e.g.:

ENV

where you would usually expect

ENV=FOO

I think there should be clearer warnings about this, since a beginner user would have no idea that this is because of an empty env file.

[flemay]

We face the same issue and, although having values for the keys fixes the issue, there is one scenario that has not been covered: passing environment variables that are already set from the host to a container.

For this scenario, let's have the following .env file:

MESSAGE

Running the command below will pass the value of the environment variable from the host to the container.

MESSAGE="HELLO, WORLD" docker run --rm --env-file .env alpine sh -c 'echo ${MESSAGE}'

This scenario is very useful for CI/CD server where the environment variables are set and we can provide a .env file with keys-only.

Before the latest version of Compose, passing .env file worked perfectly. Environment variables inside the docker-compose.yml could also be passed the same way.

services:
  alpine:
    image: alpine
    env_file: .env
    environment:
      - MESSAGE_2=${MESSAGE}

Now, this fails because for some reasons, Compose does not accept keys-only .env file anymore.

Passing a different file like env_file: .env.local with keys-only to a container works without problem. However, MESSAGE_2=${MESSAGE} does not work because .env has key without value. This use case worked before.

Is this an intended breaking change? or simply a bug?

[dikderoy]

I have same use-case as @flemay , we use "key-only" env files heavily to define sets of variables which should be included from host env to containers with like 40 keys where. really helps for local development setups. this case is broken now, will it work same if we just provide KEY= in all of them? huge backwards-compatibility issue =(

and it only breaks on exec and run, rest of the commands work just fine..

UPD: it does not work the same if we prodive KEY=, -> exec and run work, but actual values gathered by docker-compose to be set in container's environment are not overriden by ones from .env or host

[flemay]

@dikderoy Yes, KEY= is the same as KEY="" and therefore takes precedence over values set from the host.

[ulyssessouza]

Thank you for the feedback! I'm debugging it now.

[flemay]

Thank you so much @ulyssessouza !

[ulyssessouza]

https://github.com/docker/compose/releases/tag/1.25.4-rc2 Should include the fix.

[flemay]

I've quickly tested the Compose version 1.25.4-rc2 and it works the same way as before. i.e: a key-only .env file passes the values of environment variables to the container.

Well done, @ulyssessouza !

[dericlima]

The issue is closed but I'm leaving this comment in case someone ends up with the same issue like me.

On my case I had a colon (don't ask me how) instead of a equal sign in my .env for one of the variables:

ENV:value, so I just updated to ENV=value and everything was working again.

[radiantone]

Yeah, my compose file no longer works because of this error. I don't even have a .env anywhere.

docker-compose version 1.25.5, build unknown

This sucks

I'm on Ubuntu 20.04 btw

[radiantone]

Traceback (most recent call last):
  File "/usr/bin/docker-compose", line 11, in <module>
    load_entry_point('docker-compose==1.25.0', 'console_scripts', 'docker-compose')()
  File "/usr/lib/python3/dist-packages/compose/cli/main.py", line 72, in main
    command()
  File "/usr/lib/python3/dist-packages/compose/cli/main.py", line 125, in perform_command
    project = project_from_options('.', options)
  File "/usr/lib/python3/dist-packages/compose/cli/command.py", line 53, in project_from_options
    return get_project(
  File "/usr/lib/python3/dist-packages/compose/cli/command.py", line 156, in get_project
    execution_context_labels(config_details, environment_file),
  File "/usr/lib/python3/dist-packages/compose/cli/command.py", line 163, in execution_context_labels
    '{0}={1}'.format(LABEL_CONFIG_FILES, config_files_label(config_details)),
  File "/usr/lib/python3/dist-packages/compose/cli/command.py", line 172, in config_files_label
    return ",".join(
  File "/usr/lib/python3/dist-packages/compose/cli/command.py", line 173, in <genexpr>
    map(str, (os.path.normpath(c.filename) for c in config_details.config_files)))
  File "/usr/lib/python3.8/posixpath.py", line 336, in normpath
    path = os.fspath(path)
TypeError: expected str, bytes or os.PathLike object, not NoneType

[ilgala]

I had the same error message as @yeclense and after looking at my .env file thanks to @Ovsyanka I discovered I had env variables without their values set that were also missing the = at the end

TEST

vs

TEST=

Adding = or removing the variables solved the problem for me.

Thanks!

Had the same error on Ubuntu 20.04 LTS with:

• Docker version 19.03.11, build 42e35e61f3
• docker-compose version 1.25.0, build 0a186604
• Python 3.8.2

The problem was a missing = on .env file.

[flemay]

@ilgala Adding = works as long as you don't need the environment variable to be set from the host which I described here. Version 1.25.5, build 8a1c60f6 fixed it.