Synapse: docker ... generate results in permission denied

[ma-fl]

Description

Cannot generate initial configuration by running

docker run -it --rm --mount type=volume,src=synapse-data,dst=/data -e SYNAPSE_SERVER_NAME=myserver.invalid -e SYNAPSE_REPORT_STATS=no matrixdotorg/synapse:latest generate

Container running as UserID 0:0, ENV (or defaults) requests 991:991
Creating log config /data/myserver.invalid.log.config
Generating config file /data/homeserver.yaml
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/local/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.7/site-packages/synapse/app/homeserver.py", line 659, in <module>
    main()
  File "/usr/local/lib/python3.7/site-packages/synapse/app/homeserver.py", line 654, in main
    hs = setup(sys.argv[1:])
  File "/usr/local/lib/python3.7/site-packages/synapse/app/homeserver.py", line 333, in setup
    "Synapse Homeserver", config_options
  File "/usr/local/lib/python3.7/site-packages/synapse/config/_base.py", line 576, in load_or_generate_config
    with open(config_path, "w") as config_file:
PermissionError: [Errno 13] Permission denied: '/data/homeserver.yaml'

Steps to reproduce

docker run -it --rm --mount type=volume,src=synapse-data,dst=/data -e SYNAPSE_SERVER_NAME=myserver.invalid -e SYNAPSE_REPORT_STATS=no matrixdotorg/synapse:latest generate

Expectation was a config file is generated.

I believe the following patch might fix this (untested):
https://github.com/ma-fl/synapse/commit/70ed6630fbcf96aafde6724d5eec651d408831ea#diff-d3b843a551028930ec04a76a93e6efcb

As a quick workaround I just adjusted the ownership of /data in the volume and retried.

[richvdh]

well, it works ok here. Are you starting with a non-empty volume?

[ma-fl]

I am starting with an empty volume and pulling the latest image from dockerhub:

root@t490s ~ # docker image ls
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
root@t490s ~ # docker volume ls
DRIVER              VOLUME NAME
root@t490s ~ # docker run -it --rm --mount type=volume,src=synapse-data,dst=/data -e SYNAPSE_SERVER_NAME=myserver.invalid -e SYNAPSE_REPORT_STATS=no matrixdotorg/synapse:latest generate
Unable to find image 'matrixdotorg/synapse:latest' locally
latest: Pulling from matrixdotorg/synapse
89d9c30c1d48: Pull complete 
910c49c00810: Pull complete 
e573a2f39893: Pull complete 
a147d60101b9: Pull complete 
04550de40735: Pull complete 
0c11f24dcd85: Pull complete 
c6d1926fc15c: Pull complete 
d989f8d32303: Pull complete 
784f32c3eda1: Pull complete 
Digest: sha256:eabdabefe72487785427939ac3f55d37a20dad53c005496dbbac3f6ef248a3b0
Status: Downloaded newer image for matrixdotorg/synapse:latest
Container running as UserID 0:0, ENV (or defaults) requests 991:991
Creating log config /data/myserver.invalid.log.config
Generating config file /data/homeserver.yaml
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/local/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.7/site-packages/synapse/app/homeserver.py", line 659, in <module>
    main()
  File "/usr/local/lib/python3.7/site-packages/synapse/app/homeserver.py", line 654, in main
    hs = setup(sys.argv[1:])
  File "/usr/local/lib/python3.7/site-packages/synapse/app/homeserver.py", line 333, in setup
    "Synapse Homeserver", config_options
  File "/usr/local/lib/python3.7/site-packages/synapse/config/_base.py", line 576, in load_or_generate_config
    with open(config_path, "w") as config_file:
PermissionError: [Errno 13] Permission denied: '/data/homeserver.yaml'

Now the volume exists but still is owned by root. Manually fixing ownership:

root@t490s ~ # docker run -it --rm --mount type=volume,src=synapse-data,dst=/data -e SYNAPSE_SERVER_NAME=myserver.invalid -e SYNAPSE_REPORT_STATS=no --entrypoint /bin/sh matrixdotorg/synapse:latest        
/ # chown -R 991:991 /data
/ #                                                                                                                                   

And then rerunning works:

root@t490s ~ # docker run -it --rm --mount type=volume,src=synapse-data,dst=/data -e SYNAPSE_SERVER_NAME=myserver.invalid -e SYNAPSE_REPORT_STATS=no matrixdotorg/synapse:latest generate
Container running as UserID 0:0, ENV (or defaults) requests 991:991
Generating config file /data/homeserver.yaml
Generating signing key file /data/myserver.invalid.signing.key
A config file has been generated in '/data/homeserver.yaml' for server name 'myserver.invalid'. Please review this file and customise it to your needs.

Docker version, in case that is relevant:

root@t490s ~ # docker --version
Docker version 19.03.4-ce, build 9013bf583a

I think the issue is in /start.py in the container in the function run_generate_config:

# cat start.py | tail -n 89 | head -n 6
    if ownership is not None:
        args = ["su-exec", ownership] + args
        os.execv("/sbin/su-exec", args)

        # make sure that synapse has perms to write to the data dir.
        subprocess.check_output(["chown", ownership, data_dir])

It's first execve'ing synapse under user id 991 to write the config to /data. The chown is expected to be executed afterwards as can be seen above, however this is never reached, so /data is never going to be owned by 991.

[dan0v]

I ran into this same issue yesterday and was finally able to resolve it and get synapse up and running by using the migrate_config option instead of generate to get the initial config files. After editing the created config files, everything else went smoothly. (I used docker-compose from /contrib/docker)

[edgycoder]

I seem to have the same issues when trying to generate on CentOS 7

Container running as UserID 0:0, ENV (or defaults) requests 991:991
Generating config file /data/homeserver.yaml
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/local/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.7/site-packages/synapse/app/homeserver.py", line 659, in <module>
    main()
  File "/usr/local/lib/python3.7/site-packages/synapse/app/homeserver.py", line 654, in main
    hs = setup(sys.argv[1:])
  File "/usr/local/lib/python3.7/site-packages/synapse/app/homeserver.py", line 333, in setup
    "Synapse Homeserver", config_options
  File "/usr/local/lib/python3.7/site-packages/synapse/config/_base.py", line 576, in load_or_generate_config
    with open(config_path, "w") as config_file:
PermissionError: [Errno 13] Permission denied: '/data/homeserver.yaml'

[richvdh]

my apologies, it seems I was testing with an old copy of the image; this was introduced in synapse 1.4.0 (presumably by #5970)

[anoadragon453]

So it does indeed strike me as odd that one part of the PR would execute those actions in one order:

https://github.com/matrix-org/synapse/pull/5970/files#diff-d3b843a551028930ec04a76a93e6efcbR122-R123

And the next (which breaks) is executed in the reverse order:

https://github.com/matrix-org/synapse/pull/5970/files#diff-d3b843a551028930ec04a76a93e6efcbR172-R176

Before the PR things indeed ran in 1. check data dir perms 2. execute on them order:

https://github.com/matrix-org/synapse/pull/5970/files#diff-d3b843a551028930ec04a76a93e6efcbL152-L173

I've tested the patch locally and it works for me.

I've created a PR for this here: https://github.com/matrix-org/synapse/pull/6389

[anoadragon453]

The patch has been merged in https://github.com/matrix-org/synapse/pull/6389, thank you for providing it. Let us know if you experience any further problems by creating a new issue.