Synapse: Synapse Docker container doesn't start due to prometheus_client version 0.9.0

[anoadragon453]

Version v0.9.0 of the prometheus_client python module was just released, and now our docker containers fail with the following:

❯ docker run --name synapse \                                                                                                                                                                                        
    --mount type=volume,src=synapse-data,dst=/data \
    -p 8008:8008 \
    matrixdotorg/synapse:latest
ERROR:root:Needed prometheus_client>=0.4.0,<0.9.0, got prometheus-client==0.9.0
Missing Requirements: 'prometheus_client>=0.4.0,<0.9.0'
To install run:
    pip install --upgrade --force 'prometheus_client>=0.4.0,<0.9.0'
❯ 

We need to investigate why the docker container is installing a version of a module outside of what's specified in python_dependencies.py.

Looks like this is due to us installing the library separately without consulting the version requirements: https://github.com/matrix-org/synapse/blob/3dc1871219f845954a4b7d31fc06739831d67d2e/docker/Dockerfile#L34-L46

Currently anyone that builds a Synapse container will be affected, but our release containers built earlier should be fine.

[anoadragon453]

A quick fix here is to add the version constraints to prometheus-client in this list, but a longer term fix should somehow apply the version constraints to the items in this list.

[anoadragon453]

While we're in the area, we may want to make sure that all the dependencies here are actually still used.

[ddevault]

Did anyone consider, I dunno, addressing the breaking changes in Synapse and upgrading to the latest version of prometheus client?

[anoadragon453]

@ddevault this was brought up in the linked PR: https://github.com/matrix-org/synapse/pull/8767#issuecomment-728997738

[ddevault]

You should not institutionalize a habit of pinning your dependencies as a permanent solution. You should always be aiming to support the latest version of them, and ingrain this at a cultural level.

You're in for a world of hurt if you don't. When you're 6 versions behind on something, and suddenly it matters very much that you be on the latest version, but you have punted for long enough that you've ballooned your work 10x in the middle of a crisis.

[ddevault]

Personally, I would not have even merged a pull request like #8767 unless the submitter outlined a plan for upgrading and justified the temporary version pinning approach.

[callahad]

You should not institutionalize a habit of pinning your dependencies as a permanent solution. You should always be aiming to support the latest version of them, and ingrain this at a cultural level. You're in for a world of hurt if you don't.

Acknowledged and agreed; we will not treat this as a permanent solution. There's some debt to unravel here, but we'll get there.