Flask-migrate: How to manage migrations in Flask with Docker Compose?

[adnanmuttaleb]

I have a simple Flask service that rely on Postgres database. I know I have first to create the migration scripts locally using something like that:

flask db init 
flask db migrate

And in my web service Dockerfile I issue flask db upgrade when the container start using:

CMD flask db upgrade && flask run

My docker-compose.yaml is:

web:
    build:
      context: .
    expose:
      - "5000"
    depends_on:
      - db 

db:
  image: postgres
  volumes:
    - my_volume:/var/lib/postgresql/data
  ports:
    - "5432:5432"
  environment:
    POSTGRES_PASSWORD: 123456

The first two commands init and migrate need to be issued either:

• from host, but this is not possible because the database is running on a container.
• Within container (using exec), this also not possible because the container issue the upgrade command (when it start) which requires the migrations dir to exist.

How to overcome that? and is there any mechanism I should follow to to manage the migrations with docker?

Note: Flask-migrate docs state that migrations scripts need to be reviewed before being committed (so I think creation of migrations can not be automated):

The migration script needs to be reviewed and edited, as Alembic
currently does not detect every change you make to your models. In
particular, Alembic is currently unable to detect table name changes,
column name changes, or anonymously named constraints.

[miguelgrinberg]

I'm not sure I understand what you need. When you add a migration you are basically modifying the source code of your application, since you are adding a new migration script. Do you intend to do this from Docker? As in, mounting the source code as a Docker volume so that you can then insert a migration script from a container? I guess you need to explain better what you want.

[adnanmuttaleb]

I completely agree with you, migrations script are part of my source code, but because I am running all my services using docker compose, database instance is not reachable from my host (I could make the db service reachable but this is totally wrong I think, especially if you have more than one postgres instance are running, which is my case), so executing the migrate command (from host) will raise an error, something like:

sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) could not translate host name "<Database Container name>" to address: Name or service not known

[miguelgrinberg]

Well, then we agree that the change needs to happen in the host, correct?

You have two options:

1. Make a database available to the host so that you can work on your migrations. It does not need to be your containerized database, it can be a mirror of it that you just use for migrations.

2. Make the source code accessible to a container in which you run the migrate command as a mounted volume.

3. I assume your database files are in your host, correct? You can then start a separate database process in the host using a copy of the files that are mounted on your database container, then you have a local database that you can migrate without containers.

Option 1 requires some tricky database setup. Option 2 requires some tricky container setup. Option 3 is a bit esoteric and also probably tricky to set up.

These are all the options I can think of.

[adnanmuttaleb]

Thank you for you time, I really appreciate your help, Options one and three are conceptually clear for me, but for Now I do not know how to apply any of them, I think I need to do more search.

I never thought that schema modification could be that hard, and this problem seems to me as part of a common routine, so it is strange is not mentioned before on the web (I did a lot of search), thank you again.

[miguelgrinberg]

@adnanmuttaleb What makes it hard is your development environment choice of having everything locked inside containers. I think you need to make a distinction between development and production. On a development machine you should be a bit more open, like for example exporting the database port so that it can be seen by the host.

But even if you don't want to change anything and can't implement 1 and 3, option number 2 should give you what you want.