Flask-migrate: Can't detect models.py schema

[jeffjzx]

Hi, miguelgrinberg:
I'm trying out this library along with instruction, in the following style of code i have put all the code in one file:

from flask.ext.sqlalchemy import SQLAlchemy
from flask.ext.script import Manager
from flask.ext.migrate import Migrate, MigrateCommand

app = Flask(__name__, static_path='/static')
app.config['SQLALCHEMY_DATABASE_URI'] = 'sqlite:///data.db'
db = SQLAlchemy(app)
migrate = Migrate(app, db)
manager = Manager(app)
manager.add_command('db', MigrateCommand)

class User(db.Model):
        #code here

if __name__ == '__main__':
    manager.run()

And when i type "python app.py db init" and "python app.py db migrate", the migration script is generated just fine.

Then i delete .db file and "migrations" folder and want to try this in the following setting:
i put all the database model definition in the file "models.py" which is at the same level with "app.py"

from app import db, app

class User(db.Model):
        #code here

And in "app.py"

from flask.ext.sqlalchemy import SQLAlchemy
from flask.ext.script import Manager
from flask.ext.migrate import Migrate, MigrateCommand

import models

app = Flask(__name__, static_path='/static')
app.config['SQLALCHEMY_DATABASE_URI'] = 'sqlite:///data.db'
db = SQLAlchemy(app)
migrate = Migrate(app, db)
manager = Manager(app)
manager.add_command('db', MigrateCommand)


if __name__ == '__main__':
    manager.run()

And after initialization and "python app.py db migrate", the migration script shows:

# revision identifiers, used by Alembic.
revision = ******
down_revision = None

from alembic import op
import sqlalchemy as sa


def upgrade():
    ### commands auto generated by Alembic - please adjust! ###
    pass
    ### end Alembic commands ###


def downgrade():
    ### commands auto generated by Alembic - please adjust! ###
    pass
    ### end Alembic commands ###

Which means flask-migrate doesn't detect database model definition in this type of code layout.
I'm not sure why this is the case, definitely somewhere has been messed up, but i'm not sure where is it.

Thanks in advance

[miguelgrinberg]

Hi @jeffjzx. The problem is that models.py uses the db instance, so you need to import it after db has been created. Try this:

from flask.ext.sqlalchemy import SQLAlchemy
from flask.ext.script import Manager
from flask.ext.migrate import Migrate, MigrateCommand

app = Flask(__name__, static_path='/static')
app.config['SQLALCHEMY_DATABASE_URI'] = 'sqlite:///data.db'

db = SQLAlchemy(app)
import models     # <--- move this anywhere after db is instantiated

migrate = Migrate(app, db)
manager = Manager(app)
manager.add_command('db', MigrateCommand)


if __name__ == '__main__':
    manager.run()

I'm surprised you are not getting a circular reference issue with your code.

[jbaiter]

I have the exact same problem with the pattern, but even with the design from the last comment my generated migration script comes out empty:

from flask import Flask
from flask.ext.migrate import Migrate, MigrateCommand
from flask.ext.script import Manager
from flask.ext.sqlalchemy import SQLAlchemy

app = Flask(__name__)
app.config.from_object('myapp.webservice.config')
if 'MYAPP_SERVICE_SETTINGS' in os.environ:
    app.config.from_envvar('MYAPP_SERVICE_SETTINGS')
db = SQLAlchemy(app)
import myapp.webservice.models 
migrate = Migrate(app, db)
cli_manager = Manager(app)
cli_manager.add_command('db', MigrateCommand)


if __name__ == '__main__':
    cli_manager.run()

The generated migration script looks like this:

"""empty message

Revision ID: 7ba986fc62
Revises: None
Create Date: 2015-04-26 12:00:31.400071

"""

# revision identifiers, used by Alembic.
revision = '7ba986fc62'
down_revision = None

from alembic import op
import sqlalchemy as sa


def upgrade():
    ### commands auto generated by Alembic - please adjust! ###
    pass
    ### end Alembic commands ###


def downgrade():
    ### commands auto generated by Alembic - please adjust! ###
    pass
    ### end Alembic commands ###

[miguelgrinberg]

@jbaiter: any chance you have a database that already has the tables in it? If that's the case, then Alembic is not going to find any difference between your models and the database. Try dropping all the tables in your database.

[jbaiter]

@miguelgrinberg, no, the table was empty. I managed to solve it now, though, through some restructuring of my code.

The problem in my case was that the initialization code was located in the package's __init__.py. Due to app.config.from_object(...), which was loading a module located inside of the package, the code in __init__.py would be executed twice. The first time, all models were correctly identified and could be found in the metadata, however the second time they were not.

[llazzaro]

In my case I was not using flask-sqlalchemy, just plain alchemy. (My models were subclass of Base).
Solve it by using flask-alchemy (db.Model)

[beebek]

I used this to recursively import all models.

# manager.py

import importlib
import os

from app import manager

MODELS_DIRECTORY = "models"
EXCLUDE_FILES = ["__init__.py"]

def import_models():
    for dir_path, dir_names, file_names in os.walk(MODELS_DIRECTORY):
        for file_name in file_names:
            if file_name.endswith("py") and not file_name in EXCLUDE_FILES:
                file_path_wo_ext, _ = os.path.splitext((os.path.join(dir_path, file_name)))
                module_name = file_path_wo_ext.replace(os.sep, ".")
                importlib.import_module(module_name)


if __name__ == '__main__':
    import_models()
    manager.run()

[asif-mahmud]

@beebek , your import_models is little bit like the scan method of Pyramid's configurator. I was happily using Pyramid until i faced WebSockets.

[asif-mahmud]

I have modified your version slightly to use with large applications with blueprints and all. Putting the following code in a top level source and calling the scan_models from the application's __init__.py or entry point just after registering all blueprints and extensions should work.

MODEL_DIRS = [
    'models',
]
MODEL_EXCLUDE_FILES = [
    '__init__.py',
]

def scan_models():
    for dirpath, dirnames, filenames in os.walk('.'):
        head, tail = os.path.split(dirpath)
        if tail in MODEL_DIRS:
            # there should be models
            for filename in filenames:
                if filename.endswith('.py') and \
                        filename not in MODEL_EXCLUDE_FILES:
                    # lets import the module
                    filename_no_ext, _ = os.path.splitext(
                        os.path.join(
                            dirpath, filename
                        )
                    )
                    # remove first . character
                    filename_no_ext = filename_no_ext[2:]
                    module_path = filename_no_ext.replace(os.sep, '.')
                    importlib.import_module(module_path)

This also assumes that you have installed your application as development mode or actually installed it as distribution.

[mchensd]

Hi Miguel,

About the original issue with the circular import, after I create db with db=SQLAlchemy(app), I try from models import User, but I am still getting a circular import error. Any reason why?

[miguelgrinberg]

@mchensd can't really tell you without seeing the code and the stack trace of the error. Can you provide that?

[mchensd]

In my main file, app.py:

basedir = os.path.abspath(os.path.dirname(__file__))
app = Flask(__name__)
app.config['SECRET_KEY'] = os.environ.get("secret_key")
app.config['SQLALCHEMY_DATABASE_URI'] = 'sqlite:///' + os.path.join(basedir, 'data.sqlite')
app.config['SQLALCHEMY_TRACK_MODIFICATIONS'] = False
db = SQLAlchemy(app)
from models import User

In models.py:

from app import db
from flask_login import UserMixin

class User(UserMixin, db.Model):
    # code

When I try to perform migrations, I get this:

Traceback (most recent call last):
  File "app.py", line 17, in <module>
    from models import User
  File "/Users/chen.young/PycharmProjects/MyPoll/models.py", line 1, in <module>
    from app import db
  File "/Users/chen.young/PycharmProjects/MyPoll/app.py", line 17, in <module>
    from models import User
ImportError: cannot import name 'User'

Meaning there is still a circular import.

[miguelgrinberg]

@mchensd is app.py your main module? If it is, then this module is called __main__, not app. So the problem with the circular import is in a different place than what you would normally expect. The sequence of imports goes as follows:

• app is imported, but since it is the main module, it is stored as __main__ in the module list.
• __main__ imports models
• models imports app. Since app is not a known module (the main module is called __main__), Python imports a duplicate copy of that module and stores it as app
• the duplicate app imports models recursively, which fails.

The easy solution is to replace from app import db with from __main__ import db in models.py.

[mchensd]

It works now!

Kind of trivial, but in models.py, my ide is giving warning that __main__ is an unresolved reference. Any way to get rid of this?

[miguelgrinberg]

I think this is a bug in the IDE's linter, but it is a very common one. Do you have a way to add exceptions in your IDE? For example, flake8 and pylint both allow you to tell the linter to ignore certain errors by adding a comment on the offending line. Not sure what else you can do if the IDE does not have a bypass option.

You may also consider not having app.py as your main module. Add another module that imports app, and make that your main module, and then the problem will go away, because now the __main__ module is going to be this new dummy module and not app.

[mchensd]

Ok, thanks for the help!

[miguelgrinberg]

This issue will be automatically closed due to being inactive for more than six months. Seeing that I haven't responded to your last comment, it is quite possible that I have dropped the ball on this issue and I apologize about that. If that is the case, do not take the closing of the issue personally as it is an automated process doing it, just reopen it and I'll get back to you.

[vald-phoenix]

@beebek IMHO, this structure is more beautiful to solve a problem with imports of models, we avoide of the circle import and we follow PEP 8 as well.
E.g.:
app/__init__.py

# Hook up models.
from app import models

app/views.py

from flask import Blueprint, render_template

from app.models import User

app_bp = Blueprint('app', __name__)


@app_bp.route('/')
def home():
    users = User.query.all()
    return render_template('app/home.html', users=users)

app/models.py

from core import db


class User(db.Model):
    id = db.Column(db.Integer, primary_key=True)
    username = db.Column(db.String(64), index=True, unique=True)
    email = db.Column(db.String(120), index=True, unique=True)
    password_hash = db.Column(db.String(128))

    def __repr__(self):
        return '<User {}>'.format(self.username)

core/__init__.py

import os

from flask import abort, Flask, g
from flask_sqlalchemy import SQLAlchemy
from flask_migrate import Migrate

try:
    from core import local_settings as settings
except ImportError:
    from core import settings

__version__ = '1.0.0'


babel = Babel()
db = SQLAlchemy()
migration = Migrate()


def create_app() -> Flask:
    """
    Creates application with predefined settings that depends on
    environment variable of a system.
    """

    application = Flask(
        __name__,
        template_folder=settings.TEMPLATE_DIR,
        static_folder=settings.STATIC_DIR
    )

    # Load configuration.
    environment = os.environ.get('APP_ENV', 'dev')
    environments = {
        'dev': settings.Dev,
        'prod': settings.Prod
    }
    application.config.from_object(environments[environment])

    # Initialize third-party libs.
    babel.init_app(application)
    db.init_app(application)
    migration.init_app(application, db)

    # Register blueprints
    from app.views import app_bp

    application.register_blueprint(app_bp)

    return application