Incubator-superset: Alembnic current isn't updated to the head after migration.

[bkyryliuk]

caravel db upgrade

INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade  -> 4e6a06bad7a8, Init
INFO  [alembic.runtime.migration] Running upgrade 4e6a06bad7a8 -> 5a7bad26f2a7, empty message
INFO  [alembic.runtime.migration] Running upgrade 5a7bad26f2a7 -> 1e2841a4128, empty message
INFO  [alembic.runtime.migration] Running upgrade 1e2841a4128 -> 2929af7925ed, TZ offsets in data sources
INFO  [alembic.runtime.migration] Running upgrade 2929af7925ed -> 289ce07647b, Add encrypted password field
INFO  [alembic.runtime.migration] Running upgrade 289ce07647b -> 1a48a5411020, adding slug to dash
INFO  [alembic.runtime.migration] Running upgrade 1a48a5411020 -> 315b3f4da9b0, adding log model
INFO  [alembic.runtime.migration] Running upgrade 315b3f4da9b0 -> 55179c7f25c7, sqla_descr
INFO  [alembic.runtime.migration] Running upgrade 55179c7f25c7 -> 12d55656cbca, is_featured
/usr/local/lib/python2.7/site-packages/alembic-0.8.6-py2.7.egg/alembic/util/messaging.py:69: UserWarning: Skipping unsupported ALTER for creation of implicit constraint
  warnings.warn(msg)
INFO  [alembic.runtime.migration] Running upgrade 12d55656cbca -> 2591d77e9831, user_id
INFO  [alembic.runtime.migration] Running upgrade 2591d77e9831 -> 8e80a26a31db, empty message
INFO  [alembic.runtime.migration] Running upgrade 8e80a26a31db -> 7dbf98566af7, empty message
INFO  [alembic.runtime.migration] Running upgrade 7dbf98566af7 -> 43df8de3a5f4, empty message
INFO  [alembic.runtime.migration] Running upgrade 43df8de3a5f4 -> d827694c7555, css templates
INFO  [alembic.runtime.migration] Running upgrade d827694c7555 -> 430039611635, log more
INFO  [alembic.runtime.migration] Running upgrade 430039611635 -> 18e88e1cc004, making audit nullable
INFO  [alembic.runtime.migration] Running upgrade 18e88e1cc004 -> 836c0bf75904, cache_timeouts
INFO  [alembic.runtime.migration] Running upgrade 18e88e1cc004 -> a2d606a761d9, adding favstar model
INFO  [alembic.runtime.migration] Running upgrade a2d606a761d9, 836c0bf75904 -> d2424a248d63, empty message
INFO  [alembic.runtime.migration] Running upgrade d2424a248d63 -> 763d4b211ec9, fixing audit fk
INFO  [alembic.runtime.migration] Running upgrade d2424a248d63 -> 1d2ddd543133, log dt
INFO  [alembic.runtime.migration] Running upgrade 1d2ddd543133, 763d4b211ec9 -> fee7b758c130, empty message
INFO  [alembic.runtime.migration] Running upgrade fee7b758c130 -> 867bf4f117f9, Adding extra field to Database model
INFO  [alembic.runtime.migration] Running upgrade 867bf4f117f9 -> bb51420eaf83, add schema to table model
INFO  [alembic.runtime.migration] Running upgrade bb51420eaf83 -> b4456560d4f3, change_table_unique_constraint
INFO  [alembic.runtime.migration] Running upgrade b4456560d4f3 -> 4fa88fe24e94, owners_many_to_many
INFO  [alembic.runtime.migration] Running upgrade 4fa88fe24e94 -> c3a8f8611885, Materializing permission
INFO  [alembic.runtime.migration] Running upgrade c3a8f8611885 -> f0fbf6129e13, Adding verbose_name to tablecolumn
INFO  [alembic.runtime.migration] Running upgrade f0fbf6129e13 -> 956a063c52b3, adjusting key length
INFO  [alembic.runtime.migration] Running upgrade 956a063c52b3 -> 1226819ee0e3, Fix wrong constraint on table columns
WARNI [root] Could not find or drop constraint on `columns`
INFO  [alembic.runtime.migration] Running upgrade 1226819ee0e3 -> d8bc074f7aad, Add new field 'is_restricted' to SqlMetric and DruidMetric
INFO  [alembic.runtime.migration] Running upgrade d8bc074f7aad -> 27ae655e4247, Make creator owners
INFO  [alembic.runtime.migration] Running upgrade 27ae655e4247 -> 960c69cb1f5b, add dttm_format related fields in table_columns
INFO  [alembic.runtime.migration] Running upgrade 960c69cb1f5b -> f162a1dea4c4, d3format_by_metric

caravel db current

INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
960c69cb1f5b

Current is 960c69cb1f5b, but expected: f162a1dea4c4

Migration is successful:

CREATE TABLE metrics (
    id INTEGER NOT NULL, 
    metric_name VARCHAR(512), 
    verbose_name VARCHAR(1024), 
    metric_type VARCHAR(32), 
    datasource_name VARCHAR(255), 
    json TEXT, 
    description TEXT, changed_by_fk INTEGER, changed_on DATETIME, created_by_fk INTEGER, created_on DATETIME, is_restricted BOOLEAN, **d3format** VARCHAR(128), 
    PRIMARY KEY (id), 
    FOREIGN KEY(datasource_name) REFERENCES datasources (datasource_name), 
    FOREIGN KEY(datasource_name) REFERENCES datasources (datasource_name)
);
CREATE TABLE sql_metrics (
    created_on DATETIME NOT NULL, 
    changed_on DATETIME NOT NULL, 
    id INTEGER NOT NULL, 
    metric_name VARCHAR(512), 
    verbose_name VARCHAR(1024), 
    metric_type VARCHAR(32), 
    table_id INTEGER, 
    expression TEXT, 
    description TEXT, 
    created_by_fk INTEGER, 
    changed_by_fk INTEGER, is_restricted BOOLEAN, **d3format** VARCHAR(128), 
    PRIMARY KEY (id), 
    FOREIGN KEY(table_id) REFERENCES tables (id), 
    FOREIGN KEY(created_by_fk) REFERENCES ab_user (id), 
    FOREIGN KEY(changed_by_fk) REFERENCES ab_user (id)
);

[bkyryliuk]

In addition: caravel db downgrade fails with the error message:
It is not possible to DROP the column in the SQLite DB:
http://stackoverflow.com/questions/8442147/how-to-delete-or-add-column-in-sqlite

INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
INFO  [alembic.runtime.migration] Running downgrade 960c69cb1f5b -> 27ae655e4247, add dttm_format related fields in table_columns

  File "build/bdist.macosx-10.10-x86_64/egg/sqlalchemy/engine/base.py", line 1146, in _execute_context
  File "build/bdist.macosx-10.10-x86_64/egg/sqlalchemy/engine/base.py", line 1341, in _handle_dbapi_exception
  File "build/bdist.macosx-10.10-x86_64/egg/sqlalchemy/util/compat.py", line 202, in raise_from_cause
  File "build/bdist.macosx-10.10-x86_64/egg/sqlalchemy/engine/base.py", line 1139, in _execute_context
  File "build/bdist.macosx-10.10-x86_64/egg/sqlalchemy/engine/default.py", line 450, in do_execute
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) near "DROP": syntax error [SQL: u'ALTER TABLE table_columns DROP COLUMN python_date_format']

[bkyryliuk]

caravel db stamp f162a1dea4c4 - is the workaround

[mistercrunch]

This appears to be coming from flask-migrate or alembic itself, some sort of off-by-one error when stamping the migration into the database.

[xrmx]

I think i've seen something similar, upgrading alembic from 0.8.6 to 0.8.7 fixed the issue for me on sqlite.

[niparis]

i had the same problem, with alembic 0.8.7 & sqlite.
workaround posted by bkyryliuk fixed it.

[bkyryliuk]

Thanks to @mistercrunch for the fix ! https://github.com/airbnb/caravel/pull/967 resolved it.

[TorokLev]

I suspect:

superset upgrade db

is the solution.