Incubator-superset: DB migration 937d04c16b64 -> 7f2635b51f5d fails

Created on 26 Sep 2019  路  8Comments  路  Source: apache/incubator-superset

Superset upgrade from 0.29.0rc7 to 0.34.0 fails on superset db upgrade on migration mentioned above

Expected results

Success superset db upgrade

Actual results

Loaded your LOCAL configuration at [/usr/local/bin/superset_config.py]
2019-09-25 22:34:18,870:INFO:root:Configured event logger of type <class 'superset.utils.log.DBEventLogger'>
INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade 937d04c16b64 -> 7f2635b51f5d, update base columns
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1249, in _execute_context
    cursor, statement, parameters, context
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 552, in do_execute
    cursor.execute(statement, parameters)
sqlite3.OperationalError: table _alembic_tmp_columns already exists

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/bin/superset", line 31, in <module>
    cli()
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/flask/cli.py", line 586, in main
    return super(FlaskGroup, self).main(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/flask/cli.py", line 426, in decorator
    return __ctx.invoke(f, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/flask_migrate/cli.py", line 134, in upgrade
    _upgrade(directory, revision, sql, tag, x_arg)
  File "/usr/local/lib/python3.6/site-packages/flask_migrate/__init__.py", line 95, in wrapped
    f(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/flask_migrate/__init__.py", line 280, in upgrade
    command.upgrade(config, revision, sql=sql, tag=tag)
  File "/usr/local/lib/python3.6/site-packages/alembic/command.py", line 276, in upgrade
    script.run_env()
  File "/usr/local/lib/python3.6/site-packages/alembic/script/base.py", line 475, in run_env
    util.load_python_file(self.dir, "env.py")
  File "/usr/local/lib/python3.6/site-packages/alembic/util/pyfiles.py", line 90, in load_python_file
    module = load_module_py(module_id, path)
  File "/usr/local/lib/python3.6/site-packages/alembic/util/compat.py", line 156, in load_module_py
    spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/usr/local/lib/python3.6/site-packages/superset/migrations/env.py", line 119, in <module>
    run_migrations_online()
  File "/usr/local/lib/python3.6/site-packages/superset/migrations/env.py", line 111, in run_migrations_online
    context.run_migrations()
  File "<string>", line 8, in run_migrations
  File "/usr/local/lib/python3.6/site-packages/alembic/runtime/environment.py", line 839, in run_migrations
    self.get_context().run_migrations(**kw)
  File "/usr/local/lib/python3.6/site-packages/alembic/runtime/migration.py", line 361, in run_migrations
    step.migration_fn(**kw)
  File "/usr/local/lib/python3.6/site-packages/superset/migrations/versions/7f2635b51f5d_update_base_columns.py", line 73, in upgrade
    batch_op.alter_column("column_name", existing_type=String(255), nullable=False)
  File "/usr/local/lib/python3.6/contextlib.py", line 88, in __exit__
    next(self.gen)
  File "/usr/local/lib/python3.6/site-packages/alembic/operations/base.py", line 325, in batch_alter_table
    impl.flush()
  File "/usr/local/lib/python3.6/site-packages/alembic/operations/batch.py", line 102, in flush
    batch_impl._create(self.impl)
  File "/usr/local/lib/python3.6/site-packages/alembic/operations/batch.py", line 281, in _create
    op_impl.create_table(self.new_table)
  File "/usr/local/lib/python3.6/site-packages/alembic/ddl/impl.py", line 251, in create_table
    self._exec(schema.CreateTable(table))
  File "/usr/local/lib/python3.6/site-packages/alembic/ddl/impl.py", line 134, in _exec
    return conn.execute(construct, *multiparams, **params)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 988, in execute
    return meth(self, multiparams, params)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/sql/ddl.py", line 72, in _execute_on_connection
    return connection._execute_ddl(self, multiparams, params)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1050, in _execute_ddl
    compiled,
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1253, in _execute_context
    e, statement, parameters, cursor, context
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1473, in _handle_dbapi_exception
    util.raise_from_cause(sqlalchemy_exception, exc_info)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 398, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 152, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1249, in _execute_context
    cursor, statement, parameters, context
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 552, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) table _alembic_tmp_columns already exists
[SQL: 
CREATE TABLE _alembic_tmp_columns (
    created_on DATETIME NOT NULL, 
    changed_on DATETIME NOT NULL, 
    id INTEGER NOT NULL, 
    column_name VARCHAR(255) NOT NULL, 
    is_active BOOLEAN, 
    type VARCHAR(32), 
    groupby BOOLEAN, 
    filterable BOOLEAN, 
    description TEXT, 
    created_by_fk INTEGER, 
    changed_by_fk INTEGER, 
    dimension_spec_json TEXT, 
    verbose_name VARCHAR(1024), 
    datasource_id INTEGER, 
    PRIMARY KEY (id), 
    CHECK (is_active IN (0, 1)), 
    CHECK (groupby IN (0, 1)), 
    CHECK (filterable IN (0, 1)), 
    CONSTRAINT fk_columns_datasource_id_datasources FOREIGN KEY(datasource_id) REFERENCES datasources (id), 
    CONSTRAINT uq_columns_column_name UNIQUE (column_name, datasource_id), 
    CONSTRAINT fk_columns_created_by_fk_ab_user FOREIGN KEY(created_by_fk) REFERENCES ab_user (id), 
    CONSTRAINT fk_columns_changed_by_fk_ab_user FOREIGN KEY(changed_by_fk) REFERENCES ab_user (id)
)

]
(Background on this error at: http://sqlalche.me/e/e3q8)

Screenshots

If applicable, add screenshots to help explain your problem.

How to reproduce the bug

  1. Upgrade non-empty superset v0.29.0rc7 to 0.34.0
  2. Run superset db upgrade

Environment

(please complete the following information):

  • superset version: 0.34.0
  • python version: 3.6.8
  • node.js version: none
  • npm version: none

Checklist

  • [x] I have checked the superset logs for python stacktraces and included it here as text if there are any.
  • [x] I have reproduced the issue with at least the latest released version of superset.
  • [x] I have checked the issue tracker for the same issue and I haven't found one similar.
#bug
Source
picture dusatvoj

Most helpful comment

Issue-Label Bot is automatically applying the label #bug to this issue, with a confidence of 0.97. Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback!

Links: app homepage, dashboard and code for this bot.

picture issue-label-bot[bot] on 26 Sep 2019
❤1 👍1

All 8 comments

Issue-Label Bot is automatically applying the label #bug to this issue, with a confidence of 0.97. Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback!

Links: app homepage, dashboard and code for this bot.

issue-label-bot[bot] picture issue-label-bot[bot] on 26 Sep 2019
❤1 👍1

A few recommendations:

  • sqllite is not recommended for production, look into using mysql or postgres
  • I'm guessing you had errors on previous runs and this error message is from a subsequent run
  • always backup before running migrations, if you did you can retry getting the original error message on the original database that didn't fail mid-migration
  • try deleting _alembic_tmp_columns manually and re-run
mistercrunch picture mistercrunch on 26 Sep 2019

Okay, I will migrate from sqlite to postgres.
Yes, when I tried again on backup I got this error:

Loaded your LOCAL configuration at [/usr/local/bin/superset_config.py]
2019-09-26 07:59:20,411:INFO:root:Configured event logger of type <class 'superset.utils.log.DBEventLogger'>
INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade 55e910a74826 -> 4ce8df208545, empty message
scanning dashboard (1/5) >>>>
scanning dashboard (2/5) >>>>
scanning dashboard (3/5) >>>>
scanning dashboard (4/5) >>>>
scanning dashboard (5/5) >>>>
INFO  [alembic.runtime.migration] Running upgrade 4ce8df208545 -> 46f444d8b9b7, remove_coordinator_from_druid_cluster_model.py
INFO  [alembic.runtime.migration] Running upgrade 46f444d8b9b7 -> a61b40f9f57f, remove allow_run_sync
INFO  [alembic.runtime.migration] Running upgrade a61b40f9f57f -> 6c7537a6004a, models for email reports
INFO  [alembic.runtime.migration] Running upgrade 6c7537a6004a -> 3e1b21cd94a4, change_owner_to_m2m_relation_on_datasources.py
INFO  [alembic.runtime.migration] Running upgrade 6c7537a6004a -> cefabc8f7d38, Increase size of name column in ab_view_menu
INFO  [alembic.runtime.migration] Running upgrade 55e910a74826 -> 0b1f1ab473c0, Add extra column to Query
INFO  [alembic.runtime.migration] Running upgrade 0b1f1ab473c0, cefabc8f7d38, 3e1b21cd94a4 -> de021a1ca60d, empty message
INFO  [alembic.runtime.migration] Running upgrade de021a1ca60d -> fb13d49b72f9, better_filters
INFO  [alembic.runtime.migration] Running upgrade fb13d49b72f9 -> a33a03f16c4a, Add extra column to SavedQuery
INFO  [alembic.runtime.migration] Running upgrade 4451805bbaa1, 1d9e835a84f9 -> c829ff0b37d0, empty message
INFO  [alembic.runtime.migration] Running upgrade c829ff0b37d0 -> 7467e77870e4, remove_aggs
INFO  [alembic.runtime.migration] Running upgrade 7467e77870e4, de021a1ca60d -> fbd55e0f83eb, empty message
INFO  [alembic.runtime.migration] Running upgrade fbd55e0f83eb, fb13d49b72f9 -> 8b70aa3d0f87, empty message
INFO  [alembic.runtime.migration] Running upgrade 8b70aa3d0f87, a33a03f16c4a -> 18dc26817ad2, empty message
INFO  [alembic.runtime.migration] Running upgrade 18dc26817ad2 -> c617da68de7d, form nullable
INFO  [alembic.runtime.migration] Running upgrade c617da68de7d -> c82ee8a39623, Add implicit tags
INFO  [alembic.runtime.migration] Running upgrade 18dc26817ad2 -> e553e78e90c5, add_druid_auth_py.py
INFO  [alembic.runtime.migration] Running upgrade e553e78e90c5, c82ee8a39623 -> 45e7da7cfeba, empty message
INFO  [alembic.runtime.migration] Running upgrade 45e7da7cfeba -> 80aa3f04bc82, Add Parent ids in dashboard layout metadata
adding parents for dashboard layout, id = 1 (1/5) >>>>
adding parents for dashboard layout, id = 2 (2/5) >>>>
adding parents for dashboard layout, id = 3 (3/5) >>>>
adding parents for dashboard layout, id = 4 (4/5) >>>>
adding parents for dashboard layout, id = 5 (5/5) >>>>
INFO  [alembic.runtime.migration] Running upgrade 80aa3f04bc82 -> d94d33dbe938, form strip
INFO  [alembic.runtime.migration] Running upgrade d94d33dbe938 -> 937d04c16b64, update datasources
INFO  [alembic.runtime.migration] Running upgrade 937d04c16b64 -> 7f2635b51f5d, update base columns
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1249, in _execute_context
    cursor, statement, parameters, context
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 552, in do_execute
    cursor.execute(statement, parameters)
sqlite3.IntegrityError: NOT NULL constraint failed: _alembic_tmp_table_columns.column_name

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/bin/superset", line 31, in <module>
    cli()
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/flask/cli.py", line 586, in main
    return super(FlaskGroup, self).main(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/flask/cli.py", line 426, in decorator
    return __ctx.invoke(f, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/flask_migrate/cli.py", line 134, in upgrade
    _upgrade(directory, revision, sql, tag, x_arg)
  File "/usr/local/lib/python3.6/site-packages/flask_migrate/__init__.py", line 95, in wrapped
    f(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/flask_migrate/__init__.py", line 280, in upgrade
    command.upgrade(config, revision, sql=sql, tag=tag)
  File "/usr/local/lib/python3.6/site-packages/alembic/command.py", line 276, in upgrade
    script.run_env()
  File "/usr/local/lib/python3.6/site-packages/alembic/script/base.py", line 475, in run_env
    util.load_python_file(self.dir, "env.py")
  File "/usr/local/lib/python3.6/site-packages/alembic/util/pyfiles.py", line 90, in load_python_file
    module = load_module_py(module_id, path)
  File "/usr/local/lib/python3.6/site-packages/alembic/util/compat.py", line 156, in load_module_py
    spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/usr/local/lib/python3.6/site-packages/superset/migrations/env.py", line 119, in <module>
    run_migrations_online()
  File "/usr/local/lib/python3.6/site-packages/superset/migrations/env.py", line 111, in run_migrations_online
    context.run_migrations()
  File "<string>", line 8, in run_migrations
  File "/usr/local/lib/python3.6/site-packages/alembic/runtime/environment.py", line 839, in run_migrations
    self.get_context().run_migrations(**kw)
  File "/usr/local/lib/python3.6/site-packages/alembic/runtime/migration.py", line 361, in run_migrations
    step.migration_fn(**kw)
  File "/usr/local/lib/python3.6/site-packages/superset/migrations/versions/7f2635b51f5d_update_base_columns.py", line 86, in upgrade
    "column_name", existing_type=String(256), nullable=False, type_=String(255)
  File "/usr/local/lib/python3.6/contextlib.py", line 88, in __exit__
    next(self.gen)
  File "/usr/local/lib/python3.6/site-packages/alembic/operations/base.py", line 325, in batch_alter_table
    impl.flush()
  File "/usr/local/lib/python3.6/site-packages/alembic/operations/batch.py", line 102, in flush
    batch_impl._create(self.impl)
  File "/usr/local/lib/python3.6/site-packages/alembic/operations/batch.py", line 294, in _create
    for transfer in self.column_transfers.values()
  File "/usr/local/lib/python3.6/site-packages/alembic/ddl/impl.py", line 134, in _exec
    return conn.execute(construct, *multiparams, **params)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 988, in execute
    return meth(self, multiparams, params)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/sql/elements.py", line 287, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1107, in _execute_clauseelement
    distilled_params,
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1253, in _execute_context
    e, statement, parameters, cursor, context
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1473, in _handle_dbapi_exception
    util.raise_from_cause(sqlalchemy_exception, exc_info)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 398, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 152, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1249, in _execute_context
    cursor, statement, parameters, context
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 552, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.IntegrityError: (sqlite3.IntegrityError) NOT NULL constraint failed: _alembic_tmp_table_columns.column_name
[SQL: INSERT INTO _alembic_tmp_table_columns (created_on, changed_on, id, table_id, column_name, is_dttm, is_active, type, groupby, filterable, description, created_by_fk, changed_by_fk, expression, verbose_name, python_date_format, database_expression) SELECT table_columns.created_on, table_columns.changed_on, table_columns.id, table_columns.table_id, table_columns.column_name, table_columns.is_dttm, table_columns.is_active, table_columns.type, table_columns.groupby, table_columns.filterable, table_columns.description, table_columns.created_by_fk, table_columns.changed_by_fk, table_columns.expression, table_columns.verbose_name, table_columns.python_date_format, table_columns.database_expression 
FROM table_columns]
(Background on this error at: http://sqlalche.me/e/gkpj)

When I tried to delete _alembic_tmp_columns, it failed too.

dusatvoj picture dusatvoj on 26 Sep 2019

I've migrated SQLite into postgres on SuperSet v0.29.0rc7 but I have same problem with upgrade to v0.34.0

# superset db upgrade
Loaded your LOCAL configuration at [/usr/local/bin/superset_config.py]
2019-09-28 11:12:42,977:INFO:root:Configured event logger of type <class 'superset.utils.log.DBEventLogger'>
INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade 55e910a74826 -> 4ce8df208545, empty message
scanning dashboard (1/5) >>>>
scanning dashboard (2/5) >>>>
scanning dashboard (3/5) >>>>
scanning dashboard (4/5) >>>>
scanning dashboard (5/5) >>>>
INFO  [alembic.runtime.migration] Running upgrade 4ce8df208545 -> 46f444d8b9b7, remove_coordinator_from_druid_cluster_model.py
INFO  [alembic.runtime.migration] Running upgrade 46f444d8b9b7 -> a61b40f9f57f, remove allow_run_sync
INFO  [alembic.runtime.migration] Running upgrade a61b40f9f57f -> 6c7537a6004a, models for email reports
INFO  [alembic.runtime.migration] Running upgrade 6c7537a6004a -> 3e1b21cd94a4, change_owner_to_m2m_relation_on_datasources.py
INFO  [alembic.runtime.migration] Running upgrade 6c7537a6004a -> cefabc8f7d38, Increase size of name column in ab_view_menu
INFO  [alembic.runtime.migration] Running upgrade 55e910a74826 -> 0b1f1ab473c0, Add extra column to Query
INFO  [alembic.runtime.migration] Running upgrade 0b1f1ab473c0, cefabc8f7d38, 3e1b21cd94a4 -> de021a1ca60d, empty message
INFO  [alembic.runtime.migration] Running upgrade de021a1ca60d -> fb13d49b72f9, better_filters
INFO  [alembic.runtime.migration] Running upgrade fb13d49b72f9 -> a33a03f16c4a, Add extra column to SavedQuery
INFO  [alembic.runtime.migration] Running upgrade 4451805bbaa1, 1d9e835a84f9 -> c829ff0b37d0, empty message
INFO  [alembic.runtime.migration] Running upgrade c829ff0b37d0 -> 7467e77870e4, remove_aggs
INFO  [alembic.runtime.migration] Running upgrade 7467e77870e4, de021a1ca60d -> fbd55e0f83eb, empty message
INFO  [alembic.runtime.migration] Running upgrade fbd55e0f83eb, fb13d49b72f9 -> 8b70aa3d0f87, empty message
INFO  [alembic.runtime.migration] Running upgrade 8b70aa3d0f87, a33a03f16c4a -> 18dc26817ad2, empty message
INFO  [alembic.runtime.migration] Running upgrade 18dc26817ad2 -> c617da68de7d, form nullable
INFO  [alembic.runtime.migration] Running upgrade c617da68de7d -> c82ee8a39623, Add implicit tags
INFO  [alembic.runtime.migration] Running upgrade 18dc26817ad2 -> e553e78e90c5, add_druid_auth_py.py
INFO  [alembic.runtime.migration] Running upgrade e553e78e90c5, c82ee8a39623 -> 45e7da7cfeba, empty message
INFO  [alembic.runtime.migration] Running upgrade 45e7da7cfeba -> 80aa3f04bc82, Add Parent ids in dashboard layout metadata
adding parents for dashboard layout, id = 2 (1/5) >>>>
adding parents for dashboard layout, id = 1 (2/5) >>>>
adding parents for dashboard layout, id = 3 (3/5) >>>>
adding parents for dashboard layout, id = 4 (4/5) >>>>
adding parents for dashboard layout, id = 5 (5/5) >>>>
INFO  [alembic.runtime.migration] Running upgrade 80aa3f04bc82 -> d94d33dbe938, form strip
INFO  [alembic.runtime.migration] Running upgrade d94d33dbe938 -> 937d04c16b64, update datasources
INFO  [alembic.runtime.migration] Running upgrade 937d04c16b64 -> 7f2635b51f5d, update base columns
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1249, in _execute_context
    cursor, statement, parameters, context
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 552, in do_execute
    cursor.execute(statement, parameters)
psycopg2.errors.NotNullViolation: column "column_name" contains null values


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/bin/superset", line 31, in <module>
    cli()
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/flask/cli.py", line 586, in main
    return super(FlaskGroup, self).main(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/flask/cli.py", line 426, in decorator
    return __ctx.invoke(f, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/flask_migrate/cli.py", line 134, in upgrade
    _upgrade(directory, revision, sql, tag, x_arg)
  File "/usr/local/lib/python3.6/site-packages/flask_migrate/__init__.py", line 95, in wrapped
    f(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/flask_migrate/__init__.py", line 280, in upgrade
    command.upgrade(config, revision, sql=sql, tag=tag)
  File "/usr/local/lib/python3.6/site-packages/alembic/command.py", line 298, in upgrade
    script.run_env()
  File "/usr/local/lib/python3.6/site-packages/alembic/script/base.py", line 489, in run_env
    util.load_python_file(self.dir, "env.py")
  File "/usr/local/lib/python3.6/site-packages/alembic/util/pyfiles.py", line 98, in load_python_file
    module = load_module_py(module_id, path)
  File "/usr/local/lib/python3.6/site-packages/alembic/util/compat.py", line 173, in load_module_py
    spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/usr/local/lib/python3.6/site-packages/superset/migrations/env.py", line 119, in <module>
    run_migrations_online()
  File "/usr/local/lib/python3.6/site-packages/superset/migrations/env.py", line 111, in run_migrations_online
    context.run_migrations()
  File "<string>", line 8, in run_migrations
  File "/usr/local/lib/python3.6/site-packages/alembic/runtime/environment.py", line 846, in run_migrations
    self.get_context().run_migrations(**kw)
  File "/usr/local/lib/python3.6/site-packages/alembic/runtime/migration.py", line 518, in run_migrations
    step.migration_fn(**kw)
  File "/usr/local/lib/python3.6/site-packages/superset/migrations/versions/7f2635b51f5d_update_base_columns.py", line 86, in upgrade
    "column_name", existing_type=String(256), nullable=False, type_=String(255)
  File "/usr/local/lib/python3.6/contextlib.py", line 88, in __exit__
    next(self.gen)
  File "/usr/local/lib/python3.6/site-packages/alembic/operations/base.py", line 325, in batch_alter_table
    impl.flush()
  File "/usr/local/lib/python3.6/site-packages/alembic/operations/batch.py", line 79, in flush
    fn(*arg, **kw)
  File "/usr/local/lib/python3.6/site-packages/alembic/ddl/postgresql.py", line 140, in alter_column
    **kw
  File "/usr/local/lib/python3.6/site-packages/alembic/ddl/impl.py", line 172, in alter_column
    existing_comment=existing_comment,
  File "/usr/local/lib/python3.6/site-packages/alembic/ddl/impl.py", line 134, in _exec
    return conn.execute(construct, *multiparams, **params)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 988, in execute
    return meth(self, multiparams, params)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/sql/ddl.py", line 72, in _execute_on_connection
    return connection._execute_ddl(self, multiparams, params)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1050, in _execute_ddl
    compiled,
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1253, in _execute_context
    e, statement, parameters, cursor, context
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1473, in _handle_dbapi_exception
    util.raise_from_cause(sqlalchemy_exception, exc_info)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 398, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 152, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1249, in _execute_context
    cursor, statement, parameters, context
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 552, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.IntegrityError: (psycopg2.errors.NotNullViolation) column "column_name" contains null values

[SQL: ALTER TABLE table_columns ALTER COLUMN column_name SET NOT NULL]
(Background on this error at: http://sqlalche.me/e/gkpj)
dusatvoj picture dusatvoj on 28 Sep 2019

Hi, you probably have already reached this conclusion, but seems that table_columns.columns_name contains nulls, fix that first, then rerun the migration

dpgaspar picture dpgaspar on 30 Sep 2019
👍1

@dpgaspar Thank you, there were a row containing NOT NULL but just nothing (IDK how). It was the error.

dusatvoj picture dusatvoj on 30 Sep 2019

I'm having the same issue. It looks like a previous migration adds a null into column_name
I get this error if I make the field not nullabel before running upgrade

INFO  [alembic.runtime.migration] Running upgrade 18dc26817ad2 -> c617da68de7d, form nullable
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1244, in _execute_context
    cursor, statement, parameters, context
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 552, in do_execute
    cursor.execute(statement, parameters)
psycopg2.IntegrityError: null value in column "column_name" violates not-null constraint
DETAIL:  Failing row contains (2019-11-20 00:48:48.196223, 2019-11-20 00:48:48.196238, 742, 57, null, f, t, null, f, f, null, 7, 7, null, null, null, null).
thejuan picture thejuan on 10 Jan 2020

Steps to fix:
superset db upgrade 937d04c16b64
update row with id 742 to have a column name (made it unknown as not sure what it should be)
superset db upgrade

thejuan picture thejuan on 10 Jan 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

tmccartan picture tmccartan  路  3Comments
deity-bram picture deity-bram  路  3Comments
josephtyler picture josephtyler  路  3Comments
sashank picture sashank  路  3Comments
fly-high-bj picture fly-high-bj  路  3Comments