Docker-airflow: Unable to use RBAC

[siddardha7]

changed Postgres server details in entrypoint.sh to an RDS in AWS
removed postgres service from LocalExecutor .yml and used that to bring up webserver
all tables are getting created in postgres backend and UI is asking for login

Once I run airflow create_user -r Admin -u test -e [email protected] -f test -l test -p test log is showing using SequentialExecutor instead of LocalExecutor

airflow@ac85221fed9f:~$ airflow create_user -r Admin -u test -e [email protected] -f test -l test -p test
[2019-04-10 07:21:44,605] {{__init__.py:51}} INFO - Using executor SequentialExecutor
[2019-04-10 07:21:44,847] {{cli_action_loggers.py:69}} ERROR - Failed on pre-execution callback using
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1236, in _execute_context
cursor, statement, parameters, context
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 536, in do_execute
cursor.execute(statement, parameters)
sqlite3.OperationalError: no such table: log

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/airflow/utils/cli_action_loggers.py", line 67, in on_pre_execution
cb(**kwargs)
File "/usr/local/lib/python3.6/site-packages/airflow/utils/cli_action_loggers.py", line 98, in default_action_log
session.add(log)
File "/usr/local/lib/python3.6/contextlib.py", line 88, in __exit__
next(self.gen)
File "/usr/local/lib/python3.6/site-packages/airflow/utils/db.py", line 44, in create_session
session.commit()
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 1023, in commit
self.transaction.commit()
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 487, in commit
self._prepare_impl()
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 466, in _prepare_impl
self.session.flush()
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 2446, in flush
self._flush(objects)
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 2584, in _flush
transaction.rollback(_capture_exception=True)
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 67, in __exit__
compat.reraise(exc_type, exc_value, exc_tb)
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 277, in reraise
raise value
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 2544, in _flush
flush_context.execute()
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/unitofwork.py", line 416, in execute
rec.execute(self)
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/unitofwork.py", line 583, in execute
uow,
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj
insert,
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/persistence.py", line 1116, in _emit_insert_statements
statement, params
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 980, in execute
return meth(self, multiparams, params)
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/sql/elements.py", line 273, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1099, in _execute_clauseelement
distilled_params,
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1240, in _execute_context
e, statement, parameters, cursor, context
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1458, in _handle_dbapi_exception
util.raise_from_cause(sqlalchemy_exception, exc_info)
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 296, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 276, in reraise
raise value.with_traceback(tb)
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1236, in _execute_context
cursor, statement, parameters, context
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 536, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) no such table: log [SQL: 'INSERT INTO log (dttm, dag_id, task_id, event, execution_date, owner, extra) VALUES (?, ?, ?, ?, ?, ?, ?)'] [parameters: ('2019-04-10 07:21:44.843311', None, None, 'cli_create_user', None, 'airflow', '{"host_name": "ac85221fed9f", "full_command": "[\'/usr/local/bin/airflow\', \'create_user\', \'-r\', \'Admin\', \'-u\', \'test\', \'-e\', \'[email protected]\', \'-f\', \'test\', \'-l\', \'test\', \'-p\', \'test\']"}')] (Background on this error at: http://sqlalche.me/e/e3q8)
[2019-04-10 07:21:44,984] {{manager.py:525}} WARNING - No user yet created, use fabmanager command to do it.
[2019-04-10 07:21:46,055] {{models.py:273}} INFO - Filling up the DagBag from /usr/local/airflow/dags
[2019-04-10 07:21:48,279] {{security.py:437}} INFO - Start syncing user roles.
[2019-04-10 07:21:48,331] {{security.py:186}} INFO - Initializing permissions for role:Viewer in the database.
[2019-04-10 07:21:48,504] {{security.py:186}} INFO - Initializing permissions for role:User in the database.
[2019-04-10 07:21:48,680] {{security.py:186}} INFO - Initializing permissions for role:Op in the database.
[2019-04-10 07:21:48,809] {{security.py:346}} INFO - Fetching a set of all permission, view_menu from FAB meta-table
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1236, in _execute_context
cursor, statement, parameters, context
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 536, in do_execute
cursor.execute(statement, parameters)
sqlite3.OperationalError: no such table: dag

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/usr/local/bin/airflow", line 32, in
args.func(args)
File "/usr/local/lib/python3.6/site-packages/airflow/utils/cli.py", line 74, in wrapper
return f(args, *kwargs)
File "/usr/local/lib/python3.6/site-packages/airflow/bin/cli.py", line 1341, in create_user
appbuilder = cached_appbuilder()
File "/usr/local/lib/python3.6/site-packages/airflow/www_rbac/app.py", line 214, in cached_appbuilder
cached_app(config, testing)
File "/usr/local/lib/python3.6/site-packages/airflow/www_rbac/app.py", line 207, in cached_app
app, _ = create_app(config, session, testing)
File "/usr/local/lib/python3.6/site-packages/airflow/www_rbac/app.py", line 167, in create_app
security_manager.sync_roles()
File "/usr/local/lib/python3.6/site-packages/airflow/www_rbac/security.py", line 445, in sync_roles
self.create_custom_dag_permission_view()
File "/usr/local/lib/python3.6/site-packages/airflow/utils/db.py", line 73, in wrapper
return func(args, *kwargs)
File "/usr/local/lib/python3.6/site-packages/airflow/www_rbac/security.py", line 366, in create_custom_dag_permission_view
.filter(~models.DagModel.is_subdag).all()
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 2925, in all
return list(self)
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3081, in __iter__
return self._execute_and_instances(context)
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3106, in _execute_and_instances
result = conn.execute(querycontext.statement, self._params)
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 980, in execute
return meth(self, multiparams, params)
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/sql/elements.py", line 273, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1099, in _execute_clauseelement
distilled_params,
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1240, in _execute_context
e, statement, parameters, cursor, context
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1458, in _handle_dbapi_exception
util.raise_from_cause(sqlalchemy_exception, exc_info)
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 296, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 276, in reraise
raise value.with_traceback(tb)
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1236, in _execute_context
cursor, statement, parameters, context
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 536, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) no such table: dag [SQL: 'SELECT dag.dag_id AS dag_dag_id, dag.is_paused AS dag_is_paused, dag.is_subdag AS dag_is_subdag, dag.is_active AS dag_is_active, dag.last_scheduler_run AS dag_last_scheduler_run, dag.last_pickled AS dag_last_pickled, dag.last_expired AS dag_last_expired, dag.scheduler_lock AS dag_scheduler_lock, dag.pickle_id AS dag_pickle_id, dag.fileloc AS dag_fileloc, dag.owners AS dag_owners \nFROM dag \nWHERE (dag.is_active = 1 OR dag.is_paused = 1) AND dag.is_subdag = 0'] (Background on this error at: http://sqlalche.me/e/e3q8)

[siddardha7]

FIXED
to run airflow create_user or any other airflow command, open bash as
docker exec -it /entrypoint.sh bash

[r96941046]

@siddardha7 saved my day, thank you! For anyone running airflow helm chart on k8s, run airflow create_user after opening bash as kubectl exec -it <pod_name> /entrypoint.sh bash