Redash: Unable to connect to data sources after upgrading to version v7.0.0.b17489

[tomwganem]

Issue Summary

Unable to query any data source after upgrading from 6.0.0.b8537 to 7.0.0.b17489

Steps to Reproduce

1. Upgrade redash container from 6.0.0.b8537 to 7.0.0.b17489
2. Run python /app/manage.py db upgrade

redash@redash-qa-d6c878bc6-dcxtb:/app$ python /app/manage.py db upgrade
[2019-03-21 09:14:04,538][PID:13961][INFO][root] Generating grammar tables from /usr/lib/python2.7/lib2to3/Grammar.txt
[2019-03-21 09:14:04,562][PID:13961][INFO][root] Generating grammar tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
[2019-03-21 09:14:06,738][PID:13961][INFO][alembic.runtime.migration] Context impl PostgresqlImpl.
[2019-03-21 09:14:06,739][PID:13961][INFO][alembic.runtime.migration] Will assume transactional DDL.
[2019-03-21 09:14:06,764][PID:13961][INFO][alembic.runtime.migration] Running upgrade 71477dadd6ef -> 640888ce445d, Add new scheduling data.
[2019-03-21 09:14:06,873][PID:13961][INFO][alembic.runtime.migration] Running upgrade 640888ce445d -> e7f8a917aa8e, Add user details JSON column.
[2019-03-21 09:14:06,900][PID:13961][INFO][alembic.runtime.migration] Running upgrade e7f8a917aa8e -> 73beceabb948, bring_back_null_schedule
[2019-03-21 09:14:06,934][PID:13961][INFO][alembic.runtime.migration] Running upgrade 73beceabb948 -> 98af61feea92, add_encrypted_options_to_data_sources
[2019-03-21 09:14:07,005][PID:13961][INFO][alembic.runtime.migration] Running upgrade 98af61feea92 -> e5c7a4e2df4d, remove_query_tracker_keys

1. Login, attempt to sync any existing query or create a new one

Technical details:

I See this stacktrace corresponding to every attempt by the background workers to sync a query
[2019-03-21 10:08:49,559][PID:124][ERROR][ForkPoolWorker-8] Task redash.tasks.refresh_queries[45cf443f-30e1-4065-b665-0ee704c05fb8] raised unexpected: InvalidToken() Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/celery/app/trace.py", line 382, in trace_task R = retval = fun(*args, **kwargs) File "/app/redash/worker.py", line 64, in __call__ return TaskBase.__call__(self, *args, **kwargs) File "/usr/local/lib/python2.7/dist-packages/celery/app/trace.py", line 641, in __protected_call__ return self.run(*args, **kwargs) File "/app/redash/tasks/queries.py", line 176, in refresh_queries elif query.data_source is None: File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/attributes.py", line 242, in __get__ return self.impl.get(instance_state(instance), dict_) File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/attributes.py", line 601, in get value = self.callable_(state, passive) File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/strategies.py", line 631, in _load_for_state session, state, primary_key_identity, passive) File "<string>", line 1, in <lambda> File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/strategies.py", line 724, in _emit_lazyload primary_key_identity) File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/ext/baked.py", line 519, in _load_on_pk_identity result = list(bq.for_session(self.session).params(**params)) File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/loading.py", line 98, in instances util.raise_from_cause(err) File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 265, in raise_from_cause reraise(type(exception), exception, tb=exc_tb, cause=cause) File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/loading.py", line 79, in instances rows = [proc(row) for row in fetch] File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/loading.py", line 511, in _instance loaded_instance, populate_existing, populators) File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/loading.py", line 611, in _populate_full dict_[key] = getter(row) File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/type_api.py", line 1226, in process return process_value(impl_processor(value), dialect) File "/app/redash/models/types.py", line 28, in process_result_value return ConfigurationContainer.from_json(super(EncryptedConfiguration, self).process_result_value(value, dialect)) File "/usr/local/lib/python2.7/dist-packages/sqlalchemy_utils/types/encrypted/encrypted_type.py", line 409, in process_result_value decrypted_value = self.engine.decrypt(value) File "/usr/local/lib/python2.7/dist-packages/sqlalchemy_utils/types/encrypted/encrypted_type.py", line 216, in decrypt decrypted = self.fernet.decrypt(value) File "/usr/local/lib/python2.7/dist-packages/cryptography/fernet.py", line 103, in decrypt raise InvalidToken InvalidToken

[MrCirca]

I think that REDASH_COOKIE_SECRET is wrong

[tomwganem]

REDASH_COOKIE_SECRET did not change between deploys.

[MrCirca]

What deployment you have; Did you try run your deployment without set REDASH_COOKIE_SECRET?

[tomwganem]

I'm deploying this to a kubernetes cluster. Should I deploy 6.0.0.b8537 with REDASH_COOKIE_SECRET set, and then deploy v7.0.0.b17489 without REDASH_COOKIE_SECRET set?

[tomwganem]

It should also be noted that we tried to 7.0.0.b17489 first, no upgrading from 6.0.0.b8537, and ran into the same issue. REDASH_COOKIE_SECRET was definitely set.

[phlash-zero]

I had similar issue. I had to set REDASH_COOKIE_SECRET for both redash worker and server.

[0daypwn]

I had same error. I set REDASH_COOKIE_SECRET for worker to solve this error. But the upgrade doc did not metion it.

[arikfr]

You need to set REDASH_COOKIE_SECRET or REDASH_SECRET_KEY before running the migrations. This is actually mentioned in the release notes, the CHANGELOG and the forum post announcing this version.

All data source options are now encrypted in the database. By default the encryption uses the REDASH_COOKIE_SECRET value (redash.settings.COOKIE_SECRET), but you can specify a different value by setting the REDASH_SECRET_KEY environment variable value. Note that you need to set this before doing the upgrade.

If you didn't set the COOKIE_SECRET it will use the default one (c292a0a3aa32397cdb050e233733900f) as the encryption key during migration.

The library we use has the ability to update the encryption key, so you will need to use this process to update from the default one to the one you actually use.

[jasonsoderberg]

I have my REDASH_COOKIE_SECRET set in my .env file and it appears the upgrade is not using my .env REDASH_COOKIE_SECRET during the upgrade. It appears that its using the default set in the upgrade Python script. I see 0daypwn mentioned setting the REDASH_COOKIE_SECRET for both redash worker and server, can you provide instructions on this since its not the .env file that I expect redash to be reading from.

[Abhishek-lodha]

I am trying to upgrade my redash setup on docker. I have REDASH_COOKIE_SECRET already set up, and then I tried to insert some row in postgres 'public.data_sources' table through dump commands. psql -f dump.out postgres
I am still getting InvalidToken exception. Please suggest.

[lxcid]

@arikfr can I ask about how do we change the encryption key?

[jeremyfrady]

I was running into this same token issue after migrating from 7.0.0.b18042 to the same 7.0.0.b18042 on a new host.

In my old host, $REDASH_COOKIE_SECRET was defined in containers and populated correctly from env file, however $REDASH_SECRET_KEY was empty and not defined in env.

Checking my new install, $REDASH_COOKIE_SECRET was also defined correctly, however $REDASH_SECRET_KEY was a 32 character string that I was not familiar with.

By setting $REDASH_SECRET_KEY to the same value as $REDASH_COOKIE_SECRET in env and then issuing sudo docker-compose up -d I was able to access my old data sources on my new host.

Hope this helps someone.

[hades-6]

I was running into this same token issue after migrating from 7.0.0.b18042 to the same 7.0.0.b18042 on a new host.

In my old host, $REDASH_COOKIE_SECRET was defined in containers and populated correctly from env file, however $REDASH_SECRET_KEY was empty and not defined in env.

Checking my new install, $REDASH_COOKIE_SECRET was also defined correctly, however $REDASH_SECRET_KEY was a 32 character string that I was not familiar with.

By setting $REDASH_SECRET_KEY to the same value as $REDASH_COOKIE_SECRET in env and then issuing sudo docker-compose up -d I was able to access my old data sources on my new host.

Hope this helps someone.

I'm running into a similar issue. Can you please elaborate a bit more? You set the REDASH_SECRET_KEY from your old instance? My env file does not have REDASH_SECRET_KEY but it does have REDASH_COOKIE_SECRET. Are you saying I should copy the REDASH_COOKIE_SECRET from the old instance into the new one? I actually tried this but it still didn't work.