Incubator-superset: AttributeError on superset db upgrade

[KilimAnnejaro]

Hi, when I tried to run superset db upgrade I got the error "AttributeError: module 'sqlalchemy.util' has no attribute 'deprecated_params'". I already tried downgrading my version of sqlalchemy but it didn't help.

Screenshots

How to reproduce the bug

Run installation commands from https://superset.incubator.apache.org/installation.html, except with pip3 rather than pip.

Environment

(please complete the following information):

• superset version: Most recent
• python version: Python 2.7.15 :: Anaconda, Inc. (Note: I am using pip3 and python3 to get around version 2 req's.)
  UPDATE: python version now Python 3.6.7 :: Anaconda, Inc. -- still getting the same error.
• node.js version: v11.6.0
• npm version: 6.5.0-next.0

Checklist

Make sure these boxes are checked before submitting your issue - thank you!

• [ x ] I have checked the superset logs for python stacktraces and included it here as text if there are any.
• [ x ] I have reproduced the issue with at least the latest released version of superset.
• [ x ] I have checked the issue tracker for the same issue and I haven't found one similar.

Additional context

Add any other context about the problem here.

[issue-label-bot[bot]]

Issue-Label Bot is automatically applying the label #bug to this issue, with a confidence of 0.88. Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback!

Links: app homepage, dashboard and code for this bot.

[villebro]

Is py3.6 installed via Anaconda?

[KilimAnnejaro]

It wasn't at first, and when I ran conda install python3 I got the following error:
PackagesNotFoundError: The following packages are not available from current channels:

• python3

I also ran the command conda install python==3.6.7, which did update the version, but I was still getting the deprecated params issue.

[elukey]

@KilimAnnejaro what sqlalchemy version have you tried? Also, are you testing superset 0.32 or the latest master? pip freeze's output would also probably help to have in here. I think that we could easily find when sqlalchemy.utils removed the "deprecated_params" and see what version is best?

EDIT: I think that sqlalchemy >= 1.3.0 is needed.

[KilimAnnejaro]

This is my pip freeze output:

alembic==1.0.10
amqp==2.4.2
apispec==1.3.2
asn1crypto==0.24.0
attrs==19.1.0
Babel==2.6.0
billiard==3.6.0.0
bleach==3.1.0
boto3==1.4.7
botocore==1.7.48
cchardet==2.1.4
celery==4.3.0
certifi==2019.3.9
cffi==1.12.3
chardet==3.0.4
click==6.7
colorama==0.3.9
conda==4.6.14
contextlib2==0.5.5
croniter==0.3.30
cryptography==2.6.1
decorator==4.4.0
defusedxml==0.6.0
docutils==0.14
et-xmlfile==1.0.1
Flask==0.12.4
Flask-AppBuilder==2.0.0
Flask-Babel==0.12.2
Flask-Caching==1.7.1
Flask-Compress==1.4.0
Flask-JWT-Extended==3.18.1
Flask-Login==0.4.1
Flask-Migrate==2.4.0
Flask-OpenID==1.2.5
Flask-SQLAlchemy==2.4.0
Flask-WTF==0.14.2
flower==0.9.3
future==0.16.0
geographiclib==1.49
geopy==1.19.0
gunicorn==19.9.0
humanize==0.5.1
idna==2.8
ijson==2.3
isodate==0.6.0
itsdangerous==1.1.0
jdcal==1.4.1
Jinja2==2.10.1
jmespath==0.9.4
jsonlines==1.2.0
jsonschema==3.0.1
kombu==4.5.0
linear-tsv==1.1.0
Mako==1.0.9
Markdown==3.1
MarkupSafe==1.1.1
marshmallow==2.19.2
marshmallow-enum==1.4.1
marshmallow-sqlalchemy==0.16.2
mkl-fft==1.0.12
mkl-random==1.0.2
numpy==1.16.3
openpyxl==2.4.11
pandas==0.24.2
parsedatetime==2.4
pathlib2==2.3.3
polyline==1.3.2
prison==0.1.0
psycopg2==2.7.7
py==1.8.0
pycosat==0.6.3
pycparser==2.19
pydruid==0.5.2
PyHive==0.6.1
PyJWT==1.7.1
pyOpenSSL==19.0.0
pyrsistent==0.15.1
PySocks==1.6.8
python-dateutil==2.8.0
python-editor==1.0.4
python-geohash==0.8.5
python3-openid==3.1.0
pytz==2019.1
PyYAML==5.1
requests==2.21.0
retry==0.9.2
rfc3986==1.3.1
ruamel-yaml==0.15.46
s3transfer==0.1.13
sasl==0.2.1
selenium==3.141.0
simplejson==3.16.0
six==1.12.0
SQLAlchemy==1.3.3
SQLAlchemy-Utils==0.33.11
sqlparse==0.3.0
superset==0.28.1
tableschema==1.4.1
tabulator==1.20.0
thrift==0.11.0
thrift-sasl==0.3.0
tornado==5.1.1
unicodecsv==0.14.1
Unidecode==1.0.23
urllib3==1.24.3
vine==1.3.0
webencodings==0.5.1
Werkzeug==0.15.2
WTForms==2.2.1
WTForms-JSON==0.3.3
xlrd==1.2.0

I'm testing the latest master.

Thank you!

[elukey]

@KilimAnnejaro I just noticed that marshmallow-sqlalchemy is at the top of the stacktrace, maybe you'd need to use a more recent version? I can see this one in master's requirements.txt: https://github.com/apache/incubator-superset/blob/master/requirements.txt#L53

[KilimAnnejaro]

I think I'm using the same version as in the requirements.txt? But at any rate I just tried pip install marshmallow-sqlalchemy and the Attribute error didn't change

[elukey]

You are totally right, confused some package names. Really strange then..
The error seems coming from sqlachemy itself, maybe it could be worth to test another version of sqlalchemy? https://pypi.org/project/SQLAlchemy/1.3.3/#history

[KilimAnnejaro]

I tried that and got this error: sqlalchemy.exc.InvalidRequestError: Can't determine which FROM clause to join from, there are multiple FROMS which can join to this entity. Try adding an explicit ON clause to help resolve the ambiguity.

So I used the fix from https://github.com/apache/incubator-superset/issues/6977 and got the new error "ModuleNotFoundError: No module named 'sqlalchemy'". Which feels really strange to me, since when I run pip freeze I get this output:

alembic==1.0.10
amqp==2.4.2
apispec==1.3.2
asn1crypto==0.24.0
attrs==19.1.0
Babel==2.6.0
billiard==3.6.0.0
bleach==3.1.0
boto3==1.4.7
botocore==1.7.48
cchardet==2.1.4
celery==4.3.0
certifi==2019.3.9
cffi==1.12.3
chardet==3.0.4
click==6.7
colorama==0.3.9
conda==4.6.14
contextlib2==0.5.5
croniter==0.3.30
cryptography==2.6.1
decorator==4.4.0
defusedxml==0.6.0
docutils==0.14
et-xmlfile==1.0.1
Flask==0.12.4
Flask-AppBuilder==2.0.0
Flask-Babel==0.12.2
Flask-Caching==1.7.1
Flask-Compress==1.4.0
Flask-JWT-Extended==3.18.1
Flask-Login==0.4.1
Flask-Migrate==2.4.0
Flask-OpenID==1.2.5
Flask-SQLAlchemy==2.4.0
Flask-WTF==0.14.2
flower==0.9.3
future==0.16.0
geographiclib==1.49
geopy==1.19.0
gunicorn==19.9.0
humanize==0.5.1
idna==2.8
ijson==2.3
isodate==0.6.0
itsdangerous==1.1.0
jdcal==1.4.1
Jinja2==2.10.1
jmespath==0.9.4
jsonlines==1.2.0
jsonschema==3.0.1
kombu==4.5.0
linear-tsv==1.1.0
Mako==1.0.9
Markdown==3.1
MarkupSafe==1.1.1
marshmallow==2.19.2
marshmallow-enum==1.4.1
marshmallow-sqlalchemy==0.16.2
mkl-fft==1.0.12
mkl-random==1.0.2
numpy==1.16.3
openpyxl==2.4.11
pandas==0.24.2
parsedatetime==2.4
pathlib2==2.3.3
polyline==1.3.2
prison==0.1.0
psycopg2==2.7.7
py==1.8.0
pycosat==0.6.3
pycparser==2.19
pydruid==0.5.2
PyHive==0.6.1
PyJWT==1.7.1
pyOpenSSL==19.0.0
pyrsistent==0.15.1
PySocks==1.6.8
python-dateutil==2.8.0
python-editor==1.0.4
python-geohash==0.8.5
python3-openid==3.1.0
pytz==2019.1
PyYAML==5.1
requests==2.21.0
retry==0.9.2
rfc3986==1.3.1
ruamel-yaml==0.15.46
s3transfer==0.1.13
sasl==0.2.1
selenium==3.141.0
simplejson==3.16.0
six==1.12.0
SQLAlchemy==1.3.2
SQLAlchemy-Utils==0.33.11
sqlparse==0.3.0
superset==0.28.1
tableschema==1.4.1
tabulator==1.20.0
thrift==0.11.0
thrift-sasl==0.3.0
tornado==5.1.1
unicodecsv==0.14.1
Unidecode==1.0.23
urllib3==1.24.3
vine==1.3.0
webencodings==0.5.1
Werkzeug==0.15.2
WTForms==2.2.1
WTForms-JSON==0.3.3
xlrd==1.2.0

[KilimAnnejaro]

I think I should note that when I run python in interactive mode and try to 'import sqlalchemy', I get the error initially described in this issue ("AttributeError: module 'sqlalchemy.util' has no attribute 'deprecated_params'"). So I think the problem here is definitely sqlalchemy, and not so much superset.

[elukey]

Have you also tried with sqlalchemy==1.2.18 ?

[KilimAnnejaro]

Okay, I fixed this issue.

I was developing outside the virtual environment recommended by the installation instructions. When I did this, I couldn't install a sufficiently low version of sqlalchemy. Once I got my venv environment to work, the issue was fixed.

Thanks to everyone for their help!

[villebro]

hmm, you seem to have superset 0.28.1 installed via pip. I'm fairly certain that doesn't work with sqlalchemy 1.3. I would recommend making a fresh venv with eg. pipenv and stay away from conda etc and make sure you install 0.32rc2, which works much better with recent deps.

[KilimAnnejaro]

@villebro that's what I did and it worked

[villebro]

haha I seem to have typed that at the same time as you. happy to hear things worked out 👍

[vietvudanh]

Why does version not defined in setup script? both latest pandas and sqlalchemy failed with latest version of superset.