Google-cloud-python: Bigquery: import error with v.1.23.0

[smdmts]

Steps to reproduce

1. Using google.cloud.bigquery v.1.23.0
   
   
   
   • https://pypi.org/project/google-cloud-bigquery/1.23.0/
   
   
2. Insead of v.1.22.0 works fine.

Code example

from concurrent.futures import ThreadPoolExecutor
from logging import StreamHandler, Formatter, INFO, getLogger
from datetime import datetime, timedelta, timezone

from google.cloud import bigquery
from google.cloud.bigquery import LoadJobConfig
from google.cloud.bigquery import SchemaField

Stack trace

---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<command-778378> in <module>
      3 from datetime import datetime, timedelta, timezone
      4 
----> 5 from google.cloud import bigquery
      6 from google.cloud.bigquery import LoadJobConfig
      7 from google.cloud.bigquery import SchemaField

/databricks/python/lib/python3.7/site-packages/google/cloud/bigquery/__init__.py in <module>
     33 __version__ = get_distribution("google-cloud-bigquery").version
     34 
---> 35 from google.cloud.bigquery.client import Client
     36 from google.cloud.bigquery.dataset import AccessEntry
     37 from google.cloud.bigquery.dataset import Dataset

/databricks/python/lib/python3.7/site-packages/google/cloud/bigquery/client.py in <module>
     56 from google.cloud.bigquery._helpers import _verify_job_config_type
     57 from google.cloud.bigquery._http import Connection
---> 58 from google.cloud.bigquery import _pandas_helpers
     59 from google.cloud.bigquery.dataset import Dataset
     60 from google.cloud.bigquery.dataset import DatasetListItem

/databricks/python/lib/python3.7/site-packages/google/cloud/bigquery/_pandas_helpers.py in <module>
     38     pyarrow = None
     39 
---> 40 from google.cloud.bigquery import schema
     41 
     42 

/databricks/python/lib/python3.7/site-packages/google/cloud/bigquery/schema.py in <module>
     15 """Schemas for BigQuery tables / queries."""
     16 
---> 17 from six.moves import collections_abc
     18 
     19 from google.cloud.bigquery_v2 import types

ImportError: cannot import name 'collections_abc' from 'six.moves' (unknown location)

[SaschaHeyer]

I can reproduce the same behavior

[shihabuddinbuet]

My pipeline broke today. I install google-cloud-bigquery using bootstrap script while creating dataproc cluster and i'm getting the following error after submitting my spark job :

File "/usr/local/lib/python2.7/dist-packages/google/cloud/bigquery/__init__.py", line 35, in <module> from google.cloud.bigquery.client import Client File "/usr/local/lib/python2.7/dist-packages/google/cloud/bigquery/client.py", line 58, in <module> from google.cloud.bigquery import _pandas_helpers File "/usr/local/lib/python2.7/dist-packages/google/cloud/bigquery/_pandas_helpers.py", line 40, in <module> from google.cloud.bigquery import schema File "/usr/local/lib/python2.7/dist-packages/google/cloud/bigquery/schema.py", line 17, in <module> from six.moves import collections_abc ImportError: cannot import name collections_abc
So i've migrated to previous version(1.22.0) and it's working fine.

[HemangChothani]

@smdmts @shihabuddinbuet @SaschaHeyer Please update or install the latest version of six which is 1.13.0. It's due to the older version six like 1.12.0.

[SaschaHeyer]

@HemangChothani
Shouldn't it work out of the box after installing google.cloud.bigquerywithout any need to install further packages?

[HemangChothani]

@SaschaHeyer google.cloud.bigquery install required packages ,but due to some reason package six hasn't been updated. Updated version of six works for me.

[maxzzze]

Can reproduce the same

[winsonhys]

Adding six==1.13.0 to the requirements file of the main project worked for me

[HemangChothani]

Please verify via python -m pip freeze that this python environment has up to date packages. In a clean environment this seems to work.

$ python -m pip freeze
cachetools==3.1.1
certifi==2019.11.28
chardet==3.0.4
google-api-core==1.14.3
google-auth==1.7.2
google-cloud-core==1.1.0
google-cloud-iam==0.2.1
google-cloud-bigquery==1.23.0
googleapis-common-protos==1.6.0
idna==2.8
protobuf==3.11.1
pyasn1==0.4.8
pyasn1-modules==0.2.7
pytz==2019.3
requests==2.22.0
rsa==4.0
six==1.13.0
urllib3==1.25.7

And the imports work as expected.

[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from google.cloud import bigquery
>>> from google.cloud.bigquery import LoadJobConfig

[HemangChothani]

feel free to reopen if this wouldn't solve the problem.

[beckjake]

Hi,

Would it be possible for either google-cloud-bigquery to explicitly list that it requires six >= 1.13.0, or for google-cloud-core to increase its lower bound and google-cloud-bigquery require the new version?

The problem is that pip install --upgrade on projects that depend upon google-cloud-bigquery now fail, because google-cloud-core incorrectly asserts that it only needs six >= 1.10.0 (see here).

I agree that this problem can be fixed by manually upgrading six, but the point of listing dependencies and their versions is that this happens automatically.

[tswast]

Sorry that you have encountered this issue. I should have checked the minimum version of six for the collections.abc fix for Python 3.8. I do see it was added in 1.13.0. https://github.com/benjaminp/six/blob/203b81c2a719466ed13681f0062a4426c07c7481/CHANGES#L11

Indeed, the correct fix is for bigquery to add six >=1.13.0,<2.0.0dev as a dependency in setup.py.

[tswast]

google-cloud-bigquery release with this fix is pending: https://github.com/googleapis/google-cloud-python/pull/9981

[shounakG]

I'm still facing this issue with the updated package. google-cloud-bigquery==1.22.0 works fine.

I'm using the google-cloud-bigquery==1.23.1 and six==1.13.0 package in a jupyter notebooks environment.

https://pypi.org/project/google-cloud-bigquery/1.23.1/
https://pypi.org/project/six/1.13.0/

Code example:
from google.cloud import bigquery

Stacktrace:

ImportError                               Traceback (most recent call last)
<ipython-input-4-1035661e8528> in <module>
----> 1 from google.cloud import bigquery

/opt/conda/lib/python3.6/site-packages/google/cloud/bigquery/__init__.py in <module>
     33 __version__ = get_distribution("google-cloud-bigquery").version
     34 
---> 35 from google.cloud.bigquery.client import Client
     36 from google.cloud.bigquery.dataset import AccessEntry
     37 from google.cloud.bigquery.dataset import Dataset

/opt/conda/lib/python3.6/site-packages/google/cloud/bigquery/client.py in <module>
     56 from google.cloud.bigquery._helpers import _verify_job_config_type
     57 from google.cloud.bigquery._http import Connection
---> 58 from google.cloud.bigquery import _pandas_helpers
     59 from google.cloud.bigquery.dataset import Dataset
     60 from google.cloud.bigquery.dataset import DatasetListItem

/opt/conda/lib/python3.6/site-packages/google/cloud/bigquery/_pandas_helpers.py in <module>
     38     pyarrow = None
     39 
---> 40 from google.cloud.bigquery import schema
     41 
     42 

/opt/conda/lib/python3.6/site-packages/google/cloud/bigquery/schema.py in <module>
     15 """Schemas for BigQuery tables / queries."""
     16 
---> 17 from six.moves import collections_abc
     18 
     19 from google.cloud.bigquery_v2 import types

ImportError: cannot import name 'collections_abc'

[tswast]

@shounakG I'm sorry that you're still experiencing this issue. Can you run

!pip freeze

from the Jupyter notebook to double-check the package versions installed in the kernel?

[shounakG]

Hi @tswast thanks for getting back! I did check pip freeze and it showed the latest version (1.23.1), but still gave out the same error. In the jupyter notebooks environment however, a restart of kernel was needed for changes to take effect. It worked flawlessly after a kernel restart. Thanks for the help!

[jordangonzales]

I'm having this issue and can't seem to figure out what the deal is. I got around this by manually editing the schema.py file to include the following:

try: # works in Python >= 3.3 import collections.abc as collections_abc except ImportError: # Python <= 3.2 including Python 2 import collections as collections_abc

This is driving me nuts

Python 3.7.5
Six = 1.13.0 (Cannot seem to import collections_abc outside of BQ either.)

[sohamsil]

@jordangonzales Latest version of google.cloud.bigquery package is 1.23.1. Try downgrading it to 1.23.0 as it's working with six==1.14

[harzus]

had the same issue with 1.24.0, using 1.23.0 instead fixed it

[SaschaHeyer]

Why is this issue closed?
The behavior can be still reproduced

[HemangChothani]

@SaschaHeyer I have tried to reproduce the issue but not able to do that, Could please share your environment details?

My Environment:

    google-cloud-bigquery==1.24.0
    six==1.14.0
    six==1.13.0  # also working with this version.

[tvlk-karl]

I'm also facing the same error here

[mkalpar]

I am reproducing this issue with these versions:
google-cloud-bigquery==1.24.0 six==1.14.0

[davidsmandrade]

I'm also facing the same error here

[sagydr]

i am facing the same error 😭
google-cloud-bigquery==1.24.0 six==1.14.0

[plamut]

To all people still facing the trouble - this issue lives in the old repository, which is why it probably did not receive enough attention lately, sorry for that. I suggest continuing the thread under the new issue @sagydr created.

Please also double check that six>=1.13.0 is indeed installed (comment) in the Python environment that actually runs the application.

[anistark]

Still facing this issue. My jenkins deployment pipeline broke with the same message.

Packages on my server:

google-cloud-bigquery==2.0.0
six==1.15.0

Please advice.

[fransiscakenny]

@smdmts @shihabuddinbuet @SaschaHeyer Please update or install the latest version of six which is 1.13.0. It's due to the older version six like 1.12.0.

This works for me, thank you!

[austinaoliu]

Still facing this issue. My jenkins deployment pipeline broke with the same message.

Packages on my server:

google-cloud-bigquery==2.0.0
six==1.15.0

Please advice.

Same issue here. Did you solve it?

[ELWAFIHIND]

The same problem here.
from google.cloud import storage File "/env/lib/python3.7/site-packages/google/cloud/storage/__init__.py", line 38, in from google.cloud.storage.batch import Batch File "/env/lib/python3.7/site-packages/google/cloud/storage/batch.py", line 31, in from google.cloud.storage._http import Connection File "/env/lib/python3.7/site-packages/google/cloud/storage/_http.py", line 17, in from google.cloud import _http File "/env/lib/python3.7/site-packages/google/cloud/_http.py", line 22, in from six.moves import collections_abc ImportError: cannot import name 'collections_abc' from 'six.moves' (unknown location)

[austinaoliu]

@JoshuaCrestone six==1.15.0 works?

[JoshuaCrestone]

@aoliu95 - this fixed it for me.

I am using Bazel to build a python project with python rules from https://github.com/dillon-giacoppo/rules_python_external. Even though my six dependency was at 1.15.0, when running my tests, the version would show as 1.12.0. It turned out, 1.12.0 was a hard dependency of the protobuf Bazel rules I was importing. Perhaps it could have been fixed by changing WORKSPACE rule ordering to get the whl dependency to override the protobuf dependency?

[austinaoliu]

@JoshuaCrestone It seems the issue persists even I upgrade my six to 1.15.0

Package                  Version   
------------------------ ----------
adal                     1.2.0     
ansiwrap                 0.8.3     
asn1crypto               0.24.0    
azure-common             1.1.16    
azure-datalake-store     0.0.39    
azure-nspkg              3.0.2     
azure-storage-blob       1.4.0     
azure-storage-common     1.4.0     
backcall                 0.1.0     
bleach                   3.0.2     
boto3                    1.9.51    
botocore                 1.12.51   
cachetools               4.1.1     
certifi                  2018.10.15
cffi                     1.11.5    
chardet                  3.0.4     
Click                    7.0       
cryptography             2.4.2     
decorator                4.3.0     
defusedxml               0.5.0     
docutils                 0.14      
entrypoints              0.2.3     
future                   0.17.1    
google-api-core          1.22.4    
google-api-python-client 1.7.11    
google-auth              1.22.1    
google-auth-httplib2     0.0.4     
google-auth-oauthlib     0.4.1     
google-cloud-bigquery    2.0.0     
google-cloud-core        1.4.3     
google-crc32c            1.0.0     
google-resumable-media   1.1.0     
googleapis-common-protos 1.52.0    
grpcio                   1.32.0    
httplib2                 0.18.1    
idna                     2.7       
ipykernel                5.1.0     
ipython                  7.1.1     
ipython-genutils         0.2.0     
jedi                     0.13.1    
Jinja2                   2.10      
jmespath                 0.9.3     
jsonschema               2.6.0     
jupyter-client           5.2.3     
jupyter-core             4.4.0     
libcst                   0.3.13    
MarkupSafe               1.1.0     
mistune                  0.8.4     
mypy-extensions          0.4.3     
nbconvert                5.4.0     
nbformat                 4.4.0     
numpy                    1.15.4    
oauthlib                 3.1.0     
pandas                   1.1.3     
pandas-gbq               0.12.0    
pandocfilters            1.4.2     
papermill                0.16.1    
parso                    0.3.1     
pexpect                  4.6.0     
pickleshare              0.7.5     
pip                      20.0.2    
prompt-toolkit           2.0.7     
proto-plus               1.10.2    
protobuf                 3.13.0    
psycopg2                 2.7.6.1   
ptyprocess               0.6.0     
pyasn1                   0.4.8     
pyasn1-modules           0.2.8     
pycparser                2.19      
pydata-google-auth       1.1.0     
Pygments                 2.3.0     
PyJWT                    1.6.4     
pyodbc                   4.0.24    
python-dateutil          2.7.5     
pytz                     2018.7    
PyYAML                   5.3.1     
pyzmq                    17.1.2    
requests                 2.20.1    
requests-oauthlib        1.3.0     
rsa                      4.6       
s3transfer               0.1.13    
setuptools               40.6.2    
six                      1.15.0    
SQLAlchemy               1.2.14    
testpath                 0.4.2     
textwrap3                0.9.1     
tornado                  5.1.1     
tqdm                     4.28.1    
traitlets                4.3.2     
typing-extensions        3.7.4.3   
typing-inspect           0.6.0     
uritemplate              3.0.1     
urllib3                  1.24.1    
wcwidth                  0.1.7     
webencodings             0.5.1     
wheel                    0.32.2    

Which version are you using for google-cloud-bigquery and google-api-python-client?

[JoshuaCrestone]

@aoliu95 - are you using Bazel? In my case, I am using Bazel, and a workspace dependency was overwriting a project dependency. If you add:

import six
print(six.__version__)

to the failing code, you will see which version of six is being resolved at runtime. For me, my requirements.txt specified six 1.15.0, but at runtime, it was resolving to 1.12.0, and this conflict was coming from my WORKSPACE dependency on protobuf.

P.S. google-cloud-bigquery==1.28.0, google-api-python-client==1.12.3

[austinaoliu]

@aoliu95 - are you using Bazel? In my case, I am using Bazel, and a workspace dependency was overwriting a project dependency. If you add:

import six
print(six.__version__)

to the failing code, you will see which version of six is being resolved at runtime. For me, my requirements.txt specified six 1.15.0, but at runtime, it was resolving to 1.12.0, and this conflict was coming from my WORKSPACE dependency on protobuf.

P.S. google-cloud-bigquery==1.28.0, google-api-python-client==1.12.3

I solved the issue at https://github.com/pydata/pandas-gbq/issues/304#issuecomment-713803811. It was because my Jupyter was not picking the newly installed six after running pip update. Reloading six solves the issue

[pblin]

google-cloud-bigquery==1.24.0 six==1.14 works with python 2.7 but not python 3.7

[aditya-novasignal]

I am also getting this same error message with google-cloud-storage==1.33.0 and six==1.15.0. I have other dependencies that require Python 3.7, did anyone find a solution that works with 3.7?