Google-cloud-python: BigQuery storage client error when downloading to pandas dataframe

[YvesJacquot]

Hi,

I am trying to download query results using the BigQuery client library, as per instructions on this page.

import google.auth
from google.cloud import bigquery
from google.cloud import bigquery_storage_v1beta1
# Make clients.
bqclient = bigquery.Client(
    credentials=credentials,
    project=your_project_id,
)
bqstorageclient = bigquery_storage_v1beta1.BigQueryStorageClient(
    credentials=credentials
)

# Download query results.
table_name = 'project.dataset.table'
query = f'''\
SELECT *
FROM   `{table_name}`
'''
dataframe = (
    bqclient.query(query)
    .result()
    .to_dataframe(bqstorage_client=bqstorageclient)
)
print(dataframe.shape)

I get the following error message.

File "/Users/**/anaconda3/lib/python3.7/site-packages/google/cloud/bigquery/_pandas_helpers.py", line 618, in _download_table_bqstorage
read_options = bigquery_storage_v1beta1.types.TableReadOptions()
AttributeError: 'NoneType' object has no attribute 'types'

Any idea how I could fix this?

I have the following config:

macOS High Sierra Version 10.13.6
Python 3.7.6.
google-cloud 0.34.0
google-cloud-bigquery 1.22.0
google-cloud-bigquery-storage 0.7.0

Many thanks

[IlyaFaer]

Looking at this lines, I'd say you just need to install BigQuery Storage library:
https://github.com/googleapis/google-cloud-python/blob/0384238268a9cc84275eb5350458a91ec81e1d68/bigquery/google/cloud/bigquery/_pandas_helpers.py#L24-L27
See this folder:
https://github.com/googleapis/google-cloud-python/tree/master/bigquery_storage/google/cloud

[IlyaFaer]

@tswast, I believe, such a failure should be covered with some kind of a message, 'cause no attribute seems not very descriptive to me. WDYT?

[YvesJacquot]

Hi
Yes it seems like _pandas_helpers.py doesn't manage to import the BigQuery Storage library.
However I did install it (version 0.7.0.).
What should I look for in the folder you mentioned?
Thank you

[IlyaFaer]

@YvesJacquot, well, this is just the directory you should install
Did you installed it like described in client readme, with virtualenv?
https://github.com/googleapis/google-cloud-python/blob/master/bigquery_storage/README.rst
I think, this should be enough to make things work. If it still fails, we should take a look at import error traceback (maybe there is something more missing).

[YvesJacquot]

@IlyaFaer, yes, installing it in a virtual environment solved it
Thank you for your help!

[piktiv-ilkin]

I was using this example to get data from my dataset and got same error. I deleted bqstorage_client=bqstorageclient param from to_dataframe function and it solved my problem. New code:

dataframe = (
    bqclient.query(query_string)
    .result()
    .to_dataframe()
)

I'm using AI-Platform Notebooks server, with

• python 3.7.6
• google-cloud-bigquery==1.25.0
• google-cloud-bigquery-storage==1.0.0

[kstamatiou]

I confirm that I had to delete bqstorage_client=bqstorageclient from this example in order to get it to work.

[ivanmkc]

@tswast I can also confirm that I had to delete bqstorage_client=bqstorageclient for it to work. Seems like a bug.

I didn't require this fix when running my script locally, but when running on Vertex AI training, it didn't work until I used this fix. Ping me for the code.