Google-cloud-python: BigQuery: No errors, but rows sent from python script to BigQuery not appearing.

[Jordan-Gillard]

OS: Ubuntu 14.04.5 LTS
Python 2.7.6 and running in c9.
google-cloud-python version: google-cloud-bigquery==0.31.0

Hey guys,
My code seems to be working fine, I get no errors when I run it (so therefore no traceback). The problem is that the row I'm sending does not appear in my BigQuery table. I've tried waiting, and after a day it still does not show. I know the table name is correct, because I create the table with it earlier in my code.

client = bigquery.Client()
        dataset = client.dataset(dataset_id)
        schema = [
        bigquery.SchemaField('product_name', 'STRING', mode='NULLABLE'),
        bigquery.SchemaField('country', 'STRING', mode='NULLABLE'),
        bigquery.SchemaField('date_of_data', 'DATE', mode='NULLABLE'),
        bigquery.SchemaField('platform', 'STRING', mode='NULLABLE'),
        bigquery.SchemaField('active_users', 'INTEGER', mode='NULLABLE')
        ]
        table_name = 'active_users_{}'.format(date_for_bq)
        table = dataset.table(table_name)
        row = 'Hornet','FR','2018-02-10','android_phone','34193'
        client.insert_rows(table, row, selected_fields = schema)

I've also tried it with row=['Hornet','FR','2018-02-10','android_phone','34193'] and row=['Hornet,FR,2018-02-10,android_phone,34193'] and gotten the same result. The table exists in my dataset, and it shares the same schema as what you see in my code above. Any idea what I'm doing wrong?

[tseaver]

Client.insert_rows takes as its second argument a sequence of row tuples. You are passing a single tuple (or the equivalent list). So, your last line should read:

client.insert_rows(table, [row], selected_fields = schema)

[Jordan-Gillard]

Ugh. Sorry man, I still can't get it to work. This is what my code looks like now:

def active_users(self, base_values, daily_active_users, date_for_bq):
        bq_tuple = base_values + (daily_active_users,)
        table_name = 'active_users_{}'.format(date_for_bq)
        client = bigquery.Client()
        dataset = client.dataset(dataset_id)
        schema = [
        bigquery.SchemaField('product_name', 'STRING', mode='NULLABLE'),
        bigquery.SchemaField('country', 'STRING', mode='NULLABLE'),
        bigquery.SchemaField('date_of_data', 'DATE', mode='NULLABLE'),
        bigquery.SchemaField('platform', 'STRING', mode='NULLABLE'),
        bigquery.SchemaField('active_users', 'INTEGER', mode='NULLABLE')
        ]
        table = dataset.table(table_name)
        try:
            create_active_users_table(table_name)
            print('{} created!'.format(table_name))
        except Exception:
            pass #come back to this ugly exception later

        client.insert_rows(table, [bq_tuple], selected_fields = schema)

With bq_tuble = (u'Hornet', u'FR', u'2018-02-10', u'android_phone', 34193). Thanks for the help. I'm a bit of a noobie so I appreciate it a ton!

[Jordan-Gillard]

@tseaver Anyway you could just take a quick glance at this? Im sorry for the inconvenience! :(

[Jordan-Gillard]

To anyone reading this: The problem was two things. I was streaming correctly, but when I'd click Preview it wouldn't show up. The data was still in the table, but it seems to be an issue with BQ's UI. I could still query that data. The bigger problem was that I couldn't stream into a freshly created table. You have to wait a minute or so after a table is created to stream into it.

[tseaver]

@Jordan-Gillard There are some latencies outlined in the data availability docs for streamed data. I'm glad you found a solution.

[TrinhDinhPhuc]

@Jordan-Gillard Thank you very much for your information. I thought that my code has errors. But actually, data has imported already. Something's wrong with Bigquery UI.

[cucer]

Hi @Jordan-Gillard
I had same issue, just check your date and reformat it by "datetime" module and it will work.