Extensions: Error when mirroring data to BigQuery PartialFailureError

[zdenulo]

[REQUIRED] Step 2: Describe your configuration

• Extension name: Export Collections to BigQuery
• Extension version: 0.1.2

[REQUIRED] Step 3: Describe the problem

I set up the extension and upon inserting a document in Firestore, Cloud Function throws an error in logs:
Error when mirroring data to BigQuery { PartialFailureError
at request (/srv/node_modules/@google-cloud/bigquery/build/src/table.js:1548:23)
at Util.handleResp (/srv/node_modules/@google-cloud/common/build/src/util.js:141:9)
at retryRequest (/srv/node_modules/@google-cloud/common/build/src/util.js:422:22)
at onResponse (/srv/node_modules/retry-request/index.js:206:7)
at /srv/node_modules/teeny-request/build/src/index.js:222:13
at
at process._tickDomainCallback (internal/process/next_tick.js:229:7)
errors: [ { errors: [Array], row: [Object] } ],
name: 'PartialFailureError',
response:
{ kind: 'bigquery#tableDataInsertAllResponse',
insertErrors: [ [Object] ] },
message: '' }

And of course, there is no data in BigQuery table.

I saw there were similar errors in the recent past regarding batch import but since this is in Cloud Function, I'm creating a separate issue.

Cloud Function source location
https://storage.cloud.google.com/firebase-mod-sources-prod/e1a82bec6981c73a5fe6bff28483f3125ca703584c5fad43d1c4b7fa0f67d388

[matteosb]

I am seeing the same error for extensions installed in the last 24 hours. In some cases we get a permission denied error as well as the error above, even though the IAM roles look correct. Identical extensions installed day before yesterday are working fine:
{ Error: Access Denied: Dataset <project>:firestore_export: User does not have bigquery.datasets.get permission for dataset <project>:firestore_export. at Util.parseHttpRespBody (/srv/node_modules/@google-cloud/common/build/src/util.js:190:38) at Util.handleResp (/srv/node_modules/@google-cloud/common/build/src/util.js:134:117) at retryRequest (/srv/node_modules/@google-cloud/common/build/src/util.js:422:22) at onResponse (/srv/node_modules/retry-request/index.js:206:7) at /srv/node_modules/teeny-request/build/src/index.js:222:13 at <anonymous> at process._tickDomainCallback (internal/process/next_tick.js:229:7) code: 403, errors: [ { message: 'Access Denied: Dataset <project>:firestore_export: User does not have bigquery.datasets.get permission for dataset <project>:firestore_export.', domain: 'global', reason: 'accessDenied' } ], response: undefined, message: 'Access Denied: Dataset <project>:firestore_export: User does not have bigquery.datasets.get permission for dataset <project>:firestore_export.' }

[livingspree]

I'm facing the exact same issue as @zdenulo

Following this thread

[Shailevy]

Same here

[russellwheatley]

Hi @zdenulo, there was a fix for the firestore-bigquery-export that was merged into master (#192 updated). I suggest tracking the @firebaseextensions/firestore-bigquery-change-tracker package for its latest release which ought to fix the PartialFailureError you're currently experiencing.

[zdenulo]

@russellwheatley thanks for the info. Didn't you mean perhaps #192? I did my deployment via firebase web console. I now checked, i.e. downloaded code from the url I included (and based on which my Cloud Function is deployed) and CHANGELOG file there it says that it's release 0.1.2 which contains fixes for #192, #193 and #197. Maybe I'm missing something, Javascript/NodeJS/Typescript is exactly my area. on NPM latest release is 0.1.1

[pavitra15]

Same error here , tried but failed again.
Can anyone help??

[russellwheatley]

Sorry, you're correct, it is #192. I've updated my comment for anyone else coming to this thread. This is the package you ought to be tracking: https://www.npmjs.com/package/@firebaseextensions/firestore-bigquery-change-tracker. I would ignore the version number as it has been incorrectly set. This package is nested within the firestore-bigquery-export extension. Here is a reference to it: https://github.com/firebase/extensions/tree/next/firestore-bigquery-export/firestore-bigquery-change-tracker

[zdenulo]

@russellwheatley thanks for the clarification.

[balderdash]

@russellwheatley Thanks for looking into this bug. Do you have a rough idea of when a new version of @firebaseextensions/firestore-bigquery-change-tracker will be published?

[nedols]

I'm facing the exact same issue.
Following this thread

[pgoode41]

Same issue here. Following as well.

[daviduzan]

I'll join the followers

[Shunpoco]

I have same issue now. Following as well.

[renahnschultz]

Same here. Following as well

[russellwheatley]

I will update this thread once it has been published, but I don't know when it will be published as It isn't my decision I'm afraid.

[konato]

I have the same issue here. Had it installed last week into the our dev project . It is still working.
But then yesterday try to add BigQuery using Console to our Alpha project as previously done with dev.

But in this case, we have the partial error :

And no data are being transferred to BigQuery.

[nebnhoj]

I encountered same issue while I'm mirroring my collection to BigQuery

Error when mirroring data to BigQuery { PartialFailureError at request (/srv/node_modules/@google-cloud/bigquery/build/src/table.js:1548:23) at Util.handleResp (/srv/node_modules/@google-cloud/common/build/src/util.js:141:9) at retryRequest (/srv/node_modules/@google-cloud/common/build/src/util.js:422:22) at onResponse (/srv/node_modules/retry-request/index.js:206:7) at /srv/node_modules/teeny-request/build/src/index.js:222:13 at <anonymous> at process._tickDomainCallback (internal/process/next_tick.js:229:7) errors: [ { errors: [Array], row: [Object] } ], name: 'PartialFailureError', response: { kind: 'bigquery#tableDataInsertAllResponse', insertErrors: [ [Object] ] }, message: '' }

[fabiangruss]

Same issue & following as well.

[abegehr]

Same issue here. @russellwheatley, can the update be used on some beta-channel already?
The issue made exporting impossible for almost 2 months now.

[abegehr]

For everyone searching for a temporary solution, check out the workaround by @joaqcid: https://github.com/firebase/extensions/issues/192#issuecomment-584132076 and https://github.com/firebase/extensions/issues/192#issuecomment-584632553. Worked well for me 👍

[mindyzwan]

Same issue here! Not really sure how to implement the workaround listed.

[joaqcid]

@IanWyszynski looks like when they merged the fix the forgot to bump @firebaseextensions/firestore-bigquery-change-tracker, its still using old version

[joaqcid]

the workaround works if you want to run the import manually @mindyzwan

[russellwheatley]

@joaqcid The bigquery-export extension does need this package bumping@firebaseextensions/firestore-bigquery-change-tracker at some point. It isn't necessary however, as the bigquery-export extension is installing the latest minor version because it is using the caret symbol in the package.json: https://github.com/firebase/extensions/blob/master/firestore-bigquery-export/package.json#L23.

It is more important to publish the fix you created in #197 that was merged in 27 days ago. The last time @firebaseextensions/firestore-bigquery-change-tracker was published was 2 months ago.

[pmohun]

+1 running into the same issue with a Firestore database using the BigQuery extension.

Error when mirroring data to BigQuery { PartialFailureError
    at request (/srv/node_modules/@google-cloud/bigquery/build/src/table.js:1548:23)
    at Util.handleResp (/srv/node_modules/@google-cloud/common/build/src/util.js:141:9)
    at retryRequest (/srv/node_modules/@google-cloud/common/build/src/util.js:422:22)
    at onResponse (/srv/node_modules/retry-request/index.js:206:7)
    at /srv/node_modules/teeny-request/build/src/index.js:222:13
    at <anonymous>
    at process._tickDomainCallback (internal/process/next_tick.js:229:7)
  errors: [ { errors: [Array], row: [Object] } ],
  name: 'PartialFailureError',
  response: 
   { kind: 'bigquery#tableDataInsertAllResponse',
     insertErrors: [ [Object] ] },
  message: '' }

[chhib]

@pmohun What I did was download this library from the comment above. https://github.com/firebase/extensions/issues/192#issuecomment-590147765

Checkout the run-manual-import branch. Then ZIP the export.

git clone [email protected]:joaqcid/extensions.git
cd extensions
git checkout run-manual-import
zip archive.zip -r firestore-bigquery-export

Then I went to GCP, find the Function Details of it in my project, hit the Edit button.

I uploaded the zip into a storage bucket I had to create. And now it is running fine.

[edward-armstrong]

@chhib thanks, this looks to have worked for me. If anyone else gets the error File index.js or function.js that is expected to define function doesn't exist in the root directory just make sure you zip the files inside the firestore-bigquery-export directory not the directory itself (like I did).

[konato]

@chhib thanks, It did work for me.
@edward-armstrong, yes I had to zip the files inside the firestore-bigquery-export directory not the directory itself.

So for me steps were:
git clone [email protected]:joaqcid/extensions.git
cd extensions
git checkout run-manual-import
cd firestore-bigquery-export
zip archive.zip -r

Thank you,

Stephane

Note: I had error first but I created a bucket to store the zip file and then use the option ZIP from Cloud Storage instead ZIP Upload option.

[fguespe]

Hey , same happening here! it was working already fine for other 3 collections. But when i install the extension four the 4th time, for another collection, it stopped working! Same error

 Error when mirroring data to BigQuery { PartialFailureError
    at request (/srv/node_modules/@google-cloud/bigquery/build/src/table.js:1548:23)
    at Util.handleResp (/srv/node_modules/@google-cloud/common/build/src/util.js:141:9)
    at retryRequest (/srv/node_modules/@google-cloud/common/build/src/util.js:422:22)
    at onResponse (/srv/node_modules/retry-request/index.js:206:7)
    at /srv/node_modules/teeny-request/build/src/index.js:222:13
    at <anonymous>
    at process._tickDomainCallback (internal/process/next_tick.js:229:7)
  errors: [ { errors: [Array], row: [Object] } ],
  name: 'PartialFailureError',
  response: 
   { kind: 'bigquery#tableDataInsertAllResponse',
     insertErrors: [ [Object] ] },
  message: '' } 

[ducdv98]

Same issue here. Following as well!

[rafaelacb]

Same error! When this will be fixed??

[fguespe]

Any workaround?

[laurenzlong]

Hi, the package has been updated on NPM, and we are updating the extension's dependency in this week's release: https://github.com/firebase/extensions/pull/249/files

Sorry for the long delay!

[stari4ek]

subscribing.
I've got this issue after updating to 0.1.2 version.
2 exports out of 3 work well.

[Sterv]

@stari4ek 0.1.3 has the update, the extension is currently 0.1.2

[zdenulo]

I updated extension to 0.1.3 in Firebase UI and it looks like it's working ok now.

[laurenzlong]

Thanks for the update @zdenulo ! Closing this issue now.

[cubissimo]

Hello folks! I hope I don't resurrect this issue unnecessarily. But maybe was a regression in version 0.1.6.

I noticed today after upgrade from 0.1.5 to 0.1.6 through firebase extension UI the following error. I have other ext-firestore-bigquery in same project running in 0.1.5 that works like a charm.

Someone can help me confirming that?

ext-firestore-bigquery-vouchercosts-fsexportbigquery
Error when mirroring data to BigQuery { PartialFailureError
    at request (/srv/node_modules/@firebaseextensions/firestore-bigquery-change-tracker/node_modules/@google-cloud/bigquery/build/src/table.js:1550:23)
    at util.makeRequest.params (/srv/node_modules/@google-cloud/common/build/src/util.js:367:25)
    at Util.handleResp (/srv/node_modules/@google-cloud/common/build/src/util.js:144:9)
    at retryRequest (/srv/node_modules/@google-cloud/common/build/src/util.js:432:22)
    at onResponse (/srv/node_modules/retry-request/index.js:206:7)
    at /srv/node_modules/teeny-request/build/src/index.js:233:13
    at <anonymous>
    at process._tickDomainCallback (internal/process/next_tick.js:229:7)
  errors: [ { errors: [Array], row: [Object] } ],
  name: 'PartialFailureError',
  response: 
   { kind: 'bigquery#tableDataInsertAllResponse',
     insertErrors: [ [Object] ] },
  message: '' } 

How can i downgrade to 0.1.5 to test it out?

[laurenzlong]

Hi @cubissimo it's not possible to downgrade unfortunately, but you could uninstall the extension, and reinstall an older version of the extension via the CLI by running:
firebase ext:install [email protected] --project=your-project-id

@russellwheatley Could you please see if you can repro this issue?

[russellwheatley]

Hey @cubissimo, I've just installed version 0.1.6 and I had no problem mirroring data from Firestore:

Could you provide further context for reproducing this error, please? 😄

[neelance]

We are also seeing PartialFailureErrors after upgrading to 0.1.6. Unfortunately I can not give more information because a PartialFailureError does not get logged in a way that allows looking at the individual errors in the errors field.

[russellwheatley]

I've just tried this again starting with version 0.1.5, creating a table and upgrading to version 0.1.6 via the Firebase UI and I didn't encounter this error.

However, I have an idea what might be happening. @cubissimo @neelance, do you mind letting me know if your changelog table has the document_id column? Thanks.

[neelance]

There is no document_id column.

[russellwheatley]

Ok, I think I understand the problem. The firestore-bigquery-change-tracker package is expecting a table that has a document_id column. I do apologise for this, but the way around it is to delete your changelog table & reinstall the extension. To backfill your data you can use the import script here.

[karayu]

We're working on a fix right now for this and will likely push a version of the extension with this fix. Thanks so much for the feedback!

[neelance]

Awesome! Are you going to improve the logging, too?

[jobondi]

I had this same issue after updating from 0.1.5 to 0.1.6, I was able to get it working again by manually adding a document_id column via the BigQuery console. @karayu I expect this is also an ok workaround to get it working quicker since dropping and then re-importing would be a long process with the amount of data we'd have to backfill.

[jhuleatt]

The fix for this is in #398

[jhuleatt]

We've released a new version of the extension, 0.1.7, that contains the fix. You should be prompted to update in the Firebase console, or can update via the CLI.

[neelance]

~The update worked fine.~ Thanks for your work.

[neelance]

@jhuleatt Unfortunately this resulted in bad data. The *_latest views were returning documents twice, one with document_id set and one without.

I now resorted to dropping and backfilling all tables.

[jhuleatt]

@russellwheatley can you please take a look at this and see if we need to investigate another fix, or if it can be closed?

[russellwheatley]

@jhuleatt I installed version 0.1.5 (pre document id column feature) & inserted two documents via Firestore. I upgraded to version 0.1.7 and inserted one document. The latest & changelog table appeared as expected; the two documents inserted using version 0.1.5 had a null value in the document_id column.

The document inserted using version 0.1.7 had a document_id column with its Firestore document id as its value.

As far as I can tell, there isn't an issue with this fix #398. I'm going to close this issue, and any further problems should have a new issue associated with it.