Extensions: firestore-bigquery-export failing

[madmacc]

• Extension name: firestore-bigquery-export

I have installed the Export Collections to BigQuery extension and it works fine however now I am attempting to import my existing data using firestore-bigquery-export by following the instructions here:
https://github.com/firebase/extensions/blob/master/firestore-bigquery-export/guides/IMPORT_EXISTING_DOCUMENTS.md

However I am getting the error detailed below.
Note: I get exactly the same error even if I enter a firebase projectid or collection that does not exist.

This may be an local environment issue, if so and this is not an appropriate place for this issue please let me know and close it.

$ node -v
v12.13.0

$ npm -v
6.12.0

Steps to reproduce:

Run: npx @firebaseextensions/fs-bq-import-collection
Enter the ${PROJECT_ID}, ${COLLECTION_PATH} and ${DATASET_ID} parameters as follows:
my-fb-project-id, members, qmtbc_dev_export

Expected result

Collection imports successfully into BigQuery

Actual result

Error returned:

Importing data from Cloud Firestore Collection: members, to BigQuery Dataset: qmtbc_dev_export, Table: members_raw_changelog
Error unlinking journal file C:\Users\fullo\AppData\Roaming\npm-cache\_npx\10884\node_modules\@firebaseextensions\fs-bq-import-collection\lib/from-members-to-qmtbc-dev:qmtbc_dev_export:members_raw_changelog
after successful import: Error: ENOENT: no such file or directory, unlink 'C:\Users\fullo\AppData\Roaming\npm-cache\_npx\10884\node_modules\@firebaseextensions\fs-bq-import-collection\lib\from-members-to-qmtbc-dev:qmtbc_dev_export:members_raw_changelog'
---------------------------------------------------------
Finished importing 0 Firestore rows to BigQuery
---------------------------------------------------------
Error importing Collection to BigQuery: Error: EPERM: operation not permitted, unlink 'C:\Users\fullo\AppData\Roaming\npm-cache\_npx\10884\node_modules\@firebaseextensions\fs-bq-import-collection\node_modules\grpc\src\node\extension_binary\node-v64-win32-x64-unknown\grpc_node.node'

Also I looked at the script here:
https://github.com/firebase/extensions/blob/master/firestore-bigquery-export/scripts/import/src/index.ts

And the error is being generated at line 187.
It looks like it may throw this error if no data is found or if can't access the project/collection.

[Ehesp]

Looks like a Windows permission issue. I'll get a local Window environment running and give it a go.

[madmacc]

@Ehesp Much appreciated. It would be good to know if it actually works so I can continue to try and resolve. I have 2 other computers including a Mac that I can try on but will take a bit of time setting up.

Among other things I tried:
node v8 and v10 (currently on v10).
I tried using task manager to make sure no other versions on node were running and locking it.
I deleted and reinstalled all my npm packages several times.

[madmacc]

Any update on this? I would like to know if it is working for others using Windows. If so then I can justify spending some more time trying to get it working.

[chenlevy]

Same error here, any workaround?

[madmacc]

@chenlevy Not yet. I am still waiting to see if anyone has an answer. Others have said it works on a Mac so I may try that.

[chenlevy]

@madi8229 I think I figured it out, the problem is with colons in the cursor pointer file name mentioned in the script. Windows file names shouldn't contain colons. I've changed it to underscores and run it locally and it worked perfectly.

131 Also opened a pull request

[Ehesp]

Fixed via https://github.com/firebase/extensions/pull/131

[iDeliverApps]

I am still having this issue with Mac. No problem uploading schema files and having data populate against those, but I can't get any other result for import than Error unlinking journal file '/Users/.... ... Error: ENOENT: no such file or directory, unlink '/Users/....

`? What is your Firebase project ID? ideliver-cloud
? What is the path of the the Cloud Firestore Collection you would like to import from? (This may, or may not, be the same Collection for which you p
lan to mirror changes.) idriver-app\events\incoming
? What is the ID of the BigQuery dataset that you would like to use? (A dataset will be created if it doesn't already exist) ideliver_app
? What is the identifying prefix of the BigQuery table that you would like to import to? (A table will be created if one doesn't already exist) incom
ing
? How many documents should the import stream into BigQuery at once? 300
Importing data from Cloud Firestore Collection: idriver-app\events\incoming, to BigQuery Dataset: ideliver_app, Table: incoming_raw_changelog
Error unlinking journal file /Users/tim/.npm/_npx/4672/lib/node_modules/@firebaseextensions/fs-bq-import-collection/lib/from-idriver-app\events\incoming-to-ideliver-cloud_ideliver_app_incoming_raw_changelog after successful import: Error: ENOENT: no such file or directory, unlink '/Users/tim/.npm/_npx/4672/lib/node_modules/@firebaseextensions/fs-bq-import-collection/lib/from-idriver-app\events\incoming-to-ideliver-cloud_ideliver_app_incoming_raw_changelog'
---------------------------------------------------------
Finished importing 0 Firestore rows to BigQuery
---------------------------------------------------------`

Any help would be greatly appreciated

[mustafaekim]

I am having the same error on a Windows machine