Extensions: Error exporting data to Bigquery

[brunoschuermans]

[READ] Step 1: Are you in the right place?

Issues filed here should be about bugs for a specific extension in this repository.
If you have a general question, need help debugging, or fall into some
other category use one of these other channels:

• For general technical questions, post a question on StackOverflow
  with the firebase tag.
• For general Firebase discussion, use the firebase-talk
  google group.
• To file a bug against the Firebase Extensions platform, or for an issue affecting multiple extensions, please reach out to
  Firebase support directly.

[REQUIRED] Step 2: Describe your configuration

• Extension name: firestore-bigquery-export
• Extension version: 0.1.4
• Configuration values (redact info where appropriate):
  
  
  
  • Collection path: cashBoxOrders
  
  
  • Table ID: cashBocxOrders
  
  

[REQUIRED] Step 3: Describe the problem

Error when mirroring data to BigQuery TypeError: Cannot read property 'constructor' of null
at Object.traverse.map (/srv/node_modules/@firebaseextensions/firestore-bigquery-change-tracker/lib/bigquery/index.js:75:26)
at walker (/srv/node_modules/traverse/index.js:190:22)
at /srv/node_modules/traverse/index.js:208:29
at Array.forEach ()
at forEach (/srv/node_modules/traverse/index.js:298:31)
at walker (/srv/node_modules/traverse/index.js:203:13)
at /srv/node_modules/traverse/index.js:208:29
at Array.forEach ()
at forEach (/srv/node_modules/traverse/index.js:298:31)
at walker (/srv/node_modules/traverse/index.js:203:13)

Steps to reproduce:

What happened? How can we make the problem occur?
This could be a description, log/console output, etc.

Expected result

That my data is exported correctly

Actual result

Only 52 of the 5123 records are copied successfully.
And when I try again to reexport all, again only the same 52 are exported.

Example of one record exported without problems:
{ "passenger": { "SearchName": [ "CATHERINE", "OLIGMUELLER", "[email protected]", "771634" ], "Age": 38, "LastName": "Oligmueller", "DropOffLocation": "CHUQUIRAGUA LODGE AT THE ENTRANCE TO COTOPAXI NATIONAL PARK", "LastGuide": "Mario Urvina", "NextBus": "2020-03-13", "Pass": "GET TO QUITO QUICK", "Comments": "", "StartPickupLocation": "HANGA ROA HOSTAL,\tBARRIO LA PUNTA, PROVINCIA SANTA ELENA COMUNA MONTANITA 241754", "DiscountCode": "ICD747969", "LastBus": { "_seconds": 1583949600, "_nanoseconds": 0 }, "Language": "EN", "ExtraDiscount": 0, "RouteName": "QUILOTOA - COTOPAXI", "ReservationId": 771634, "createAt": { "_seconds": 1584032258, "_nanoseconds": 275000000 }, "PickupTime": "From 4:15 PM To 4:30 PM", "TotalDue": 0, "DocumentNumber": "CH1HY0C52", "id": "2020-03-12-QUILOTOA - COTOPAXI-771634", "End": "COTOPAXI", "Company": "ECUADOR HOP", "OnHold": false, "Extras": [], "TotalCost": 93.56, "Departure": "16:30", "PreviousBusDepartureDate": { "_seconds": 1583949600, "_nanoseconds": 0 }, "NextBusDisplay": "13/03/2020", "StartPickupTime": "From 2:30 PM To 3:00 PM", "CurrentGuide": "Jonathan Castro", "RouteId": 146, "OrderIndex": 3, "LinkedRouteName": "QUILOTOA - COTOPAXI", "BusFullId": "71d8ef70-6482-11ea-9b44-b72f43f93d50", "DiscountId": 350972, "ReservationStatus": "CONFIRMED", "PreviousBusPickupTime": "From 12:45 PM To 1:00 PM", "TotalPaid": 93.56, "FirstName": "Catherine", "Country": "GERMANY", "StartDate": "2020-03-12", "PreviousBusPickUpLocation": "BIBLIOTECA MUNICPAL (LIBRARY) AT MONTALVO PARK", "PickupLocation": "HOSTAL CHUKIRAWA (OPPOSITE THE MAIN VIEWPOINT)", "ExtraPack": "", "Gender": "Female", "PaymentMethod": "CARD", "ArrivalDate": "2020-03-12", "EmailAddress": "[email protected]", "StartDeparture": "15:00" }, "comment": "", "scriptUpdateAt": { "_seconds": 1587742404, "_nanoseconds": 829000000 }, "option": "", "id": "4HSXI10WqEgIzoTtougD", "currency": "dollars", "quantity": 1, "isDeleted": false, "startDate": { "_seconds": 1584053028, "_nanoseconds": 323000000 }, "updateBy": "[email protected]", "user": { "hasDebt": true, "createAt": { "_seconds": 1581466617, "_nanoseconds": 266000000 }, "password": "jonathan2020", "roleCode": "guide", "guideId": "BKqil0qSo8gSDqbFhQMp", "email": "[email protected]", "agencyCode": "ecuador_hop", "updateAt": { "_seconds": 1583296188, "_nanoseconds": 754000000 }, "id": "Xpoc6PTsGJ86ZjueU0ar", "dollars": 188.70000000000002, "bolivianos": 0, "soles": 0 }, "item": { "createAt": { "_seconds": 1583588955, "_nanoseconds": 467000000 }, "routeId": "cOyTE2Q1INetuOrGckkd", "options": [], "isDebit": false, "name": "COTOPAXI TOUR", "agencyCode": "ecuador_hop", "option": "", "id": "9HF7FJG38R2cOqVbEPmQ", "updateAt": { "_seconds": 1583934069, "_nanoseconds": 694000000 }, "dollars": "40", "isDeleted": false, "bolivianos": "", "startDate": "today", "soles": "", "updateBy": "[email protected]" }, "type": "order", "isDeposited": false, "busFull": { "arrivalDate": "2020-03-12", "otherGuides": [], "createAt": { "_seconds": 1584032200, "_nanoseconds": 680000000 }, "passengerSelected": 8, "route": { "isDeleted": false, "name": "QUILOTOA - COTOPAXI", "agencyCode": "ecuador_hop", "scriptUpdateAt": { "_seconds": 1581432199, "_nanoseconds": 320000000 }, "id": "cOyTE2Q1INetuOrGckkd", "createAt": { "_seconds": 1581432199, "_nanoseconds": 320000000 } }, "invoiceNumber": "", "agencyCode": "ecuador_hop", "id": "71d8ef70-6482-11ea-9b44-b72f43f93d50", "busFullId": "71d8ef70-6482-11ea-9b44-b72f43f93d50", "updateAt": { "_seconds": 1584032295, "_nanoseconds": 598000000 }, "bus": { "capacity": 26, "createAt": { "_seconds": 1581432264, "_nanoseconds": 802000000 }, "isDeleted": false, "licensePlate": "PAB6995", "busCompanyId": "vaKTXlZPiEVOzcc6uFR3", "scriptUpdateAt": { "_seconds": 1582132897, "_nanoseconds": 621000000 }, "model": "INTERNATIONAL", "scriptToBigQueryAt": { "_seconds": 1582132897, "_nanoseconds": 621000000 }, "id": "bl7HCauvUsK6TFXhSICc", "updateAt": { "_seconds": 1581432264, "_nanoseconds": 802000000 }, "busInformation": "PAB6995 / INTERNATIONAL" }, "departureTime": "16:30", "searchGuides": [ "BKqil0qSo8gSDqbFhQMp" ], "isDeleted": false, "startDate": "2020-03-12", "company": { "scriptUpdateAt": { "_seconds": 1581432374, "_nanoseconds": 826000000 }, "id": "vaKTXlZPiEVOzcc6uFR3", "createAt": { "_seconds": 1581432374, "_nanoseconds": 826000000 }, "isDeleted": false, "address": "CALLE PARIS N41-44 Y GRANADOS", "ruc": "1791872827001", "busCompanyName": "C.P.T TRANSPORTES", "agencyCode": "ecuador_hop" }, "guide": { "lastRouteDate": "2020-03-12", "firstName": "Jonathan", "isDeleted": false, "updateBy": "[email protected]", "lastRoute": { "name": "BAÑOS - QUILOTOA", "agencyCode": "ecuador_hop", "scriptUpdateAt": { "_seconds": 1581432199, "_nanoseconds": 320000000 }, "id": "p6aWdE1NlhiSOV0RChfV", "createAt": { "_seconds": 1581432199, "_nanoseconds": 320000000 }, "isDeleted": false }, "fullName": "Jonathan Castro", "dni": "1751397363", "createAt": { "_seconds": 1581019013, "_nanoseconds": 78000000 }, "agencyCode": "ecuador_hop", "scriptUpdateAt": { "_seconds": 1581432173, "_nanoseconds": 711000000 }, "id": "BKqil0qSo8gSDqbFhQMp", "lastName": "Castro" }, "user": { "createAt": { "_seconds": 1572036564, "_nanoseconds": 921000000 }, "password": "123456", "roleCode": "agency_admin", "email": "[email protected]", "agencyCode": "ecuador_hop", "id": "vu0Rg1MIXBqsgtKz97Uk" }, "busFullCounter": "00017163", "drivers": [ { "firstName": "Angel", "isDeleted": false, "busCompanyId": "vaKTXlZPiEVOzcc6uFR3", "updateBy": "[email protected]", "id": "UXiPuvlaj6FAhGMCxdw2", "driverInformation": "Angel Carvajal / Driving License: E", "drivingLicense": "E", "lastName": "Carvajal", "createAt": { "_seconds": 1583255323, "_nanoseconds": 202000000 } } ] }, "price": 40, "createAt": { "_seconds": 1584053070, "_nanoseconds": 654000000 } }

Example of one record not exported (with error):
{ "isDeleted": false, "startDate": { "_seconds": 1581469676, "_nanoseconds": 8000000 }, "updateBy": "[email protected]", "user": { "roleCode": "guide", "password": "alejandrocartagena", "soles": -195, "guideId": "hw0sbYcnnd2VzZE8UAcH", "email": "[email protected]", "agencyCode": "peru_hop", "id": "mzZPtQPwgKaTSN70kMzA", "lastDepositAt": { "_seconds": 1581445187, "_nanoseconds": 758000000 }, "dollars": 0, "hasDebt": false, "createAt": { "_seconds": 1574440895, "_nanoseconds": 517000000 } }, "item": { "updateAt": { "_seconds": 1576876654, "_nanoseconds": 865000000 }, "id": "txWtFFsyThdaTQPHQGci", "dollars": "15", "bolivianos": null, "startDate": "today", "soles": "50", "createAt": { "_seconds": 1576876521, "_nanoseconds": 463000000 }, "routeId": "wyjbV08W9jvA0HIpRCWI", "options": [ { "name": "8AM" }, { "name": "10AM" } ], "isDebit": false, "name": "BALLESTAS ISLAND", "agencyCode": "peru_hop" }, "type": "order", "isDeposited": true, "price": 50, "busFull": { "passengerSelected": 17, "route": { "name": "LIMA - PARACAS (7AM)", "agencyCode": "peru_hop", "id": "wyjbV08W9jvA0HIpRCWI" }, "invoiceNumber": "", "agencyCode": "peru_hop", "scriptUpdateAt": { "_seconds": 1581432913, "_nanoseconds": 320000000 }, "feedbackId": "fRGMXTn5PCXOu2YrjNcE", "id": "a361b750-4c56-11ea-a3db-85de3a0b45d5", "busFullId": "a361b750-4c56-11ea-a3db-85de3a0b45d5", "updateAt": { "_seconds": 1581387820, "_nanoseconds": 773000000 }, "bus": { "licensePlate": "D1K958", "busCompanyId": "422e8860-039f-11e9-a19e-47235d75a55e", "company": "DC TRAVEL SA", "model": "IRIZAR I6", "id": "iT5cYHBbLowweuT7TWoA", "updateAt": { "_seconds": 1569510930, "_nanoseconds": 214000000 }, "busInformation": "D1K958 / IRIZAR I6", "capacity": 46 }, "departureTime": "07:00", "searchGuides": [ "hw0sbYcnnd2VzZE8UAcH" ], "isDeleted": false, "startDate": "2020-02-11", "company": { "agencyCode": "peru_hop", "busCompanyName": "DC TRAVEL SA", "id": "422e8860-039f-11e9-a19e-47235d75a55e", "updateAt": { "_seconds": 1579618637, "_nanoseconds": 92000000 }, "companyName": "DC TRAVEL SA", "address": "JR, JAZPAMPA 340, CALLAO", "ruc": "20513504137" }, "guide": { "firstName": "Alejandro", "agencyCode": "peru_hop", "fullName": "Alejandro Cartagena", "scriptUpdateAt": { "_seconds": 1581019013, "_nanoseconds": 78000000 }, "id": "hw0sbYcnnd2VzZE8UAcH", "dni": "42698262", "lastName": "Cartagena", "createAt": { "_seconds": 1581019013, "_nanoseconds": 78000000 } }, "user": { "guideId": null, "email": "[email protected]", "agencyCode": "peru_hop", "id": "plfC2HySXoDONNzx6GeG", "isAdmin": true, "roleCode": "agency_admin" }, "hasFeedback": true, "busFullCounter": "00015964", "drivers": [ { "lastName": "Rodriguez", "drivingLicense": "Q09953850", "firstName": "Segundo", "busCompanyId": "422e8860-039f-11e9-a19e-47235d75a55e", "id": "NuVExa8A2uYKyALiWqQS", "driverInformation": "Segundo Rodriguez / Driving License: Q09953850" } ], "arrivalDate": "2020-02-11", "otherGuides": [], "createAt": { "_seconds": 1581432913, "_nanoseconds": 320000000 } }, "createAt": { "_seconds": 1581469742, "_nanoseconds": 558000000 }, "depositId": "jvIT8ft0QXgXZHZMFqvy", "passenger": { "LastGuide": "", "NextBus": "2020-02-12", "Pass": "LIMA TO CUSCO TO LA PAZ", "Comments": "", "StartPickupLocation": "PLAZA MAYOR, CALLAO 125", "DiscountCode": "", "LastBus": "", "Language": "EN", "ExtraDiscount": 0, "RouteName": "LIMA - PARACAS", "ReservationId": 705087, "createAt": { "_seconds": 1581439234, "_nanoseconds": 197000000 }, "PickupTime": "From 5:40 AM To 5:55 AM", "TotalDue": 0, "DocumentNumber": "C1WVG95GR", "id": "2020-02-11-LIMA - PARACAS (7AM)-705087", "End": "PARACAS", "Company": "PERU HOP", "OnHold": false, "Extras": [], "TotalCost": 239.4, "Departure": "07:00", "PreviousBusDepartureDate": "", "NextBusDisplay": "12/02/2020", "StartPickupTime": "From 5:40 AM To 5:55 AM", "CurrentGuide": "Alejandro Cartagena", "RouteId": 60, "OrderIndex": 1, "LinkedRouteName": "LIMA - PARACAS (7AM)", "BusFullId": "a361b750-4c56-11ea-a3db-85de3a0b45d5", "DiscountId": null, "ReservationStatus": "CONFIRMED", "PreviousBusPickupTime": null, "TotalPaid": 239.4, "FirstName": "Rebecca", "Country": "GERMANY", "StartDate": "2020-02-11", "PreviousBusPickUpLocation": null, "PickupLocation": "PLAZA MAYOR, CALLAO 125", "ExtraPack": "", "Gender": "Female", "PaymentMethod": "CARD", "ArrivalDate": "2020-02-11", "EmailAddress": "[email protected]", "StartDeparture": "07:00", "SearchName": [ "REBECCA", "ERHARD", "[email protected]", 705087 ], "Age": 22, "LastName": "Erhard", "DropOffLocation": "KOKOPELLI PARACAS" }, "scriptUpdateAt": { "_seconds": 1587742410, "_nanoseconds": 404000000 }, "option": "8AM", "id": "00DHBXOSyG48Scy7E0Rr", "currency": "soles", "quantity": 1 }

[Ehesp]

Looks like this is regression from https://github.com/firebase/extensions/pull/270 not checking a if the property value is null.

[fabiangruss]

We have the same problem, the firestore-bigquery-export (V0.1.4) and the fs-bq-import-collection script for existing documents both crash when a property value is null. Is there any fix for this?

[mikkelboschmidt]

Same here. Both the export and import scripts have this issue.

[laurenzlong]

The fix has been published in 0.1.6 of the firestore-bigquery-change-tracker NPM package - which is a dependency for the extension. When we do the next release of the firestore-bigquery-export extension, you'll receive the update. If you want to "force" the dependency update now, you can either:

• uninstall and reinstall the extension
• OR reconfigure the extension to force an update