Firebase-tools: Feature: make firebase exports cross-platform

[akshaybabloo]

Creating an export of windows and importing it on macOS gives an IO error.

[REQUIRED] Environment info

firebase-tools: 8.4.3

Platform: macOS and Windows

[REQUIRED] Steps to reproduce

1. Create an export on windows and then import it in mac.

[REQUIRED] Expected behavior

1. Should be able to run with all the imports.

[REQUIRED] Actual behavior

I get an IO error:

Exception in thread "main" com.google.cloud.datastore.core.exception.DatastoreException: /Users/akshayrajgollahalli/Documents/GitHub/accounting-ui/mock-data/firestore_export/all_namespaces\all_kinds\all_namespaces_all_kinds.export_metadata (No such file or directory)
    at com.google.cloud.datastore.emulator.impl.ExportImportUtil.parseBackupFile(ExportImportUtil.java:312)
    at com.google.cloud.datastore.emulator.impl.ExportImportUtil.fetchEntities(ExportImportUtil.java:60)
    at com.google.cloud.datastore.emulator.firestore.CloudFirestore.main(CloudFirestore.java:89)
Caused by: java.io.FileNotFoundException: /Users/akshayrajgollahalli/Documents/GitHub/accounting-ui/mock-data/firestore_export/all_namespaces\all_kinds\all_namespaces_all_kinds.export_metadata (No such file or directory)
    at java.base/java.io.FileInputStream.open0(Native Method)
    at java.base/java.io.FileInputStream.open(FileInputStream.java:212)
    at java.base/java.io.FileInputStream.<init>(FileInputStream.java:154)
    at com.google.cloud.datastore.emulator.impl.ExportImportUtil.parseBackupFile(ExportImportUtil.java:308)
    ... 2 more

[google-oss-bot]

This issue does not seem to follow the issue template. Make sure you provide all the required information.

[samtstern]

@akshaybabloo thanks for the report! This looks like a bug, we'll have to dig into it.

[yuchenshi]

Looking at the path printed out: .../firestore_export/all_namespaces\all_kinds\all_namespaces_all_kind..., it contains both forward and backward slashes. Is this an issue with our exports metadata file written by the Firebase CLI? Assigning to @samtstern for further triaging.

[samtstern]

@yuchenshi so I did find one place we have a likely bug in the code:
https://github.com/firebase/firebase-tools/blob/e10851b3b56a0473f4e3f337babcd4616e93f484/src/emulator/hubExport.ts#L57

That should use path.join(). But that's actually not the source of the issue. The problem is within the Firestore export we have files like this (excuse the binary):

$ cat data/firestore_export/firestore_export.overall_export_metadata 
�mDN3�"O
M
Aall_namespaces\all_kinds\all_namespaces_all_kinds.export_metadata osamstern@samstern-macbookpro3 tmp.3ZGMX0TU 

The overall export metadata points to a path with \ because the export was made on Windows. When the Firestore Emulator tries to read that next file (all_namespaces\all_kinds\...) it gets the FileNotFoundException.

I'm actually not sure what the best path forward is here, do you know the best practice for handling this kind of thing? Also I have attached a .zip of an export with a single Firestore document made on my Windows PC that you can use for testing.

data.zip

[yuchenshi]

I've opened b/160784464 internally to track the work needed in the Firestore Emulator.

[floogulinc]

It looks like the path in firestore_export/firestore_export.overall_export_metadata could simply always use forward slashes. Windows has no issues using an export from *nix. In our case we simply copied only that file from an export done on Linux to fix our export from Windows to work cross-platform.

[yuchenshi]

@floogulinc That's a great observation. I'll be mostly likely going down that path when I get more time to work on it.

[aabccd021]

Any progress on this?

[HenryVillavicencio]

I follow these steps to achieve import Firestore data from the windows machine on mac.

1) run firebase emulators:start --import=./mydir --export-on-exit
2) on firebastore emulator ui create a doc
3) stop emulator
4) copy all_namespaces_all_kinds.export_metadata and output-0 from your previous dir to mydir
5) run firebase emulators:start --import=./mydir --export-on-exit and now you have imported the data :)

[sFerreiraDev]

Hey!
I just wanted to say that this issue is still happening between Ubuntu (importing) and Windows (exporting).
I'm not opening another issue as it was said in this one 2433 to consolidate here, but I'll leave info to help with the reproduction of it:

[REQUIRED] Environment info

firebase-tools: 8.16.2

Platform: Ubuntu 20.04.1 LTS (importing) and Windows 10 (exporting)

[REQUIRED] Steps to reproduce

Create an export on windows and then import it in Ubuntu.

[REQUIRED] Expected behaviour

Should be able to run with all the imports.

[REQUIRED] Actual behaviour

I get an IO error:

Exception in thread "main" com.google.cloud.datastore.core.exception.DatastoreException: /path/to/project/e2e-testing/emulator_data/minimal_rtdb_populate/firestore_export/all_namespaces\all_kinds\all_namespaces_all_kinds.export_metadata (No such file or directory)
    at com.google.cloud.datastore.emulator.impl.ExportImportUtil.parseBackupFile(ExportImportUtil.java:316)
    at com.google.cloud.datastore.emulator.impl.ExportImportUtil.fetchEntities(ExportImportUtil.java:62)
    at com.google.cloud.datastore.emulator.firestore.CloudFirestore.main(CloudFirestore.java:89)
Caused by: java.io.FileNotFoundException: /home/acro-comp/Dev/acro-companion/e2e-testing/emulator_data/minimal_rtdb_populate/firestore_export/all_namespaces\all_kinds\all_namespaces_all_kinds.export_metadata (No such file or directory)
    at java.base/java.io.FileInputStream.open0(Native Method)
    at java.base/java.io.FileInputStream.open(FileInputStream.java:211)
    at java.base/java.io.FileInputStream.<init>(FileInputStream.java:153)
    at com.google.cloud.datastore.emulator.impl.ExportImportUtil.parseBackupFile(ExportImportUtil.java:312)
    ... 2 more