Extensions: No schema file found ! error while running the fs-bq-schema-views script

[sidharthmishra11]

[READ] Step 1: Are you in the right place?

yes

[REQUIRED] Step 2: Describe your configuration

• Extension name: firestore-bigquery-export (fs-bq-schema-views script)
• Extension version: Latest

[REQUIRED] Step 3: Describe the problem

I am trying to create BigQuery view of my raw changelog using fs-bq-schema-view script but while running the script I am getting the message 'No Schema files found' & Schema is not getting created in BigQuery.
Note:

1. I have created the raw_changelog table in bigquery using IMPORT script
2. Created the JSON schema file correctly which is being used in the fs-bq-schema-views script.

Steps to reproduce:

What happened? How can we make the problem occur?
npx @firebaseextensions/fs-bq-schema-views \

--non-interactive \
--project=sidh-firestore-demo \
--dataset=bq_fs_sidh_sample \
--table-name-prefix=bq_fs_city_db \
--schema-files=D:/GCP-Firebase/test_schema.json
No schema files found!
done.

Expected result

Schema View to be created in BigQuery.

Actual result

Schema view not getting created in BigQuery.

[russellwheatley]

Hi @sidharthmishra11, the reason why it isn't working is because of the path for your schema file i.e. --schema-files=D:/GCP-Firebase/test_schema.json. I'd recommend executing the command from D:/GCP-Firebase directory and changing schema file argument to a relative path like this: --schema-files=./test-schema.json.

[sidharthmishra11]

Thanks @russellwheatley for your response. I tried using --schema-files=./test-schema.json but still its giving me the same message 'No Schema files found'.

You recommended executing the command from D:/GCP-Firebase. How do I do that from Google cloud shell?

[russellwheatley]

I'm not sure how to do it from Google Cloud shell in all honesty. I know that the No schema files found message means it isn't able to locate your schema. Are you able to run that command locally? As long as you're authenticated, you're able to run that command from your local machine.

[arenphansen]

Hi @russellwheatley, @karayu, I'm having the same issue and have tried for the last two days to no avail to fix it. I have my json file in the same Google Storage Bucket as the Environment for my BigQuery is in: ext-firestore-bigquery-export-fsexportbigquery.

I have tried:

npx @firebaseextensions/fs-bq-schema-views \
  --non-interactive \
  --project myproject \
  --dataset firestore_export \
  --table-name-prefix users_sync \
  --schema-files ./users_sync.json

Does the "./" work if my users_sync.json is in the same directory as the BigQuery reference? Is there a better way to reference a Google Storage Bucket?

I'm also doing this in Google Cloud Shell (I'm new to all of this and it's the only way I understand).
-I did authenticate successfully using: gcloud auth application-default login
-I loaded the required tool: npm install --global npx

However all it keeps telling me is:
"No schema files found!
done."

Any help is greatly appreciated!

[russellwheatley]

Hey @arenphansen, do you have access to a terminal (roughly similar to Google Cloud Shell) on your machine?

Here's an example of how you could run this command from a terminal from a mac:

1. open terminal
2. run touch users_sync.json (creates file)
3. run open users_sync.json (this will open that file just created in a text editor)
4. update users_sync.json file with your desired schema & save.
5. run your command from the terminal:

npx @firebaseextensions/fs-bq-schema-views \
  --non-interactive \
  --project myproject \
  --dataset firestore_export \
  --table-name-prefix users_sync \
  --schema-files ./users_sync.json

What is important to understand is the script is looking for your schema relative to the directory where you execute the command.

It appears that Google Cloud Shell operates differently, but that is just a hunch, I am not familiar with how it works. Hope that helps!

[arenphansen]

Thanks @russellwheatley! I must admit, I had to install Homebrew, upgrade Python to v3 from V2, before I could install Google Cloud's SDK, which were a bit tricky for me as a non-developer in Apple's Terminal program (and switch from TSCH to BASH in Apple's terminal), but once I got through that your trick worked and now I have a working Schema. Just took a little more Googling of solutions along the way :). Thanks for your tip to get me there!

Much appreciated!!

[russellwheatley]

I'm really happy it worked out for you, @arenphansen. You've discovered the secret to developing along the way; Google everything :).

[arenphansen]

Hi @russellwheatley! Let's say, I assigned the wrong type to one of my variables, is there a right way to fix this? I tried changing my json file to the right type, and repeating the process above but at the end it says something like this:

"Provided Schema does not match Table myproject:firestore_export.users_schema_users_changelog. Field lastCronUpdate has changed type from NUMERIC to TIMESTAMP".

Do I always have to start over with a new view if I make a mistake, or is there an easier way to update the current view? I already started building a Google Data Studio dashboard, so would prefer to not have to start over... Thanks!

[russellwheatley]

@arenphansen, if you need to update the generated view schemas, you could do it via the BigQuery console. There is a button called "edit schema" which will allow you to update. You will have to do this with both views I believe.

Alternatively, you could just delete the two views created from the gen-schema-view script in the BigQuery console. Running the script again will recreate them.

[arenphansen]

@russellwheatley , I realized I could update via the BigQuery console but am not as well versed in that language, it's not as simple as the scheme json for me.

The idea of just deleting the two tables and recreating it is brilliant. It's simple and I hadn't thought of that, but I think that's the easiest path forward - thanks so much!