Connecting the BigQuery Sync Firebase extension

[flauc]

https://github.com/Jaspero/bigquery-sync-firebase-extension

This extension is an ETL process for synchronizing data between bigquery and firestore. Put simply this extension makes it easy to move data between firestore and bigquery. It should be used when ever you need to create analytics views and any kind of business intelligence and you're using firestore as your transactional database.

The way it works is that it configures everything in bigquery, which is to say it creates two new tables for the collection you want to sync with bigquery. The first table keeps track of any changes that happen in firestore while the second table is the one with easily readable synced data. This is the one that should be used for any queries while the tracking table is just used internally. The data in the tracking table expires monthly.

Install

1. You need to clone the repository locally first.
   git clone [email protected]:Jaspero/bigquery-sync-firebase-extension.git
2. Then you need to install dependencies npm ci --prefix functions and build the library npm run build --prefix functions
3. Now run this command in the repository of the firebase project where you want the extension installed:
   firebase ext:install /path/to/bigquery-sync-firebase-extension
4. The install guide will walk you through configuring everything you'll need.
5. Lastly you need to deploy the extension to the live project:
   firebase deploy --only extensions

Alternatively, if you want to install the extension with a non interactive CLI command you can create an extension file: extensions/biquery-sync.env that looks something like this:

BACKFILL=yes
COLLECTION_PATH=users
DATASET_ID=firestore_sync
DATASET_LOCATION=eu
FIELD_MAP=currentStep|NUMERIC,name|STRING,smoking|STRING,gender|STRING
firebaseextensions.v1beta.function/location=europe-west1
FUNCTION_LOCATION=europe-west1
SCHEDULE=0 0 * * *
TABLE_ID=users
TIME_ZONE=UTC

You'll also need to add this part to your firebase.json file as well:

  "extensions": {
    "biquery-sync": "../bigquery-sync-firebase-extension/"
  }

And install it with:

firebase ext:install /path/to/bigquery-sync-firebase-extension --params=path/to/biquery-sync.env

The above is all you need to get everything connected but a more detailed guide can be found here:

https://firebase.google.com/docs/extensions/publishers/get-started#deploy

You can also find further instructions for editing, deleting and re-installing the extension here:

https://firebase.google.com/docs/extensions/manage-installed-extensions?interface=cli