BACKEND: Make scraped hearing data available on web (push to database with code)

[bbrewington]

No description provided.

[BennyJW]

Has there been any discussion of how to do this? One option is to deploy to Google Cloud Run. Use a scheduler to activate maybe once a day. Load the resulting CSV into Cloud Storage. Then load into Google BigQuery. This would be serverless and easily within the Google free tier.

And github actions to trigger a refresh of the Cloud Run docker image.

[bbrewington]

@BennyJW I think once per day would be a good start. any thoughts on picking between these options? solid ideas, and feel free to spin up a test...maybe share details here in comments if you take a stab at it?

Options:

1. Scraper --> CSV --> Cloud Storage --> BigQuery
2. Scraper --> JSON --> Firebase

I might be conflating phone number storage / API call, with scraped data storage / API call

[BennyJW]

@bbrewington These are good options. My inclination would be #1, because then we can upload the daily raw data to BigQuery, and use SQL to process/extract the data we need. Particularly if as we go we decide we need data from other sources.

[abrie]

See PR #26, which permits an implementation of option 1; but without the cloud storage step.