-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* demo report * fix local package * crawl reports tag triggered * timeseries added * split tables * lint * tech report tables * check tech report sql * missing declaration * formatting * preOps * dataset change * cwv_tech_report tested * tech_reports moved * exporter function draft * fix depependencies * rename * dataset renamed * storage exp draft * date column for histograms * dev flag * gsc export tested * pubsub sink prepared * export fn deployed * order incompatible with partitions * monitoring * lint * event parsing draft * cleanup before inserts * event parsing * partitioned exports * exclude scripts * firestore export draft * optional description * single dataset * move * incremental operations * docs update * firestore dict tested * reports tested * full sql export * trigger params * hashed doc ids * more resources and timeout * extend timeout * gzip * event example * esm * more parallelization improvements * tested batch reports * testing fast deletion * deletion tested * limit concurrency * retries * wait to resolve * tested deployed version * cleanup for test merge * cwv-tech-report to prod db * note to unwrap pubsub payloads * cleanup * lint * revisited template builder * cleanup * tf 6.13 * lint * renamed * aligned timeout with prod * simplify tags
- Loading branch information
1 parent
5f0c2ed
commit ef54451
Showing
41 changed files
with
4,113 additions
and
201 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -3,4 +3,5 @@ node_modules/ | |
|
|
||
| # Terraform | ||
| infra/tf/.terraform/ | ||
| infra/tf/tmp/ | ||
| **/*.zip | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,14 +1,7 @@ | ||
| FN_NAME = dataform-trigger | ||
|
|
||
| .PHONY: * | ||
|
|
||
| start: | ||
| npx functions-framework --target=$(FN_NAME) --source=./infra/dataform-trigger/ --signature-type=http --port=8080 --debug | ||
|
|
||
| tf_plan: | ||
| terraform -chdir=infra/tf init -upgrade && terraform -chdir=infra/tf plan \ | ||
| -var="FUNCTION_NAME=$(FN_NAME)" | ||
| terraform -chdir=infra/tf init -upgrade && terraform -chdir=infra/tf plan | ||
|
|
||
| tf_apply: | ||
| terraform -chdir=infra/tf init && terraform -chdir=infra/tf apply -auto-approve \ | ||
| -var="FUNCTION_NAME=$(FN_NAME)" | ||
| terraform -chdir=infra/tf init && terraform -chdir=infra/tf apply -auto-approve |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
File renamed without changes.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -5,3 +5,8 @@ for (const table of stagingTables) { | |
| name: table | ||
| }) | ||
| } | ||
|
|
||
| declare({ | ||
| schema: 'wappalyzer', | ||
| name: 'apps' | ||
| }) | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,49 @@ | ||
| const pastMonth = constants.fnPastMonth(constants.currentMonth) | ||
|
|
||
| publish('cwv_tech_adoption', { | ||
| schema: 'reports', | ||
| type: 'incremental', | ||
| protected: true, | ||
| bigquery: { | ||
| partitionBy: 'date', | ||
| clusterBy: ['rank', 'geo'] | ||
| }, | ||
| tags: ['crux_ready'] | ||
| }).preOps(ctx => ` | ||
| CREATE TEMPORARY FUNCTION GET_ADOPTION( | ||
| records ARRAY<STRUCT< | ||
| client STRING, | ||
| origins INT64 | ||
| >>) | ||
| RETURNS STRUCT< | ||
| desktop INT64, | ||
| mobile INT64 | ||
| > | ||
| LANGUAGE js AS ''' | ||
| return Object.fromEntries( | ||
| records.map(({client, origins}) => { | ||
| return [client, origins] | ||
| })) | ||
| '''; | ||
| DELETE FROM ${ctx.self()} | ||
| WHERE date = '${pastMonth}'; | ||
| `).query(ctx => ` | ||
| /* {"dataform_trigger": "report_cwv_tech_complete", "date": "${pastMonth}", "name": "adoption", "type": "report"} */ | ||
| SELECT | ||
| date, | ||
| app AS technology, | ||
| rank, | ||
| geo, | ||
| GET_ADOPTION(ARRAY_AGG(STRUCT( | ||
| client, | ||
| origins | ||
| ))) AS adoption | ||
| FROM ${ctx.ref('core_web_vitals', 'technologies')} | ||
| WHERE date = '${pastMonth}' | ||
| GROUP BY | ||
| date, | ||
| app, | ||
| rank, | ||
| geo | ||
| `) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,51 @@ | ||
| const pastMonth = constants.fnPastMonth(constants.currentMonth) | ||
|
|
||
| publish('cwv_tech_categories', { | ||
| schema: 'reports', | ||
| type: 'table', | ||
| tags: ['crux_ready'] | ||
| }).query(ctx => ` | ||
| /* {"dataform_trigger": "report_cwv_tech_complete", "name": "categories", "type": "dict"} */ | ||
| WITH pages AS ( | ||
| SELECT | ||
| root_page, | ||
| technologies | ||
| FROM ${ctx.ref('crawl', 'pages')} | ||
| WHERE | ||
| date = '${pastMonth}' AND | ||
| client = 'mobile' | ||
| ${constants.devRankFilter} | ||
| ),categories AS ( | ||
| SELECT | ||
| category, | ||
| COUNT(DISTINCT root_page) AS origins | ||
| FROM pages, | ||
| UNNEST(technologies) AS t, | ||
| UNNEST(t.categories) AS category | ||
| GROUP BY category | ||
| ), | ||
| technologies AS ( | ||
| SELECT | ||
| category, | ||
| technology, | ||
| COUNT(DISTINCT root_page) AS origins | ||
| FROM pages, | ||
| UNNEST(technologies) AS t, | ||
| UNNEST(t.categories) AS category | ||
| GROUP BY | ||
| category, | ||
| technology | ||
| ) | ||
| SELECT | ||
| category, | ||
| categories.origins, | ||
| ARRAY_AGG(technology ORDER BY technologies.origins DESC) AS technologies | ||
| FROM categories | ||
| JOIN technologies | ||
| USING (category) | ||
| GROUP BY | ||
| category, | ||
| categories.origins | ||
| ORDER BY categories.origins DESC | ||
| `) |
Oops, something went wrong.