From fff064ed518bcb8a809094c6d9b6b2f9e96caf6d Mon Sep 17 00:00:00 2001 From: Olivier Leroy Date: Fri, 16 Aug 2024 11:03:15 -0400 Subject: [PATCH] correct typo and remove not needed space --- docs/index.html | 32 ++++++++++++++++++++++++++++++-- docs/listings.json | 3 ++- docs/posts/awesomejq/index.html | 21 +++++++++++---------- docs/search.json | 4 ++-- posts/awesomejq/index.qmd | 18 +++--------------- posts/fcc-duckdb/index.qmd | 8 ++++++++ 6 files changed, 56 insertions(+), 30 deletions(-) create mode 100644 posts/fcc-duckdb/index.qmd diff --git a/docs/index.html b/docs/index.html index 480d652..44b7132 100644 --- a/docs/index.html +++ b/docs/index.html @@ -311,7 +311,7 @@

-
+

@@ -325,7 +325,7 @@

-If you are manipulating a lot of GeoJSON eatures/objects and want a quick CLI tool to filter and slice them, you should give jq a try! Since there are not many tutorials… +If you are manipulating a lot of GeoJSON features/objects and want a quick CLI tool to filter and slice them, you should give jq a try! Since there are not many tutorials…
@@ -339,6 +339,34 @@

+
+
+

+

+

+
+
+

+Awesome jq and GeoJSON +

+
+ +
+
+ +
+
+
+ +
+Mar 10, 2024 +
+
+Olivier Leroy +
+ +
+
No matching items diff --git a/docs/listings.json b/docs/listings.json index f6710c4..8a5925f 100644 --- a/docs/listings.json +++ b/docs/listings.json @@ -6,7 +6,8 @@ "/posts/formd_intro/index.html", "/posts/urisa-2023/index.html", "/posts/mapping_rural_tips/index.html", - "/posts/awesomejq/index.html" + "/posts/awesomejq/index.html", + "/posts/fcc-duckdb/index.html" ] } ] \ No newline at end of file diff --git a/docs/posts/awesomejq/index.html b/docs/posts/awesomejq/index.html index a5057d2..1ad3098 100644 --- a/docs/posts/awesomejq/index.html +++ b/docs/posts/awesomejq/index.html @@ -150,6 +150,7 @@

Awesome jq and GeoJSON

geojson
json
spatial data
+
broadband
@@ -196,7 +197,7 @@

On this page

-

If you are manipulating a lot of GeoJSON eatures/objects and want a quick CLI tool to filter and slice them, you should give jq a try! Since there are not many tutorials that exist on using jq to manage objects in the GeoJSON family, we hope that these few tricks will help you on your learning journey.

+

If you are manipulating a lot of GeoJSON features/objects and want a quick CLI tool to filter and slice them, you should give jq a try! Since there are not many tutorials that exist on using jq to manage objects in the GeoJSON family, we hope that these few tricks will help you on your learning journey.

@@ -229,9 +230,9 @@

On this page

  • data/vt-bb.geojson is the path of our file as the last argument

  • '.features | length' is a jq filter. Remember, we are in the shell and whitespace has meaning. Not quoting it would mean that we have 4 arguments when we just have 2.

      -

    • .features will return the json array containing every feature

      • -

    • | will pipe the stream of features into a new filter

      • -

    • length is a built-in function that behaves differently depending the object input. Given an array, it will return the number of elements

      • +
    • .features will return the json array containing every feature
      • +
    • | will pipe the stream of features into a new filter
      • +
    • length is a built-in function that behaves differently depending the object input. Given an array, it will return the number of elements
    • @@ -296,9 +297,9 @@

    Second make th

  • (.features[0].properties | keys_unsorted) here nothing new we added parentheses to enforce precedence. We are getting the header of our csv

  • (.features[].properties | to_entries | map(.value)) :

      -

    • we are starting from all our properties (not the first one)

      • -

    • passing it to to_entries convert our object to multiple objects with “key” / “value” (see margin)

      • -

    • finally, map(.value) gets all “value” for every selected features

      • +
    • we are starting from all our properties (not the first one)
      • +
    • passing it to to_entries convert our object to multiple objects with “key” / “value” (see margin)
      • +
    • finally, map(.value) gets all “value” for every selected features
    • @@ -320,9 +321,9 @@

    Second make th

    We have just explored the surface! jq can help to filter some specific features:

      -

    • every geometries “served” in our file?

      • -

    • the first node in every geometries)?

      • -

    • etc!

      • +
    • every geometries “served” in our file?
      • +
    • the first node in every geometries)?
      • +
    • etc!

    jq is a generic tool for filtering json and lot of people are following the JSON spec in GeoJSON, so we can build on top of all their monumental work!

    diff --git a/docs/search.json b/docs/search.json index 436a334..ce640c2 100644 --- a/docs/search.json +++ b/docs/search.json @@ -25,7 +25,7 @@ "href": "index.html", "title": "Recent posts", "section": "", - "text": "The top 10 micropolitan areas for raising venture capital\n\n\n\n\n\n\n\n\n\n\n\nJul 3, 2024\n\n\nBrittany Kainen\n\n\n\n\n\n\n\n\n\n\n\n\nUsing SEC Form D to estimate venture capital\n\n\n\n\n\n\n\n\n\n\n\nJul 1, 2024\n\n\nBrittany Kainen\n\n\n\n\n\n\n\n\n\n\n\n\nMDA’s URISA-2023 presentation\n\n\n\n\n\n\n\n\n\n\n\nJun 20, 2024\n\n\nOlivier Leroy, Drew Rosebush\n\n\n\n\n\n\n\n\n\n\n\n\nSix tips for mapping rural data\n\n\n\n\n\n\n\n\n\n\n\nMay 24, 2024\n\n\nCamden Blatchly\n\n\n\n\n\n\n\n\n\n\n\n\nAwesome jq and GeoJSON\n\n\n\n\n\n\n\n\n\n\n\nMar 10, 2024\n\n\nOlivier Leroy\n\n\n\n\n\n\nNo matching items" + "text": "The top 10 micropolitan areas for raising venture capital\n\n\n\n\n\n\n\n\n\n\n\nJul 3, 2024\n\n\nBrittany Kainen\n\n\n\n\n\n\n\n\n\n\n\n\nUsing SEC Form D to estimate venture capital\n\n\n\n\n\n\n\n\n\n\n\nJul 1, 2024\n\n\nBrittany Kainen\n\n\n\n\n\n\n\n\n\n\n\n\nMDA’s URISA-2023 presentation\n\n\n\n\n\n\n\n\n\n\n\nJun 20, 2024\n\n\nOlivier Leroy, Drew Rosebush\n\n\n\n\n\n\n\n\n\n\n\n\nSix tips for mapping rural data\n\n\n\n\n\n\n\n\n\n\n\nMay 24, 2024\n\n\nCamden Blatchly\n\n\n\n\n\n\n\n\n\n\n\n\nAwesome jq and GeoJSON\n\n\n\n\n\n\n\n\n\n\n\nMar 10, 2024\n\n\nOlivier Leroy\n\n\n\n\n\n\n\n\n\n\n\n\nAwesome jq and GeoJSON\n\n\n\n\n\n\n\n\n\n\n\nMar 10, 2024\n\n\nOlivier Leroy\n\n\n\n\n\n\nNo matching items" }, { "objectID": "about.html", @@ -39,7 +39,7 @@ "href": "posts/awesomejq/index.html", "title": "Awesome jq and GeoJSON", "section": "", - "text": "If you are manipulating a lot of GeoJSON eatures/objects and want a quick CLI tool to filter and slice them, you should give jq a try! Since there are not many tutorials that exist on using jq to manage objects in the GeoJSON family, we hope that these few tricks will help you on your learning journey.\nIn these examples, we are using a GeoJSON file of Vermont census blocks with attributes related to our work on broadband data. While it is not a deeply nested JSON, it is perfect to illustrate some common use cases.\nA quick check lets us know that it is 94 MB. Not “that” big but still decent.\nFirst, let’s see how many features it has. Here’s how we can approximate that:\nThis is a decent estimate, but we are counting some rows at the top and bottom of the file that are not features (try head -n 5 and tail on it if you are curious).\nWe can also use jq:\nThis is the correct number of blocks! How did that magic work? Let’s decompose our one-liner:" + "text": "If you are manipulating a lot of GeoJSON features/objects and want a quick CLI tool to filter and slice them, you should give jq a try! Since there are not many tutorials that exist on using jq to manage objects in the GeoJSON family, we hope that these few tricks will help you on your learning journey.\nIn these examples, we are using a GeoJSON file of Vermont census blocks with attributes related to our work on broadband data. While it is not a deeply nested JSON, it is perfect to illustrate some common use cases.\nA quick check lets us know that it is 94 MB. Not “that” big but still decent.\nFirst, let’s see how many features it has. Here’s how we can approximate that:\nThis is a decent estimate, but we are counting some rows at the top and bottom of the file that are not features (try head -n 5 and tail on it if you are curious).\nWe can also use jq:\nThis is the correct number of blocks! How did that magic work? Let’s decompose our one-liner:" }, { "objectID": "posts/awesomejq/index.html#jq-and-small-examples", diff --git a/posts/awesomejq/index.qmd b/posts/awesomejq/index.qmd index d752c30..6cc7cb9 100644 --- a/posts/awesomejq/index.qmd +++ b/posts/awesomejq/index.qmd @@ -2,11 +2,11 @@ title: "Awesome jq and GeoJSON" author: "Olivier Leroy" date: "2024-03-10" -categories: [cli, code, geojson, json, spatial data] +categories: [cli, code, geojson, json, spatial data, broadband] image: thumbnail.png --- -If you are manipulating a lot of [`GeoJSON`](https://en.wikipedia.org/wiki/GeoJSON) eatures/objects and want a quick CLI tool to filter and slice them, you should give jq a try! +If you are manipulating a lot of [`GeoJSON`](https://en.wikipedia.org/wiki/GeoJSON) features/objects and want a quick CLI tool to filter and slice them, you should give jq a try! Since there are not many tutorials that exist on using jq to manage objects in the GeoJSON family, we hope that these few tricks will help you on your learning journey. @@ -44,16 +44,12 @@ jq '.features | length' data/vt-bb.geojson This is the correct number of blocks! How did that magic work? Let’s decompose our one-liner: - `jq` here we are calling jq like any other CLI tool - - `data/vt-bb.geojson` is the path of our file as the last argument - - `'.features | length'` is a **jq filter**. Remember, we are in the shell and whitespace has meaning. Not quoting it would mean that we have 4 arguments when we just have 2. - `.features` will return the json array containing every feature - - `|` will pipe the stream of features into a new filter - - `length` is a built-in function that behaves differently depending the object input. Given an array, it will return the number of elements @@ -153,13 +149,10 @@ jq -r '(.features[0].properties | keys_unsorted), (.features[].properties | to_e ``` - `(.features[0].properties | keys_unsorted)` here nothing new we added parentheses to enforce precedence. We are getting the header of our csv - - `(.features[].properties | to_entries | map(.value))` : * we are starting from all our properties (not the first one) - * passing it to `to_entries` convert our object to multiple objects with "key" / "value" (see margin) - * finally, `map(.value)` gets all "value" for every selected features ::: {.column-margin} @@ -179,17 +172,12 @@ jq -r '(.features[0].properties | keys_unsorted), (.features[].properties | to_e ``` ::: - - Finally `@csv` convert to a csv and we redirect the output later in `data/sample.csv` - We have just explored the surface! `jq` can help to filter some specific features: - - - every geometries "served" in our file? - + - every geometries "served" in our file? - the first node in every geometries)? - - etc! `jq` is a generic tool for filtering json and lot of people are following the JSON spec in GeoJSON, so we can build on top of all their monumental work! diff --git a/posts/fcc-duckdb/index.qmd b/posts/fcc-duckdb/index.qmd new file mode 100644 index 0000000..9c44b39 --- /dev/null +++ b/posts/fcc-duckdb/index.qmd @@ -0,0 +1,8 @@ +--- +title: "Fast analytics on FCC (big) dataset" +author: "Olivier Leroy" +date: "2024-08-16" +categories: [cli, code, duckDB, broadband] +image: thumbnail.png +--- +