Add Parse JSON Notices Section #2669
Conversation
Vidushi-GitHub
requested review from
lpsinger,
jracusin,
dakota002 and
tylerbarna
Vidushi-GitHub
assigned Vidushi-GitHub and tylerbarna and unassigned tylerbarna and Vidushi-GitHub
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #2669 +/- ##
========================================
- Coverage 6.06% 5.93% -0.14%
========================================
Files 171 171
Lines 4334 4329 -5
Branches 476 474 -2
========================================
- Hits 263 257 -6
- Misses 4069 4070 +1
Partials 2 2 ☔ View full report in Codecov by Sentry. |
app/routes/docs.client.samples.md
Outdated
| } | ||
| ``` | ||
|
|
||
| In your data production pipeline, you can use the encoding steps to convert your file to a bytestring and set the value of the property to this bytestring. See [non-JSON data](https://json-schema.org/understanding-json-schema/reference/non_json_data.html) for more information. |
There was a problem hiding this comment.
Great sample code, but not enough context here to understand what these instructions are for.
There was a problem hiding this comment.
I have tried to expand the instructions.
Could you please elaborate more what's missing?
There was a problem hiding this comment.
Pick an example of an actual notice type that contains embedded HEALPix data. Show how to:
- subscribe to and receive that notice type
- decode the Kafka message as JSON
- decode base64 text to bytes
- read the bytes as a FITS file
And then the user is ready to go on to the sky maps section! (I think that this is turning into a step-by-step tutorial, which will be awesome.)
There was a problem hiding this comment.
LVK JSON file is tested for decode base64 and write fits file.
|
This encoding information needs to also be referenced somewhere in the creating JSON notices documentation. |
|
Add example of pulling message off the network and getting bytes out. |
Vidushi-GitHub
force-pushed
the
ParseJSON
branch
from
261b6ab to
3f09546
Compare
app/routes/docs.client.samples.md
Outdated
| } | ||
| ``` | ||
|
|
||
| In your data production pipeline, you can use the encoding steps to convert your file to a bytestring and set the value of the property to this bytestring. See [non-JSON data](https://json-schema.org/understanding-json-schema/reference/non_json_data.html) for more information. |
There was a problem hiding this comment.
Pick an example of an actual notice type that contains embedded HEALPix data. Show how to:
- subscribe to and receive that notice type
- decode the Kafka message as JSON
- decode base64 text to bytes
- read the bytes as a FITS file
And then the user is ready to go on to the sky maps section! (I think that this is turning into a step-by-step tutorial, which will be awesome.)
Vidushi-GitHub
force-pushed
the
ParseJSON
branch
from
c2e6704 to
e9225b0
Compare
|
@dakota002 please provide your suggestions. |
Co-authored-by: Dakota Dutko <[email protected]>
app/routes/docs.client.samples.md
Outdated
| # Function to validate Base64 strings | ||
| def is_base64(s): | ||
| try: | ||
| base64.b64decode(s, validate=True) | ||
| return True | ||
| except Exception: | ||
| return False | ||
|
|
||
| # Output: A Base64 encoded bytestring, e.g., b'a1512dabc1b6adb3cd1b6dcb6d4c6......' | ||
| with open("path/to/encoded_file.txt", 'wb') as encoded_file: | ||
| encoded_file.write(encoded_string) | ||
| # Validate the skymap string | ||
| if not is_base64(skymap_string): | ||
| print("Invalid Base64 string.") | ||
| continue |
There was a problem hiding this comment.
We can remove it, it's just a check if file wasn't corrupted.
app/routes/docs.client.samples.md
Outdated
| import base64 | ||
|
|
||
| # Set Kafka Topic and Producer Configuration | ||
| TOPIC = "gcn.circulars" |
There was a problem hiding this comment.
This doesn't seem like the right topic.
| producer.flush() | ||
| ``` | ||
|
|
||
| See [non-JSON data](https://json-schema.org/understanding-json-schema/reference/non_json_data.html) for more information. |
There was a problem hiding this comment.
more information about what?
Co-authored-by: Judy Racusin <[email protected]>
This document includes how to parse, encode, decode the JSON Schema files.
Resolves: #2149
Previous branch (#2506) had 100+ commits behind, rebase was messy and touched many files.
Feedbacks need to be included:
-Provide hyperlinks to these Python methods.
-Please provide sample code for parsing Kafka records in a message consuming loop