104 save intermediate gardens #110
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…dfs in run_calculate_garden_size.py
…e_garden_size.py`
… record as an interim file
…alculate_garden_size.py
…_calculate_garden_size.py
…e_garden_size.py`
crispy-wonton
commented
Jan 16, 2025
Comment on lines
+230
to
231
| # Final round of deduplication | ||
| epc_gardens_df = garden_size.deduplicate_df_garden_size(epc_gardens_df) |
There was a problem hiding this comment.
Note, this final round of deduplication will mean that the median values we created in the loop above for duplicates within interim files will be used to generate another median. So the final deduplication for any duplicates that remain at this point will calculate the median of the median & other individual values. This is not optimal. Do you have a suggestion for how we could improve this?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #104
Description
Update script that calculates garden size to save intermediate results after every 100 file matches, and refactor deduplication process to be less memory intensive. These changes are to help run the script without killing the process, which is very memory intensive as there are ~20million gardens calculated before deduplication.
asf_heat_pump_suitability/pipeline/run_scripts/run_calculate_garden_size.py.Note: this script was run successfully on a
g3s.xlargeEC2 instance with 50GiB volume. The process was killed during deduplication after successfully saving out interim results, so I made edits to the script (which are in the final version here) and ran that part separately.Instructions for Reviewer
Please could you read through the code and ensure that files are saved out correctly. The main thing to pay attention to is ensuring that there is a) no unintended duplication of data (e.g. saving files [1-100, 1-200] instead of [1-100, 101-200]); and b) no unintended loss of data (e.g. resetting
epc_gardensto empty list when not appropriate, or missing the first or final file matches).To run:
python -i asf_heat_pump_suitability/pipeline/run_scripts/run_calculate_garden_size.py --epc s3://asf-daps/lakehouse/processed/epc/old/deduplicated/processed_dedupl-0.parquet -y 2023 -q 4 -n ewsYou can edit and/or test run the script if you like just to check it works, but I obviously wouldn't suggest running the whole thing!
Checklist:
notebooks/pre-commitand addressed any issues not automatically fixeddevREADMEs