Fix polygon get historical prices #679
Conversation
…s' into fix-polygon-get-historical-prices
|
Important Required App Permission UpdateNoise Reduction ImprovementsThis update requests write permissions for Commit Statuses in order to send updates directly to your PRs without adding comments that spam notifications. Visit our changelog to learn more. Click here to accept the updated permissions To accept the updated permissions, sufficient privileges are required |
brettelliot
temporarily deployed
to
unit-tests
GitHub Actions
Inactive
![korbit-ai[bot]](https://avatars.githubusercontent.com/in/322216?s=60&v=4){kind=small}
There was a problem hiding this comment.
Review by Korbit AI
Korbit automatically attempts to detect when you fix issues in new commits.
| Category | Issue | Fix Detected |
|---|---|---|
 |
Over-querying for legitimate no-trade days ▹ view |
Files scanned
| File Path | Reviewed |
|---|---|
| lumibot/tools/polygon_helper.py | ✅ |
Explore our documentation to understand the languages and file types we support and the files we ignore.
Need a new review? Comment
/korbit-reviewon this PR and I'll review your latest changes.Korbit Guide: Usage and Customization
Interacting with Korbit
- You can manually ask Korbit to review your PR using the
/korbit-reviewcommand in a comment at the root of your PR.- You can ask Korbit to generate a new PR description using the
/korbit-generate-pr-descriptioncommand in any comment on your PR.- Too many Korbit comments? I can resolve all my comment threads if you use the
/korbit-resolvecommand in any comment on your PR.- Chat with Korbit on issues we post by tagging @korbit-ai in your reply.
- Help train Korbit to improve your reviews by giving a 👍 or 👎 on the comments Korbit posts.
Customizing Korbit
- Check out our docs on how you can make Korbit work best for you and your team.
- Customize Korbit for your organization through the Korbit Console.
Current Korbit Configuration
General Settings
Setting Value Review Schedule Automatic excluding drafts Max Issue Count 10 Automatic PR Descriptions ✅ Issue Categories
Category Enabled Naming ✅ Database Operations ✅ Documentation ✅ Logging ✅ Error Handling ✅ Systems and Environment ✅ Objects and Data Structures ✅ Readability and Maintainability ✅ Asynchronous Processing ✅ Design Patterns ✅ Third-Party Libraries ✅ Performance ✅ Security ✅ Functionality ✅ Feedback and Support
Note
Korbit Pro is free for open source projects 🎉
Looking to add Korbit to your team? Get started with a free 2 week trial here
| # Find any dates with nan values in the df_all DataFrame. This happens for some infrequently traded assets, but | ||
| # it is difficult to know if the data is actually missing or if it is just infrequent trading, query for it again. | ||
| missing_dates += df_all[df_all.isnull().all(axis=1)].index.date.tolist() |
There was a problem hiding this comment.
Over-querying for legitimate no-trade days 
Tell me more
What is the issue?
The code treats all days with null values as missing data that needs to be re-queried, but for infrequently traded assets, these null values might legitimately represent days with no trading activity.
Why this matters
This could lead to unnecessary API calls to Polygon for dates that genuinely had no trading activity, potentially hitting rate limits and slowing down the data retrieval process without adding any valuable data.
Chat with Korbit by mentioning @korbit-ai, and give a 👍 or 👎 to help Korbit improve your reviews.
Description by Korbit AI
What change is being made?
Modify the
get_missing_datesfunction withinpolygon_helper.pyto include dates with NaN values as missing and update relevant unit tests accordingly.Why are these changes being made?
These changes address an issue where dates with NaN values were not considered "missing" for infrequently traded assets, thus ensuring data completeness by querying such dates again. The updated tests also now accurately reflect changes in the behavior of caching within
get_price_data_from_polygon.