Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add APIs to support incremental query impl #272

Merged
merged 4 commits into from
Jan 29, 2025

Conversation

xushiyan
Copy link
Member

@xushiyan xushiyan commented Jan 29, 2025

Description

  • Add new table API get_file_slices_between() to support reading incremental file slices for engine integration
  • Add time range configs for file group reader to support filtering records and log file scanning
    • hoodie.read.file_group.start_timestamp
    • hoodie.read.file_group.end_timestamp
  • Remove hoodie.read.as.of.timestamp from configs in favor of passing time travel timestamp via API
  • Refactor the table APIs impl to provide clearer flow of reading file slices
    • Push down the logic of checking base file only and composing instant range to file group reader
  • Add the corresponding Python APIs

Closes #271

How are the changes test-covered

  • N/A
  • Automated tests (unit and/or integration tests)
  • Manual tests
    • Details are described below

@xushiyan xushiyan added feature python Related to Python codebase rust Related to Rust codebase labels Jan 29, 2025
@xushiyan xushiyan added this to the release-0.3.0 milestone Jan 29, 2025
Copy link

codecov bot commented Jan 29, 2025

Codecov Report

Attention: Patch coverage is 93.85965% with 7 lines in your changes missing coverage. Please review.

Project coverage is 91.77%. Comparing base (56fa681) to head (ac54374).
Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
crates/core/src/file_group/reader.rs 93.10% 4 Missing ⚠️
crates/core/src/table/mod.rs 93.18% 3 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #272      +/-   ##
==========================================
+ Coverage   91.02%   91.77%   +0.74%     
==========================================
  Files          41       41              
  Lines        2050     2078      +28     
==========================================
+ Hits         1866     1907      +41     
+ Misses        184      171      -13     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@xushiyan xushiyan merged commit 9488ab6 into apache:main Jan 29, 2025
10 checks passed
@xushiyan xushiyan deleted the provide-api-for-incr-read-impl branch January 29, 2025 21:55
@xushiyan xushiyan mentioned this pull request Jan 30, 2025
16 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature python Related to Python codebase rust Related to Rust codebase
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Table APIs to support implementing Incremental reads in engine integration
1 participant