-
Notifications
You must be signed in to change notification settings - Fork 838
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Allow hints for upcasting parquet to arrow integer types #6892
base: main
Are you sure you want to change the base?
Conversation
4d39589
to
d0385d4
Compare
d0385d4
to
8382ab8
Compare
I'm pretty lukewarm on baking in schema coercion into the parquet reader at this level, the interpretation of how to map the parquet data types to arrow is gnarly enough without needing to also perform non-trivial schema adaption. I think it would be better to handle this at a higher level, e.g. as proposed by #6735 |
Oh I think I understand. My initial impression was that arrow-rs/parquet/src/arrow/arrow_reader/mod.rs Lines 276 to 283 in 54dccad
So the nuance is that in the original issue the goal was to supply a hint to a Parquet with missing type hints, whereas in my case I want to override an already-hinted Parquet schema, thus somewhat abusing/overloading this mechanism? This would also suggest that the above description needs to be changed as it is misleading. I guess then that #6735 could also provide a solution to this problem (albeit with more effort). Let me take a look at it. |
There is a related discussion on: |
Which issue does this PR close?
Closes #6891.
Rationale for this change
Allow casting integers so long as the precision isn't lost.
What changes are included in this PR?
Add possible matches for up-casting all integer
DataType
s.Are there any user-facing changes?
None.