Deprecate datafusion.execution.parquet.max_statistics_size
config option
#14172
Labels
enhancement
New feature or request
Is your feature request related to a problem or challenge?
The most recent version of arrow deprecates the
max_statistics_size
parquet option54.0.0
and pyo3 to0.23.3
#14153You can see the rationale from @etseidl here (it was being silently ignored)
WriterProperties
arrow-rs#6884DataFusion also exposes this setting:
https://datafusion.apache.org/user-guide/configs.html
We should deprecate it as well prior to removal
Describe the solution you'd like
The config option is defined in two places:
https://github.com/apache/datafusion/blob/04b6d4d6099f537de91e1b30a391bdfbc3ec36d5/datafusion/common/src/config.rs#L1723-L1722
https://github.com/apache/datafusion/blob/04b6d4d6099f537de91e1b30a391bdfbc3ec36d5/datafusion/common/src/config.rs#L1723-L1722
But I think they are done via macro
Describe alternatives you've considered
We can just wait until the field is removed upstream in arrow and yank it from datafusion too
Additional context
No response
The text was updated successfully, but these errors were encountered: