Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Consider using upstream arrow-avro reader #14097

Open
Tracked by #14096
alamb opened this issue Jan 12, 2025 · 2 comments
Open
Tracked by #14096

Consider using upstream arrow-avro reader #14097

alamb opened this issue Jan 12, 2025 · 2 comments
Assignees
Labels
enhancement New feature or request

Comments

@alamb
Copy link
Contributor

alamb commented Jan 12, 2025

Is your feature request related to a problem or challenge?

Currently DataFusion has its own avro --> arrow implementation in: https://github.com/apache/datafusion/blob/54a5d3fd3f98997f048c2e3e7ba67d7bfe2f124d/datafusion/core/src/datasource/avro_to_arrow/mod.rs

This was added way back in the day by @Igosuki and @jorgecarleitao in

Recently there is work upstream in arrow-rs to add additional avro support:

For example, the recent PR from @jecsand838

Describe the solution you'd like

One arrow-avro is more mature, it would be great to consider removing the DataFusion specific implementation and using the upstream arrow-avro one

This would allow DataFusion to take advantage of the upstream work and keep our codebase a bit simpler

Describe alternatives you've considered

No response

Additional context

No response

@alamb alamb added the enhancement New feature or request label Jan 12, 2025
@getChan
Copy link
Contributor

getChan commented Jan 19, 2025

take

@getChan
Copy link
Contributor

getChan commented Jan 25, 2025

I'm waiting for the arrow-avro PR below as it might include changes to the public API.
apache/arrow-rs#6965
I will resume related work once this PR is merged and arrow-avro is released.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants