Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BDP-25893: OH table PARTITIONED BY DATE #278

Merged
merged 7 commits into from
Jan 22, 2025
Merged

Conversation

HotSushi
Copy link
Collaborator

@HotSushi HotSushi commented Jan 10, 2025

Summary

Before this PR, OH didn't support following SQL:

spark.sql("CREATE TABLE db.table (data string) PARTITIONED BY (datefield DATE)").show()

After this PR, it does.

Details

OH server achieves this by considering DATE as a clustering column, just like string (say with value "2025-01-01").

There was a design choice to consider DATE as a timestamp column, but it would require major redesign on our REST TimeSpec APIs. Considering as a clustering column, doesn't miss-out on any optimization strategies.

Changes

  • Tests

For all the boxes checked, please include additional details of the changes made in this pull request.

Testing Done

✅ Added unit tests
✅ Build succeeds

  • Manually Tested on local docker setup. Please include commands ran, and their output.
  • Added new tests for the changes made.
  • Updated existing tests to reflect the changes made.
  • No tests added or updated. Please explain why. If unsure, please feel free to ask for help.
  • Some other form of testing like staging or soak time in production. Please explain.

For all the boxes checked, include a detailed description of the testing done for the changes made in this pull request.

Additional Information

  • Breaking Changes
  • Deprecations
  • Large PR broken into smaller PRs, and PR plan linked in the description.

For all the boxes checked, include additional details of the changes made in this pull request.

Copy link
Collaborator

@autumnust autumnust left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks, LGTM

@HotSushi HotSushi merged commit 7f66a73 into linkedin:main Jan 22, 2025
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants