Could not find example or guideline for the partitioning using parquet.NET? Is partitioning supported? #527
Unanswered
himensuthar
asked this question in
Q&A
Replies: 1 comment
-
Spark is a compute engine, and this is a library. The way Spark does partitioning is applying some logic and writing to separate parquet files using java library. In short the answer is no, it's outside of the scope of parquet libraries to do that, regardless of the platform or language. |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Hi, I see Apache Spark having the inbuilt method for the partioning the parquet file using partitionBy method. as below
df.write.partitionBy("column_name").parquet("path_to_output_directory")
I can partition data manually by creating subdirectories based on the partition keys and storing Parquet files accordingly.
but, I am looking similar functinality as Apache Spak into Parquet.NET, but could not found example or guideline ?
Beta Was this translation helpful? Give feedback.
All reactions