You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
It was reported on Matrix that our parquet exports are not compressed:
I just executed parquet meta code_0_100000.parquet and from the output it looks like that those parquet files support compression.. but it looks like those are not compressed ones .. e.g.
Row group 9: count: 10000 8,246 kB records start: 781314461 total(compressed): 80,530 MB total(uncompressed):80,530 MB
just compressed with bz2 to get a feeling how much compression could help
-rw-r--r--@ 1 alex staff 826M Sep 30 15:10 code_0_100000.parquet
-rw-r--r--@ 1 alex staff 165M Sep 30 20:24 code_0_100000.parquet.bz2
We should check why our parquet exports are not compressed and how we can make them compressed.
The text was updated successfully, but these errors were encountered:
It was reported on Matrix that our parquet exports are not compressed:
We should check why our parquet exports are not compressed and how we can make them compressed.
The text was updated successfully, but these errors were encountered: