Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Exports an
AppendBlock
function whichAppend
uses. Our use case is that we have a go routine that sends batches of[]map[string]interface{}
over a channel and a reader which thenAppend
s it to a block. This forces us to compress and append smaller blocks for each batch we receive. This code change allows us to build up the block over time, by converting (usingcodec
) on the fly, appending to a block, and compressing at the same time. We can then compress all data into a single data block.In the worst case, calling
Append
many times was more than doubling, and almost tripling, the actual file sizes compared to calling Append a few times per file.