Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimise read performance using e.g. io::Cursor #10

Open
drewsilcock opened this issue Jan 7, 2022 · 0 comments
Open

Optimise read performance using e.g. io::Cursor #10

drewsilcock opened this issue Jan 7, 2022 · 0 comments
Labels
enhancement New feature or request

Comments

@drewsilcock
Copy link
Owner

Currently we read all of the files into memory in one, including the .dat file which can be several GB. We need a way that:

  • Stream the .dat file when it's needed (the other files are usually a trivial size in comparison so I don't think we need to worry about that).
  • Only read the data through once. This sounds simple but the .cff combined file format complicates this - it's possible to embed the .dat data within the .cff such that there's another file section once the .dat ends (e.g. the .hdr or .inf). For ASCII data format, we have to only read the number of lines we expected from the sample rate specification, i.e. we need to read total_num_samples lines. For binary formats, we have to calculate the total number of bytes expected from the .dat section of the .cff file by using total_num_samples * num_bytes_per_scan where num_bytes_per_scan is 4 + 4 + [size of analog data value] * [number of analog channels] + ceil([number of status channels] / 16) * 2) where [size of analog data value is 2 for binary16 data type (a.k.a. "binary" in the COMTRADE specification - I find this confusing and ambiguous, I presume it's called that for backwards compatibility reasons) and 4 for binary32 and float32 data types. Making this data streaming for the CFF files work with Rust's ownership model is going to be a challenge. It might be that we stream the separate files but read the whole .cff file in, as a compromise.
@drewsilcock drewsilcock added the enhancement New feature or request label Jan 7, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant