You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently we read all of the files into memory in one, including the .dat file which can be several GB. We need a way that:
Stream the .dat file when it's needed (the other files are usually a trivial size in comparison so I don't think we need to worry about that).
Only read the data through once. This sounds simple but the .cff combined file format complicates this - it's possible to embed the .dat data within the .cff such that there's another file section once the .dat ends (e.g. the .hdr or .inf). For ASCII data format, we have to only read the number of lines we expected from the sample rate specification, i.e. we need to read total_num_samples lines. For binary formats, we have to calculate the total number of bytes expected from the .dat section of the .cff file by using total_num_samples * num_bytes_per_scan where num_bytes_per_scan is 4 + 4 + [size of analog data value] * [number of analog channels] + ceil([number of status channels] / 16) * 2) where [size of analog data value is 2 for binary16 data type (a.k.a. "binary" in the COMTRADE specification - I find this confusing and ambiguous, I presume it's called that for backwards compatibility reasons) and 4 for binary32 and float32 data types. Making this data streaming for the CFF files work with Rust's ownership model is going to be a challenge. It might be that we stream the separate files but read the whole .cff file in, as a compromise.
The text was updated successfully, but these errors were encountered:
Currently we read all of the files into memory in one, including the
.dat
file which can be several GB. We need a way that:.dat
file when it's needed (the other files are usually a trivial size in comparison so I don't think we need to worry about that)..cff
combined file format complicates this - it's possible to embed the.dat
data within the.cff
such that there's another file section once the.dat
ends (e.g. the.hdr
or.inf
). For ASCII data format, we have to only read the number of lines we expected from the sample rate specification, i.e. we need to readtotal_num_samples
lines. For binary formats, we have to calculate the total number of bytes expected from the.dat
section of the.cff
file by usingtotal_num_samples * num_bytes_per_scan
wherenum_bytes_per_scan
is4 + 4 + [size of analog data value] * [number of analog channels] + ceil([number of status channels] / 16) * 2)
where[size of analog data value
is 2 for binary16 data type (a.k.a. "binary" in the COMTRADE specification - I find this confusing and ambiguous, I presume it's called that for backwards compatibility reasons) and 4 for binary32 and float32 data types. Making this data streaming for the CFF files work with Rust's ownership model is going to be a challenge. It might be that we stream the separate files but read the whole.cff
file in, as a compromise.The text was updated successfully, but these errors were encountered: