Skip to content

FAQ

Jump to bottom
Hyo-Kyung Lee edited this page Jan 13, 2023 · 11 revisions

This FAQ can answer questions about Kerchunk Study.

How do I install Kerchunk?

Use pip install git+https://github.com/fsspec/kerchunk because the latest Kerchunk fixed a bug that we reported recently.

Which file is bigger - Kerchunk or DMR++?

DMR++ is bigger. For example, the no_dset test case has only root group and no dataset. Yet, DMR++ size is 1.8K.

Product Kerchunk DMR++
no_dset 54B 1.8K
ATL08 213K 343K

Does Kerchunk work for HDF5 on IPFS?

Yes. However, it's slower than local file system.

The table below is the GitHub Action result for generating Kerchunk JSON file:

Product Local IPFS
ATL08 0.0014s 0.0037s

How can I generate DMR++ from Hyrax on Docker?

  1. Copy local HDF5 files to /tmp/dmrpp.
  2. Run ls /usr/share/hyrax from Docker Desktop terminal for Hyrax image.
  3. You will see the HDF5 files from your local file. Run the script in recipe.

Hyrax on Docker throws 500 error. How can I fix it?

  1. Copy .dmrpp and .h5 files to /tmp/dmrpp.
  2. Restart the Hyrax Docker image.

Does DMR++ generator use hdf5_handler configuration?

Yes. However, without -c option in get_dmrpp, the EnableCF is turned off by default. HDF5 handler has EnableCF=True option by default in /etc/bes/modules/h5.conf. The EnableCF option suppresses group hierarchy. DMR++, created by the default get_dmrpp, preserves the group hierarchy.

The -c option in get_dmrpp throws an error. Why?

If you use -c /etc/bes/bes.conf option, you will get an unbalanced tags error:

#./write_dmrpp.sh
target_dir: /usr/share/hyrax
 Processing: /usr/share/hyrax/20020602090000-JPL-L4_GHRSST-SSTfnd-MUR-GLOB-v02.0-fv04.1.h5
infile_base: 20020602090000-JPL-L4_GHRSST-SSTfnd-MUR-GLOB-v02.0-fv04.1.h5
    bes_dir: /usr/share/hyrax
     Output: /usr/share/hyrax/20020602090000-JPL-L4_GHRSST-SSTfnd-MUR-GLOB-v02.0-fv04.1.h5.dmrpp
std::exception: At line 14: The document contained unbalanced tags.

FAQ

News

Clone this wiki locally