[Feature Request] Improve Data Compression

[borongyuan]

We have added RVL compression for depth images. However, in order to further reduce the size of the database, it is still necessary to compress other data. For point clouds and meshes, one possible solution is to use Google's Draco library.

Previously, other data was compressed using zlib. But I found its performance to be a bit poor. Even if only a small number of local features and descriptors are compressed, it will take more than ten milliseconds or more. Therefore, for intra-machine communication, I usually set Z_NO_COMPRESSION to avoid actual compression and use zero copy. From my experience with rosbag and ZFS, using lz4 compression should be a better choice. It sacrifices some compression, but is much faster.

[matlabbe]

Here other data (that are not images) that are currently compressed in the database:

rtabmap/corelib/src/resources/DatabaseSchema.sql.in

Line 37 in f8b1d50

scan BLOB, -- compressed data (Laser scan)

rtabmap/corelib/src/resources/DatabaseSchema.sql.in

Lines 40 to 42 in f8b1d50

	ground_cells BLOB, -- compressed data (occupancy grid)
	obstacle_cells BLOB, -- compressed data (occupancy grid)
	empty_cells BLOB, -- compressed data (occupancy grid)

These are point cloud data, so Draco library could be indeed used for that. For the compressed data under Admin table:

rtabmap/corelib/src/resources/DatabaseSchema.sql.in

Lines 122 to 130 in f8b1d50

	opt_cloud BLOB, -- compressed data
	opt_ids BLOB, -- Node ids used to generate the optimized cloud/mesh
	opt_poses BLOB, -- compressed N*3x4 float
	opt_last_localization BLOB, -- 3x4 float
	opt_polygons_size INTEGER, -- e.g., 3
	opt_polygons BLOB, -- compressed data [length_v0, i0,i1,i3, length_v1, i0,i1,i3]
	opt_tex_coords BLOB, -- compressed data [length_v0, u0,v0,u1,v1,u2,v2, length_v1, u0,v0,u1,v1,u2,v2]
	opt_tex_materials BLOB, -- compressed image
	opt_map BLOB, -- compressed CV_8SC1 occupancy grid

as they are not meant to be used online, but only loaded/saved when rtabmap starts or closes, I would focus more on the ones above first. Note also that depending of the features used, a large part of the database can be from uncompressed descriptors.

Replacing zlib by lz4 may be a reasonable solution too. Ideally, if we could detect the compression format, we could add an option to compress general data as lz4 instead, while being backward compatible with databases with compressed data done with zlib. It looks like lz4 has a header, while zlib has its own, which means we could probably do something similar than you did for rvl versus png format.

That benchmark table is pretty useful:
https://github.com/lz4/lz4?tab=readme-ov-file#benchmarks