Improving the GBA save type database format

[htv04]

Hey everyone, I decided to revisit open_agb_firm's GBA save type database, gba_db.bin, and would like to get some feedback on some findings I made.

When I originally created the build script for the database, I made it with the pre-existing format in mind, which was created back when the database was (mostly) manually maintained. After checking open_agb_firm's source code, however, I noticed that many of these values were redundant; specifically the title, serial code, last 12 bytes of the SHA-1, and size encoded into the entry attribute. While these values might seem useful for debugging, I found that the first 8 bytes of the SHA-1 used by open_agb_firm were enough to diagnose any issues using the database. Also, the current size of the database is around ~650 KiB, so I wanted to see how much I could get this down.

Here is the result.

I took this opportunity to mostly rewrite my build script for the database since the code quality of my original script isn't the best. Anyways, as of writing, I managed to get the size of the database down to ~25 KiB, and it functionally works exactly like the older database did. This is because each entry is now 9 bytes long, 8 bytes for the trimmed SHA-1, and one byte for the save type. I was tempted to make a PR after completing this; however, something bugged me.

Now that I got the database to such a small size, the format seems... inefficient. For one, the values aren't aligned at all, and reading them into the struct in open_agb_firm required me to add the packed attribute to the struct. But also, the data for each entry, apart from the 8-byte SHA-1 "key," is now one byte. Sure, if this format could potentially be expanded again later for whatever reason, I wouldn't worry about it too much. However, this seems unlikely given that the only thing we need to look up from a database we can't get from the ROM is the save type.

One approach I was thinking of is to ditch the external database and instead adapt the build script to create a C source file. Then it could be compiled with open_agb_firm and any issues with alignment would magically go away (and file I/O overhead). However, this is far from ideal because it would mean that updating the database would require a recompilation of the FIRM. Plus, it would add bloat to the FIRM, and no one wants that. Another approach could be to have two separate files, one with the trimmed SHA-1s, and after the index is found via a binary search, the corresponding save type in another file. This doesn't seem more efficient than the current implementation, but maybe it could help with alignment. Short of some annoying and non-portable compiler trickery to generate an aligned binary, I don't think there's any way to make a more efficient file-based database.

I don't know; maybe I'm just thinking about this too much. Something tells me the database could be optimized now that only one byte corresponds to each entry, but maybe the current format is optimal enough. If anyone has any thoughts on this, please let me know.

EDIT: I gave it some thought, and maybe the built-in approach isn't totally unviable. Of course, there's the issue of FIRM bloat, but otherwise, the database doesn't need to be updated often. Most homebrew titles also have their save types read via their serial code, so new entries don't typically need to be added by the user.

[profi200]

The reason the db has this many fields was/is to ease matching entries with the no-intro database and a second problem is when the hash mismatches we can fall back to product code/serial matching. I planned to remove other fields as soon as the db is 100% completed. Still have the hand made db sitting at 1500 entries and didn't continue it yet.

[htv04]

I see; then, yeah I can restore those fields in my script. Would you like me to make a PR so you can merge these changes from my fork? Although I'm still concerned about the efficiency of the database, loading it into the packed struct and everything. I believe this affects the older database too, although it gets away with being copied into an unpacked struct somehow.

As for your handmade DB, I can also send an updated version here with the updated format.

[profi200]

The compiler will add no padding if the member is already aligned. If you keep an eye on alignment packed is not needed. And yes, i did cheat a little with the current format doing unaligned u64 reads but this is supported by the CPU.

[htv04]

So, all things considered, would a 28-byte entry (20-byte SHA-1, 4-byte serial, and 4-byte attribute) be aligned?

[profi200]

I won't always be 8 bytes aligned but the types will pack tightly. You can test this with a "static_assert(sizeof(GameDbEntry) == 28, "not packed")".

[htv04]

Okay, I made a PR with these changes: #128

Would you mind uploading the latest copy of your DB here? I wrote a script that can convert all of the entries to the proposed format.