-
-
Notifications
You must be signed in to change notification settings - Fork 65
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Use 7z to create more efficient ZIP archives #81
base: main
Are you sure you want to change the base?
Conversation
090be3f
to
a312a59
Compare
cd .. | ||
rm -rf ios_xcode | ||
|
||
## Templates TPZ (Classical) ## | ||
|
||
echo "${templates_version}" > ${templatesdir}/version.txt | ||
pushd ${templatesdir}/.. | ||
zip -q -9 -r -D "${reldir}/${godot_basename}_export_templates.tpz" templates/* | ||
$ZIP -r "${reldir}/${godot_basename}_export_templates.tpz" templates/* |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think -D
was important here, so this might break the logic.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
7z
doesn't seem to have an equivalent of -D
. According to zip
's command line help:
-D do not add directory entries
The only difference I can notice between 2 ZIP archives created (one with -D
, one without -D
) is that the former will have directory entries in its 7z l
listing, but the latter will specify full paths to every file.
In practice, this appears to make no concrete difference on what you see when opening the ZIP archive: both have the exact same folder structure when viewed with GUI archive managers. In short, -D
does not make the ZIP's structure flat or remove the top-level folder as the option's name might suggest. The only real difference it seems to make is that it prevents empty folders from being added to the archive.
I think the -D
requirement is a leftover from early Godot 3.x days where Godot behaved poorly if given a TPZ that had directory entries. I've created TPZ files without -D
for a while for my custom builds and it worked just fine (before I switched to 7z
).
#!/usr/bin/env bash
mkdir -p example/{folder1,folder2,empty_folder}
touch example/{folder1,folder2}/example.txt
echo -e "\n\n================ Creating archives ================"
zip -r example.zip example
zip -r example-flat.zip example/*
zip -r -D example-D.zip example
zip -r -D example-D-flat.zip example/*
7z a example-7z.zip example
7z a example-7z-flat.zip example/*
echo -e "\n\n================ File contents created by zip ================"
7z l example.zip
7z l example-flat.zip
7z l example-D.zip
7z l example-D-flat.zip
echo -e "\n\n================ File contents created by 7z ================"
7z l example-7z.zip
7z l example-7z-flat.zip
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For the record, in the asset library I had to fix some bugs related to directory entries missing. Not sure if the export template installer handles it any better.
@@ -516,7 +518,7 @@ if [ "${build_mono}" == "1" ]; then | |||
|
|||
echo "${templates_version}.mono" > ${templatesdir_mono}/version.txt | |||
pushd ${templatesdir_mono}/.. | |||
zip -q -9 -r -D "${reldir_mono}/${godot_basename}_mono_export_templates.tpz" templates/* | |||
$ZIP -r "${reldir_mono}/${godot_basename}_mono_export_templates.tpz" templates/* |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also needs -D equivalent.
Needs rebase after #90. Then I'll test and see if I can merge for the next beta. |
For a full set of Godot 4.1.1 editor + export template builds, this saves 28 MB without affecting decompression times or archive compatibility. Any ZIP decompression software can decompress these 7z-created ZIP archives; they use the standard DEFLATE algorithm. Creating archives is expected to be about twice as slow as `zip -9`, but given the bandwidth and download time savings, it's probably worth it.
@@ -47,7 +49,7 @@ sign_macos() { | |||
codesign --force --timestamp \ | |||
--options=runtime --entitlements editor.entitlements \ | |||
-s ${OSX_KEY_ID} -v ${_appname} && \ | |||
zip -r ${_binname}_signed.zip ${_appname}" | |||
$ZIP -r ${_binname}_signed.zip ${_appname}" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So this bit forces me to have 7zip installed on macOS, which is less trivial than on Linux.
It seems available on homebrew but homebrew isn't set up on HP's Mac, could likely install it.
But this begs the question of whether there's anything more actively maintained than 7zip to compress zips. The last release dates back to 2016.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
But this begs the question of whether there's anything more actively maintained than 7zip to compress zips. The last release dates back to 2016.
The last GUI release was made in June 2023: https://7-zip.org/
However, the CLI version was never updated past 16.02 (May 2016). This is an unfortunate artifact of 7-zip being developed around its Windows GUI first, and also being developed behind closed doors.
I don't know of any alternatives that have the same level of efficiency and performance for compressing ZIP archives. Using Zopfli to create a ZIP archive will result in slightly more efficiency, but it's more than 10× slower which makes it not worth the time. zip -9
is slightly better than the default option, but still not as good as 7z a -mx9
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The last GUI release was made in June 2023: 7-zip.org
It's Windows only though, no?
I found PeaZip which seems actively maintained and properly cross-platform: https://github.com/peazip/PeaZip/releases/
Might be worth comparing with 7zip.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
But this begs the question of whether there's anything more actively maintained than 7zip to compress zips. The last release dates back to 2016.
At least in macOS homebrew, there are two 7Z packages:
p7zip
-7z
executable, built from p7zip port source from 2017 (17.04).sevenzip
-7zz
executable, built from official 2023 source (23.01, same as GUI version).
Seems like official source added support for Linux/macOS command line build in 2021.
Edit: Debian seems to use the same naming, so it should be standard - https://manpages.debian.org/unstable/7zip/7zz.1.en.html
For a full set of Godot 4.1.1 editor + export template builds, this saves 28 MB without affecting decompression times or archive compatibility. Any ZIP decompression software can decompress these 7z-created ZIP archives; they use the standard DEFLATE algorithm.
Creating archives is expected to be about twice as slow as
zip -9
, but given the bandwidth and download time savings, it's probably worth it.For Mono builds, size savings are expected to be even larger as these builds are larger in the first place. Future versions will also logically benefit more from this change, as their binaries expected to keep growing over time.
Note: I haven't tested the script locally, so I advise performing a dry run before merging. Both the machine
build-release.sh
is run on and the macOS machine used for signing releases must have7z
installed and inPATH
. On Ubuntu, this can be done usingsudo apt install p7zip-full
. On Fedora, this can be done usingdnf install p7zip
. On macOS, this can be done usingbrew install sevenzip
with Homebrew.Before
Official releases as downloaded from https://downloads.tuxfamily.org/godotengine/4.1.1/.
After
Official releases downloaded, extracted and recompressed with
7z a -mx9
. For the export templates TPZ, both the TPZ itself and ZIP files contained within were recompressed.