Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use 7z to create more efficient ZIP archives #81

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

Calinou
Copy link
Member

@Calinou Calinou commented Jul 27, 2023

For a full set of Godot 4.1.1 editor + export template builds, this saves 28 MB without affecting decompression times or archive compatibility. Any ZIP decompression software can decompress these 7z-created ZIP archives; they use the standard DEFLATE algorithm.

Creating archives is expected to be about twice as slow as zip -9, but given the bandwidth and download time savings, it's probably worth it.

For Mono builds, size savings are expected to be even larger as these builds are larger in the first place. Future versions will also logically benefit more from this change, as their binaries expected to keep growing over time.

Note: I haven't tested the script locally, so I advise performing a dry run before merging. Both the machine build-release.sh is run on and the macOS machine used for signing releases must have 7z installed and in PATH. On Ubuntu, this can be done using sudo apt install p7zip-full. On Fedora, this can be done using dnf install p7zip. On macOS, this can be done using brew install sevenzip with Homebrew.

Before

Official releases as downloaded from https://downloads.tuxfamily.org/godotengine/4.1.1/.

817,195,141   Godot_v4.1.1-stable_export_templates.tpz
 47,264,118   Godot_v4.1.1-stable_linux.x86_32.zip
 54,436,186   Godot_v4.1.1-stable_linux.x86_64.zip
104,426,977   Godot_v4.1.1-stable_macos.universal.zip
 26,028,541   Godot_v4.1.1-stable_web_editor.zip
 49,422,199   Godot_v4.1.1-stable_win32.exe.zip
 54,161,086   Godot_v4.1.1-stable_win64.exe.zip

After

Official releases downloaded, extracted and recompressed with 7z a -mx9. For the export templates TPZ, both the TPZ itself and ZIP files contained within were recompressed.

797,372,026   Godot_v4.1.1-stable_export_templates.tpz
 46,153,580   Godot_v4.1.1-stable_linux.x86_32.zip
 53,220,111   Godot_v4.1.1-stable_linux.x86_64.zip
101,821,948   Godot_V4.1.1-stable_macos.universal.zip
 25,215,322   Godot_v4.1.1-stable_web_editor.zip
 48,186,220   Godot_v4.1.1-stable_win32.exe.zip
 52,936,912   Godot_v4.1.1-stable_win64.exe.zip

build-release.sh Outdated Show resolved Hide resolved
@Calinou Calinou force-pushed the zip-use-7z branch 3 times, most recently from 090be3f to a312a59 Compare October 25, 2023 01:46
build-release.sh Outdated Show resolved Hide resolved
build-release.sh Outdated Show resolved Hide resolved
cd ..
rm -rf ios_xcode

## Templates TPZ (Classical) ##

echo "${templates_version}" > ${templatesdir}/version.txt
pushd ${templatesdir}/..
zip -q -9 -r -D "${reldir}/${godot_basename}_export_templates.tpz" templates/*
$ZIP -r "${reldir}/${godot_basename}_export_templates.tpz" templates/*
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think -D was important here, so this might break the logic.

Copy link
Member Author

@Calinou Calinou Oct 25, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

7z doesn't seem to have an equivalent of -D. According to zip's command line help:

 -D   do not add directory entries

The only difference I can notice between 2 ZIP archives created (one with -D, one without -D) is that the former will have directory entries in its 7z l listing, but the latter will specify full paths to every file.

In practice, this appears to make no concrete difference on what you see when opening the ZIP archive: both have the exact same folder structure when viewed with GUI archive managers. In short, -D does not make the ZIP's structure flat or remove the top-level folder as the option's name might suggest. The only real difference it seems to make is that it prevents empty folders from being added to the archive.

I think the -D requirement is a leftover from early Godot 3.x days where Godot behaved poorly if given a TPZ that had directory entries. I've created TPZ files without -D for a while for my custom builds and it worked just fine (before I switched to 7z).

#!/usr/bin/env bash

mkdir -p example/{folder1,folder2,empty_folder}
touch example/{folder1,folder2}/example.txt

echo -e "\n\n================ Creating archives ================"
zip -r example.zip example
zip -r example-flat.zip example/*
zip -r -D example-D.zip example
zip -r -D example-D-flat.zip example/*
7z a example-7z.zip example
7z a example-7z-flat.zip example/*

echo -e "\n\n================ File contents created by zip ================"
7z l example.zip
7z l example-flat.zip
7z l example-D.zip
7z l example-D-flat.zip

echo -e "\n\n================ File contents created by 7z ================"
7z l example-7z.zip
7z l example-7z-flat.zip

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the record, in the asset library I had to fix some bugs related to directory entries missing. Not sure if the export template installer handles it any better.

build-release.sh Outdated Show resolved Hide resolved
@@ -516,7 +518,7 @@ if [ "${build_mono}" == "1" ]; then

echo "${templates_version}.mono" > ${templatesdir_mono}/version.txt
pushd ${templatesdir_mono}/..
zip -q -9 -r -D "${reldir_mono}/${godot_basename}_mono_export_templates.tpz" templates/*
$ZIP -r "${reldir_mono}/${godot_basename}_mono_export_templates.tpz" templates/*
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also needs -D equivalent.

@akien-mga
Copy link
Member

Needs rebase after #90. Then I'll test and see if I can merge for the next beta.

For a full set of Godot 4.1.1 editor + export template builds, this saves
28 MB without affecting decompression times or archive compatibility.
Any ZIP decompression software can decompress these 7z-created ZIP archives;
they use the standard DEFLATE algorithm.

Creating archives is expected to be about twice as slow as `zip -9`,
but given the bandwidth and download time savings, it's probably worth it.
@@ -47,7 +49,7 @@ sign_macos() {
codesign --force --timestamp \
--options=runtime --entitlements editor.entitlements \
-s ${OSX_KEY_ID} -v ${_appname} && \
zip -r ${_binname}_signed.zip ${_appname}"
$ZIP -r ${_binname}_signed.zip ${_appname}"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So this bit forces me to have 7zip installed on macOS, which is less trivial than on Linux.

It seems available on homebrew but homebrew isn't set up on HP's Mac, could likely install it.

But this begs the question of whether there's anything more actively maintained than 7zip to compress zips. The last release dates back to 2016.

Copy link
Member Author

@Calinou Calinou Nov 13, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But this begs the question of whether there's anything more actively maintained than 7zip to compress zips. The last release dates back to 2016.

The last GUI release was made in June 2023: https://7-zip.org/
However, the CLI version was never updated past 16.02 (May 2016). This is an unfortunate artifact of 7-zip being developed around its Windows GUI first, and also being developed behind closed doors.

I don't know of any alternatives that have the same level of efficiency and performance for compressing ZIP archives. Using Zopfli to create a ZIP archive will result in slightly more efficiency, but it's more than 10× slower which makes it not worth the time. zip -9 is slightly better than the default option, but still not as good as 7z a -mx9.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The last GUI release was made in June 2023: 7-zip.org

It's Windows only though, no?

I found PeaZip which seems actively maintained and properly cross-platform: https://github.com/peazip/PeaZip/releases/

Might be worth comparing with 7zip.

Copy link
Member

@bruvzg bruvzg Nov 13, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But this begs the question of whether there's anything more actively maintained than 7zip to compress zips. The last release dates back to 2016.

At least in macOS homebrew, there are two 7Z packages:

  • p7zip - 7z executable, built from p7zip port source from 2017 (17.04).
  • sevenzip - 7zz executable, built from official 2023 source (23.01, same as GUI version).

Seems like official source added support for Linux/macOS command line build in 2021.

Edit: Debian seems to use the same naming, so it should be standard - https://manpages.debian.org/unstable/7zip/7zz.1.en.html

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Use 7-zip to create official release ZIP archives with better compression (and same compatibility)
4 participants