feat: Use deterministic tar header mode, use insta to assert reproducibility #75
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
environment.tar built on macos-arm64 and environment2.tar build on linux-64 ❯ pixi exec diffoscope environment*.tar
--- environment.tar
+++ environment2.tar
├── file list
│ @@ -1,24 +1,24 @@
│ -drwxr-xr-x 0 501 20 0 1970-01-01 00:00:00.000000 ./
│ -drwxr-xr-x 0 501 20 0 1970-01-01 00:00:00.000000 channel
│ -drwxr-xr-x 0 501 20 0 1970-01-01 00:00:00.000000 channel/noarch
│ --rw-r--r-- 0 501 20 1398245 2024-02-03 11:44:20.000000 channel/noarch/pip-24.0-pyhd8ed1ab_0.conda
│ --rw-r--r-- 0 501 20 483015 2024-05-22 08:55:41.000000 channel/noarch/setuptools-70.0.0-pyhd8ed1ab_0.conda
│ --rw-r--r-- 0 501 20 119815 2024-02-02 15:15:45.000000 channel/noarch/tzdata-2024a-h0c530f3_0.conda
│ --rw-r--r-- 0 501 20 57963 2024-03-27 13:26:49.000000 channel/noarch/wheel-0.43.0-pyhd8ed1ab_1.conda
│ --rw-r--r-- 0 501 20 2109 1970-01-01 00:00:00.000000 channel/noarch/repodata.json
│ -drwxr-xr-x 0 501 20 0 1970-01-01 00:00:00.000000 channel/osx-arm64
│ --rw-r--r-- 0 501 20 795131 2024-05-08 19:01:38.000000 channel/osx-arm64/ncurses-6.5-hb89a1cb_0.conda
│ --rw-r--r-- 0 501 20 39020 2021-11-09 20:09:47.000000 channel/osx-arm64/libffi-3.4.2-h3422bc3_5.tar.bz2
│ --rw-r--r-- 0 501 20 3145523 2023-11-05 16:40:32.000000 channel/osx-arm64/tk-8.6.13-h5083fa2_1.conda
│ --rw-r--r-- 0 501 20 122325 2023-11-06 14:18:14.000000 channel/osx-arm64/bzip2-1.0.8-h93a5062_5.conda
│ --rw-r--r-- 0 501 20 2891941 2024-06-05 00:04:06.000000 channel/osx-arm64/openssl-3.3.1-hfb2fe0b_0.conda
│ --rw-r--r-- 0 501 20 46768 2024-05-28 05:29:11.000000 channel/osx-arm64/libzlib-1.2.13-hfb2fe0b_6.conda
│ --rw-r--r-- 0 501 20 824794 2024-04-17 15:29:08.000000 channel/osx-arm64/libsqlite-3.45.3-h091b4b1_0.conda
│ --rw-r--r-- 0 501 20 13207557 2024-04-15 18:42:56.000000 channel/osx-arm64/python-3.12.3-h4a7b5fc_0_cpython.conda
│ --rw-r--r-- 0 501 20 63655 2024-03-13 20:40:24.000000 channel/osx-arm64/libexpat-2.6.2-hebf3989_0.conda
│ --rw-r--r-- 0 501 20 6236 1970-01-01 00:00:00.000000 channel/osx-arm64/repodata.json
│ --rw-r--r-- 0 501 20 155725 2024-02-02 03:20:34.000000 channel/osx-arm64/ca-certificates-2024.2.2-hf0a4a13_0.conda
│ --rw-r--r-- 0 501 20 250351 2023-03-23 00:48:31.000000 channel/osx-arm64/readline-8.2-h92ec313_1.conda
│ --rw-r--r-- 0 501 20 235693 2022-08-12 23:29:21.000000 channel/osx-arm64/xz-5.2.6-h57fd34a_0.tar.bz2
│ --rw-r--r-- 0 501 20 79 1970-01-01 00:00:00.000000 pixi-pack.json
│ --rw-r--r-- 0 501 20 529 1970-01-01 00:00:00.000000 environment.yml
│ +drwxrwxr-x 0 1000 1000 0 1970-01-01 00:00:00.000000 ./
│ +-rw-rw-r-- 0 1000 1000 79 1970-01-01 00:00:00.000000 pixi-pack.json
│ +drwxrwxr-x 0 1000 1000 0 1970-01-01 00:00:00.000000 channel
│ +drwxrwxr-x 0 1000 1000 0 1970-01-01 00:00:00.000000 channel/osx-arm64
│ +-rw-rw-r-- 0 1000 1000 235693 2022-08-12 23:29:21.000000 channel/osx-arm64/xz-5.2.6-h57fd34a_0.tar.bz2
│ +-rw-rw-r-- 0 1000 1000 122325 2023-11-06 14:18:14.000000 channel/osx-arm64/bzip2-1.0.8-h93a5062_5.conda
│ +-rw-rw-r-- 0 1000 1000 13207557 2024-04-15 18:42:56.000000 channel/osx-arm64/python-3.12.3-h4a7b5fc_0_cpython.conda
│ +-rw-rw-r-- 0 1000 1000 6236 1970-01-01 00:00:00.000000 channel/osx-arm64/repodata.json
│ +-rw-rw-r-- 0 1000 1000 39020 2021-11-09 20:09:47.000000 channel/osx-arm64/libffi-3.4.2-h3422bc3_5.tar.bz2
│ +-rw-rw-r-- 0 1000 1000 46768 2024-05-28 05:29:11.000000 channel/osx-arm64/libzlib-1.2.13-hfb2fe0b_6.conda
│ +-rw-rw-r-- 0 1000 1000 824794 2024-04-17 15:29:08.000000 channel/osx-arm64/libsqlite-3.45.3-h091b4b1_0.conda
│ +-rw-rw-r-- 0 1000 1000 2891941 2024-06-05 00:04:06.000000 channel/osx-arm64/openssl-3.3.1-hfb2fe0b_0.conda
│ +-rw-rw-r-- 0 1000 1000 795131 2024-05-08 19:01:38.000000 channel/osx-arm64/ncurses-6.5-hb89a1cb_0.conda
│ +-rw-rw-r-- 0 1000 1000 250351 2023-03-23 00:48:31.000000 channel/osx-arm64/readline-8.2-h92ec313_1.conda
│ +-rw-rw-r-- 0 1000 1000 63655 2024-03-13 20:40:24.000000 channel/osx-arm64/libexpat-2.6.2-hebf3989_0.conda
│ +-rw-rw-r-- 0 1000 1000 3145523 2023-11-05 16:40:32.000000 channel/osx-arm64/tk-8.6.13-h5083fa2_1.conda
│ +-rw-rw-r-- 0 1000 1000 155725 2024-02-02 03:20:34.000000 channel/osx-arm64/ca-certificates-2024.2.2-hf0a4a13_0.conda
│ +drwxrwxr-x 0 1000 1000 0 1970-01-01 00:00:00.000000 channel/noarch
│ +-rw-rw-r-- 0 1000 1000 483015 2024-05-22 08:55:41.000000 channel/noarch/setuptools-70.0.0-pyhd8ed1ab_0.conda
│ +-rw-rw-r-- 0 1000 1000 2109 1970-01-01 00:00:00.000000 channel/noarch/repodata.json
│ +-rw-rw-r-- 0 1000 1000 119815 2024-02-02 15:15:45.000000 channel/noarch/tzdata-2024a-h0c530f3_0.conda
│ +-rw-rw-r-- 0 1000 1000 1398245 2024-02-03 11:44:20.000000 channel/noarch/pip-24.0-pyhd8ed1ab_0.conda
│ +-rw-rw-r-- 0 1000 1000 57963 2024-03-27 13:26:49.000000 channel/noarch/wheel-0.43.0-pyhd8ed1ab_1.conda
│ +-rw-rw-r-- 0 1000 1000 529 1970-01-01 00:00:00.000000 environment.ymlwe should set the owners+groups and permissions for the files as well for reproducibility |
pavelzw
changed the title
|
it seems that the order depends on the OS ❯ pixi exec diffoscope abc2.tar abc3.tar
--- abc2.tar
+++ abc3.tar
├── file list
│ @@ -1,7 +1,7 @@
│ drwxr-xr-x 0 0 0 0 1970-01-01 00:00:00.000000 ./
│ --rw-r--r-- 0 0 0 79 1970-01-01 00:00:00.000000 pixi-pack.json
│ drwxr-xr-x 0 0 0 0 1970-01-01 00:00:00.000000 channel
│ drwxr-xr-x 0 0 0 0 1970-01-01 00:00:00.000000 channel/osx-arm64
│ -rw-r--r-- 0 0 0 158482 1970-01-01 00:00:00.000000 channel/osx-arm64/ca-certificates-2024.8.30-hf0a4a13_0.conda
│ -rw-r--r-- 0 0 0 552 1970-01-01 00:00:00.000000 channel/osx-arm64/repodata.json
│ +-rw-r--r-- 0 0 0 79 1970-01-01 00:00:00.000000 pixi-pack.json
│ -rw-r--r-- 0 0 0 94 1970-01-01 00:00:00.000000 environment.ymlunfortunately i don't think there is a way to sort we probably need to sort ourselves beforehand (or download only to memory while we're at it) |
pavelzw
force-pushed
the
insta-reproducibility
branch
from
f3088ae to
3ddf9cb
Compare
delsner
approved these changes
Nov 27, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
Closes #65
Changes
Use insta to store the sha of the produced tarball