* [bug#75426] [PATCH] docker: Build tarballs reproducibly.
@ 2025-01-07 22:55 Ludovic Courtès
2025-01-24 23:07 ` bug#75426: " Ludovic Courtès
0 siblings, 1 reply; 4+ messages in thread
From: Ludovic Courtès @ 2025-01-07 22:55 UTC (permalink / raw)
To: 75426
Cc: Simon Josefsson, Ludovic Courtès, Christopher Baines,
Josselin Poiret, Ludovic Courtès, Mathieu Othacehe,
Simon Tournier, Tobias Geerinckx-Rice
Fixes <https://issues.guix.gnu.org/75090>.
* guix/docker.scm (tar): New procedure.
(create-empty-tar, build-docker-image): Use it instead of calling
‘invoke’ directly.
Reported-by: Simon Josefsson <simon@josefsson.org>
Change-Id: Ia899c43ed6a3809ff845de0953e3d38cccf24609
---
guix/docker.scm | 25 ++++++++++++++++---------
1 file changed, 16 insertions(+), 9 deletions(-)
diff --git a/guix/docker.scm b/guix/docker.scm
index b33c5824dd..d9764f61fb 100644
--- a/guix/docker.scm
+++ b/guix/docker.scm
@@ -1,6 +1,6 @@
;;; GNU Guix --- Functional package management for GNU
;;; Copyright © 2017 Ricardo Wurmus <rekado@elephly.net>
-;;; Copyright © 2017, 2018, 2019, 2021 Ludovic Courtès <ludo@gnu.org>
+;;; Copyright © 2017-2019, 2021, 2025 Ludovic Courtès <ludo@gnu.org>
;;; Copyright © 2018 Chris Marusich <cmmarusich@gmail.com>
;;; Copyright © 2021 Maxim Cournoyer <maxim.cournoyer@gmail.com>
;;; Copyright © 2023 Oleg Pykhalov <go.wigust@gmail.com>
@@ -170,8 +170,15 @@ (define (size-sorted-store-items items max-layers)
(1- items-length)))))
(list head tail)))
+(define (tar . arguments)
+ "Invoke 'tar' with the given ARGUMENTS together with options to build
+tarballs in a reproducible fashion."
+ (apply invoke "tar" "--mtime=@1"
+ "--owner=0" "--group=0" "--numeric-owner"
+ "--sort=name" "--mode=go+u,go-w" arguments))
+
(define (create-empty-tar file)
- (invoke "tar" "-cf" file "--files-from" "/dev/null"))
+ (tar "-cf" file "--files-from" "/dev/null"))
(define* (build-docker-image image paths prefix
#:key
@@ -255,7 +262,7 @@ (define* (build-docker-image image paths prefix
(file-name (string-append file-hash "/layer.tar")))
(mkdir file-hash)
(rename-file "layer.tar" file-name)
- (invoke "tar" "-rf" "image.tar" file-name)
+ (tar "-rf" "image.tar" file-name)
(delete-file file-name)
file-hash))
(define layers-hashes
@@ -268,20 +275,20 @@ (define* (build-docker-image image paths prefix
(let* ((head-layers
(map
(lambda (file)
- (invoke "tar" "cf" "layer.tar" file)
+ (tar "cf" "layer.tar" file)
(seal-layer))
head))
(tail-layer
(begin
(create-empty-tar "layer.tar")
(for-each (lambda (file)
- (invoke "tar" "-rf" "layer.tar" file))
+ (tar "-rf" "layer.tar" file))
tail)
(let* ((file-hash (layer-diff-id "layer.tar"))
(file-name (string-append file-hash "/layer.tar")))
(mkdir file-hash)
(rename-file "layer.tar" file-name)
- (invoke "tar" "-rf" "image.tar" file-name)
+ (tar "-rf" "image.tar" file-name)
(delete-file file-name)
file-hash)))
(customization-layer
@@ -290,7 +297,7 @@ (define* (build-docker-image image paths prefix
(file-name (string-append file-hash "/layer.tar")))
(mkdir file-hash)
(rename-file file-id file-name)
- (invoke "tar" "-rf" "image.tar" file-name)
+ (tar "-rf" "image.tar" file-name)
file-hash))
(all-layers
(append head-layers (list tail-layer customization-layer))))
@@ -300,7 +307,7 @@ (define* (build-docker-image image paths prefix
(map (cut string-append <> "/layer.tar")
all-layers)
repository))))
- (invoke "tar" "-rf" "image.tar" "manifest.json")
+ (tar "-rf" "image.tar" "manifest.json")
all-layers))))
(let* ((directory "/tmp/docker-image") ;temporary working directory
(id (docker-id prefix))
@@ -388,7 +395,7 @@ (define* (build-docker-image image paths prefix
#:entry-point entry-point))))
(if max-layers
(begin
- (invoke "tar" "-rf" "image.tar" "config.json")
+ (tar "-rf" "image.tar" "config.json")
(if compressor
(begin
(apply invoke `(,@compressor "image.tar"))
base-commit: eeb019eb595bbb29f83389deb2fc823ed6402dd5
--
2.47.1
^ permalink raw reply related [flat|nested] 4+ messages in thread
* bug#75426: [PATCH] docker: Build tarballs reproducibly.
2025-01-07 22:55 [bug#75426] [PATCH] docker: Build tarballs reproducibly Ludovic Courtès
@ 2025-01-24 23:07 ` Ludovic Courtès
[not found] ` <7d999f7dfcfe6c2321e2456f60d43d30715b25f1.camel@josefsson.org>
0 siblings, 1 reply; 4+ messages in thread
From: Ludovic Courtès @ 2025-01-24 23:07 UTC (permalink / raw)
To: 75426-done, 75090-done
Cc: Simon Josefsson, Josselin Poiret, Simon Tournier,
Mathieu Othacehe, Tobias Geerinckx-Rice, Christopher Baines
Ludovic Courtès <ludo@gnu.org> skribis:
> Fixes <https://issues.guix.gnu.org/75090>.
>
> * guix/docker.scm (tar): New procedure.
> (create-empty-tar, build-docker-image): Use it instead of calling
> ‘invoke’ directly.
>
> Reported-by: Simon Josefsson <simon@josefsson.org>
> Change-Id: Ia899c43ed6a3809ff845de0953e3d38cccf24609
Pushed as 646202bf73f90de4f9b7cc66248b8f8e6e381014.
Ludo’.
^ permalink raw reply [flat|nested] 4+ messages in thread
* [bug#75426] [PATCH] docker: Build tarballs reproducibly.
[not found] ` <87ed0jc680.fsf@gnu.org>
@ 2025-01-31 22:44 ` Simon Josefsson via Guix-patches via
2025-02-04 16:23 ` Ludovic Courtès
0 siblings, 1 reply; 4+ messages in thread
From: Simon Josefsson via Guix-patches via @ 2025-01-31 22:44 UTC (permalink / raw)
To: Ludovic Courtès
Cc: Josselin Poiret, Simon Tournier, Mathieu Othacehe,
Tobias Geerinckx-Rice, 75090, 75426, Christopher Baines
[-- Attachment #1: Type: text/plain, Size: 3814 bytes --]
fre 2025-01-31 klockan 17:12 +0100 skrev Ludovic Courtès:
> Hi,
>
> Simon Josefsson <simon@josefsson.org> skribis:
>
> > Hi! I suspect something went wrong with this patch, now 'guix
> > pack'
> > fail and give the error below. Maybe the 'cf' has to come first?
> >
> > https://gitlab.com/debdistutils/guix/container/-/jobs/8988707317
>
> > tar: You must specify one of the '-Acdtrux', '--delete' or '--test-
> > label' options
> > Try 'tar --help' or 'tar --usage' for more information.
>
> Hmm I don’t see this message in the GitLab log above, and I cannot
> reproduce the problem:
>
> --8<---------------cut here---------------start------------->8---
> $ ./pre-inst-env guix pack -f docker sed
> /gnu/store/h2hdp469v3014b82qsvz5fkw00sfxdgh-sed-docker-pack.tar.gz
> $ git log |head -3
> commit 97fb1887ad10000c067168176c504274e29e4430
> Author: Ashish SHUKLA <ashish.is@lostca.se>
> Date: Mon Jan 20 21:46:10 2025 +0000
> --8<---------------cut here---------------end--------------->8---
>
> Could you try to come up with a command and commit that reproduces
> it?
Running the commands in the log like below on my local trisquel machine
with guix triggers the same problem. Any ideas?
(FWIW, the message is hidden inside the log, but the error and filename
of the log is printed in the gitlab log above, same as below; it is
possible to click on 'Job artifacts' to find the log files.)
/Simon
jas@kaka:~$ guix describe
Generation 26 31 jan 2025 23:37:27 (aktuell)
guix d48da2d
repository URL: https://git.savannah.gnu.org/git/guix.git
branch: master
commit: d48da2d21610f9cf5f76cd846703b12beedb1fd5
jas@kaka:~$ guix pack coreutils --save-provenance -S /bin=bin -S
/share=share -f docker --image-tag=guix --max-layers=8 --verbosity=2
net-base
The following derivation will be built:
/gnu/store/qkz6wc1qq23ah1xk387givjvk9qlgwcn-coreutils-net-base-
docker-pack.tar.gz.drv
bygger /gnu/store/qkz6wc1qq23ah1xk387givjvk9qlgwcn-coreutils-net-base-
docker-pack.tar.gz.drv…
|builder for `/gnu/store/qkz6wc1qq23ah1xk387givjvk9qlgwcn-coreutils-
net-base-docker-pack.tar.gz.drv' failed with exit code 1
bygge av /gnu/store/qkz6wc1qq23ah1xk387givjvk9qlgwcn-coreutils-net-
base-docker-pack.tar.gz.drv misslyckades
Granska bygglogg vid
”/var/log/guix/drvs/qk/z6wc1qq23ah1xk387givjvk9qlgwcn-coreutils-net-
base-docker-pack.tar.gz.drv.bz2”.
guix pack: fel: build of `/gnu/store/qkz6wc1qq23ah1xk387givjvk9qlgwcn-
coreutils-net-base-docker-pack.tar.gz.drv' failed
jas@kaka:~$ bzcat /var/log/guix/drvs/qk/z6wc1qq23ah1xk387givjvk9qlgwcn-
coreutils-net-base-docker-pack.tar.gz.drv.bz2 | tail -30
tar: You must specify one of the '-Acdtrux', '--delete' or '--test-
label' options
Try 'tar --help' or 'tar --usage' for more information.
Backtrace:
7 (primitive-load "/gnu/store/5z5322v39y5mwninj36m877zgsx?")
In guix/docker.scm:
387:6 6 (build-docker-image "/gnu/store/8ac4lljjiqp3a7zydh6l9v?"
?)
In ice-9/ports.scm:
433:17 5 (call-with-output-file _ _ #:binary _ #:encoding _)
476:4 4 (_ _)
In guix/docker.scm:
277:15 3 (_)
In srfi/srfi-1.scm:
586:17 2 (map1 ("/gnu/store/hw6g2kjayxnqi8rwpnmpraalxi0djkxc-?" ?))
In guix/docker.scm:
279:18 1 (_ "/gnu/store/hw6g2kjayxnqi8rwpnmpraalxi0djkxc-glibc-2?")
In guix/build/utils.scm:
822:6 0 (invoke "tar" "--mtime=@1" "--owner=0" "--group=0" "--?"
?)
guix/build/utils.scm:822:6: In procedure invoke:
ERROR:
1. &invoke-error:
program: "tar"
arguments: ("--mtime=@1" "--owner=0" "--group=0" "--numeric-
owner" "--sort=name" "--mode=go+u,go-w" "cf" "layer.tar"
"/gnu/store/hw6g2kjayxnqi8rwpnmpraalxi0djkxc-glibc-2.39")
exit-status: 2
term-signal: #f
stop-signal: #f
jas@kaka:~$
[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 1223 bytes --]
^ permalink raw reply [flat|nested] 4+ messages in thread
* [bug#75426] [PATCH] docker: Build tarballs reproducibly.
2025-01-31 22:44 ` [bug#75426] " Simon Josefsson via Guix-patches via
@ 2025-02-04 16:23 ` Ludovic Courtès
0 siblings, 0 replies; 4+ messages in thread
From: Ludovic Courtès @ 2025-02-04 16:23 UTC (permalink / raw)
To: Simon Josefsson
Cc: Josselin Poiret, Simon Tournier, Mathieu Othacehe,
Tobias Geerinckx-Rice, 75090, 75426, Christopher Baines
Hello Simon,
Simon Josefsson <simon@josefsson.org> skribis:
> jas@kaka:~$ guix pack coreutils --save-provenance -S /bin=bin -S
> /share=share -f docker --image-tag=guix --max-layers=8 --verbosity=2
> net-base
[...]
> guix/build/utils.scm:822:6: In procedure invoke:
> ERROR:
> 1. &invoke-error:
> program: "tar"
> arguments: ("--mtime=@1" "--owner=0" "--group=0" "--numeric-
> owner" "--sort=name" "--mode=go+u,go-w" "cf" "layer.tar"
> "/gnu/store/hw6g2kjayxnqi8rwpnmpraalxi0djkxc-glibc-2.39")
Fixed in 285a1cb449f60798dc83f7f1016700b4ab2374a8.
Ludo’.
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2025-02-04 16:24 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2025-01-07 22:55 [bug#75426] [PATCH] docker: Build tarballs reproducibly Ludovic Courtès
2025-01-24 23:07 ` bug#75426: " Ludovic Courtès
[not found] ` <7d999f7dfcfe6c2321e2456f60d43d30715b25f1.camel@josefsson.org>
[not found] ` <87ed0jc680.fsf@gnu.org>
2025-01-31 22:44 ` [bug#75426] " Simon Josefsson via Guix-patches via
2025-02-04 16:23 ` Ludovic Courtès
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/guix.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).