From: "Ludovic Courtès" <ludovic.courtes@inria.fr>
To: Simon Tournier <zimon.toutoune@gmail.com>
Cc: "Björn Höfling" <bjoern.hoefling@bjoernhoefling.de>,
guix-devel@gnu.org, "Lars-Dominik Braun" <lars@6xq.net>
Subject: Adding content-addressed URLs to https://guix.gnu.org/sources.json
Date: Mon, 24 Apr 2023 18:41:45 +0200 [thread overview]
Message-ID: <87cz3t44hi.fsf_-_@inria.fr> (raw)
In-Reply-To: <86lej8y75p.fsf@gmail.com> (Simon Tournier's message of "Mon, 03 Apr 2023 23:42:26 +0200")
[-- Attachment #1: Type: text/plain, Size: 1588 bytes --]
Hi,
Simon Tournier <zimon.toutoune@gmail.com> skribis:
> On Thu, 16 Mar 2023 at 12:48, Ludovic Courtès <ludovic.courtes@inria.fr> wrote:
>
>> 1. Reproducibility of past revisions. If we lose copies of the
>> auto-generated tarballs, then OpenJDK in past revisions of Guix is
>> irreparably lost. We should check whether/how to get them in
>> Disarchive + SWH.
>
> The file sources.json that SWH ingests only contains original upstream
> and not our copies. One step forward would be to also list the URL of
> our tarball substitutes as the last mirror in sources.json.
The patch below (against maintenance.git) does that. The result is
something like this:
--8<---------------cut here---------------start------------->8---
{
"type": "url",
"urls": [
"https://ftpmirror.gnu.org/gnu/zile/zile-2.6.2.tar.gz",
"ftp://ftp.cs.tu-berlin.de/pub/gnu/zile/zile-2.6.2.tar.gz",
"ftp://ftp.funet.fi/pub/mirrors/ftp.gnu.org/gnu/zile/zile-2.6.2.tar.gz",
"http://ftp.gnu.org/pub/gnu/zile/zile-2.6.2.tar.gz",
"https://bordeaux.guix.gnu.org/file/zile-2.6.2.tar.gz/sha256/0hf788zadmwx0xp1dhrgqcfvhwnarh6h9b51va4dr2y9yfppvsvp",
"https://ci.guix.gnu.org/file/zile-2.6.2.tar.gz/sha256/0hf788zadmwx0xp1dhrgqcfvhwnarh6h9b51va4dr2y9yfppvsvp",
"https://tarballs.nixos.org/sha256/0hf788zadmwx0xp1dhrgqcfvhwnarh6h9b51va4dr2y9yfppvsvp"
],
"integrity": "sha256-d+t9r/PJi9yI2qGsBA3MynK4HcMvwxZuB53Xpj5Cx0E="
},
--8<---------------cut here---------------end--------------->8---
How does that sound?
Thanks,
Ludo’.
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 3205 bytes --]
diff --git a/hydra/build-package-metadata.scm b/hydra/build-package-metadata.scm
index 6fa2173..1ddb409 100755
--- a/hydra/build-package-metadata.scm
+++ b/hydra/build-package-metadata.scm
@@ -30,6 +30,7 @@
(guix utils)
(guix gexp)
((guix build download) #:select (maybe-expand-mirrors))
+ ((guix base32) #:select (bytevector->nix-base32-string))
((guix base64) #:select (base64-encode))
((guix describe) #:select (current-profile))
((guix config) #:select (%guix-version))
@@ -73,6 +74,27 @@ superseded packages."
;;; Required by 'origin->json' for 'computed-origin-method' corner cases
(define gexp-references (@@ (guix gexp) gexp-references))
+(define %content-addressed-mirrors
+ ;; List of content-addressed mirrors.
+ ;; XXX: somewhat duplicated from (guix download)
+ (let ((guix-publish
+ (lambda (host)
+ (lambda (file hash)
+ ;; Files served by 'guix publish'.
+ (string-append "https://" host "/file/"
+ file "/" (symbol->string
+ (content-hash-algorithm hash))
+ "/" (bytevector->nix-base32-string
+ (content-hash-value hash)))))))
+
+ (list (guix-publish "bordeaux.guix.gnu.org")
+ (guix-publish "ci.guix.gnu.org")
+ (lambda (file hash)
+ (string-append "https://tarballs.nixos.org/"
+ (symbol->string (content-hash-algorithm hash))
+ "/" (bytevector->nix-base32-string
+ (content-hash-value hash)))))))
+
(define (origin->json origin)
"Return a list of JSON representations (an alist) of ORIGIN."
(define method
@@ -81,10 +103,17 @@ superseded packages."
(define uri
(origin-uri origin))
- (define (resolve urls)
- (map uri->string
- (append-map (cut maybe-expand-mirrors <> %mirrors)
- (map string->uri urls))))
+ (define (resolve urls hash)
+ (append (map uri->string
+ (append-map (cut maybe-expand-mirrors <> %mirrors)
+ (map string->uri urls)))
+ (if hash
+ (let ((file (origin-actual-file-name origin))
+ (hash (origin-hash origin)))
+ (map (lambda (make-url)
+ (make-url file hash))
+ %content-addressed-mirrors))
+ '())))
(if (eq? method (@@ (guix packages) computed-origin-method))
;; Packages in gnu/packages/gnuzilla.scm and gnu/packages/linux.scm
@@ -118,7 +147,8 @@ superseded packages."
(resolve
(match uri
((? string? url) (list url))
- ((urls ...) urls)))))))
+ ((urls ...) urls))
+ (origin-hash origin))))))
((eq? git-fetch method)
`(("git_url" . ,(git-reference-url uri))))
((eq? svn-fetch method)
next prev parent reply other threads:[~2023-04-24 16:42 UTC|newest]
Thread overview: 14+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <ZAmrhccMfljep8+i@noor.fritz.box>
[not found] ` <20230312220021.22bfff4f@tangletp>
[not found] ` <878rfwgbng.fsf@gnu.org>
2023-04-03 21:42 ` SWH: extend sources.json and Mercurial (or not Git and not tarball) Simon Tournier
2023-04-24 16:41 ` Ludovic Courtès [this message]
2023-04-25 9:59 ` Adding content-addressed URLs to https://guix.gnu.org/sources.json Simon Tournier
2023-04-25 12:40 ` Ludovic Courtès
2023-04-25 12:59 ` Ludovic Courtès
2023-04-25 13:52 ` Maxim Cournoyer
2023-04-28 13:39 ` Simon Tournier
2023-05-01 0:39 ` Maxim Cournoyer
2023-05-02 7:39 ` Ludovic Courtès
2023-05-02 12:52 ` Maxim Cournoyer
2023-05-02 17:35 ` Simon Tournier
2023-05-04 7:13 ` Ludovic Courtès
2023-05-07 21:25 ` Ludovic Courtès
2023-05-08 1:06 ` Maxim Cournoyer
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://guix.gnu.org/
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=87cz3t44hi.fsf_-_@inria.fr \
--to=ludovic.courtes@inria.fr \
--cc=bjoern.hoefling@bjoernhoefling.de \
--cc=guix-devel@gnu.org \
--cc=lars@6xq.net \
--cc=zimon.toutoune@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/guix.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).