SWH: extend sources.json and Mercurial (or not Git and not tarball)

SWH: extend sources.json and Mercurial (or not Git and not tarball)

[Simon Tournier]

Hi,

On Thu, 16 Mar 2023 at 12:48, Ludovic Courtès <ludovic.courtes@inria.fr> wrote:

>   1. Reproducibility of past revisions.  If we lose copies of the
>      auto-generated tarballs, then OpenJDK in past revisions of Guix is
>      irreparably lost.  We should check whether/how to get them in
>      Disarchive + SWH.

The file sources.json that SWH ingests only contains original upstream
and not our copies.  One step forward would be to also list the URL of
our tarball substitutes as the last mirror in sources.json.

Any taker? :-)

>
>   2. Mercurial/SWH bridge.  While SWH has a one-to-one mapping with Git
>      (you can ask it for a specific Git commit ID), that’s not true for
>      hg.  This is a more general problem, but as things are today,
>      there’s no automatic SWH fallback if the upstream hg server
>      vanishes.

Since most git-fetch origins use label tags, the one-to-one mapping is
not guarantee and we rely on SWH resolver using URL + label tag to get
the content from SWH.  For instance, if the label tag is changed
in-place by upstream pointing then to one different commit, then SWH
creates another snapshot but our fallback will fail (known issue:
history of history, etc.)

If we would have a list of identifiers instead of only NAR+SHA256, and
we could have Git commit ID here (or SWHID or others), then it would
ease the fallback machinery.

SWH folk is currently adding NAR hashes; they store it as ’ExtID’ (see
[1] and merge request [2]), but it is not clear yet how they would
expose the API entry point or if they would do.

Extending ’origin’ with another optional field using other
content-address keys would robustify the preservation of Guix.  Yeah,
indeed we could also build the X-to-SWH bridge with the Disarchive
database (global bridge) but it would appear to me better to have some
“local” origin-based bridge.

1: https://gitlab.softwareheritage.org/swh/meta/-/issues/4979
2: https://gitlab.softwareheritage.org/swh/devel/swh-loader-core/-/merge_requests/459

Cheers,
simon

Adding content-addressed URLs to https://guix.gnu.org/sources.json

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 1588 bytes --]

Hi,

Simon Tournier <zimon.toutoune@gmail.com> skribis:

> On Thu, 16 Mar 2023 at 12:48, Ludovic Courtès <ludovic.courtes@inria.fr> wrote:
>
>>   1. Reproducibility of past revisions.  If we lose copies of the
>>      auto-generated tarballs, then OpenJDK in past revisions of Guix is
>>      irreparably lost.  We should check whether/how to get them in
>>      Disarchive + SWH.
>
> The file sources.json that SWH ingests only contains original upstream
> and not our copies.  One step forward would be to also list the URL of
> our tarball substitutes as the last mirror in sources.json.

The patch below (against maintenance.git) does that.  The result is
something like this:

--8<---------------cut here---------------start------------->8---
{
  "type": "url",
  "urls": [
    "https://ftpmirror.gnu.org/gnu/zile/zile-2.6.2.tar.gz",
    "ftp://ftp.cs.tu-berlin.de/pub/gnu/zile/zile-2.6.2.tar.gz",
    "ftp://ftp.funet.fi/pub/mirrors/ftp.gnu.org/gnu/zile/zile-2.6.2.tar.gz",
    "http://ftp.gnu.org/pub/gnu/zile/zile-2.6.2.tar.gz",
    "https://bordeaux.guix.gnu.org/file/zile-2.6.2.tar.gz/sha256/0hf788zadmwx0xp1dhrgqcfvhwnarh6h9b51va4dr2y9yfppvsvp",
    "https://ci.guix.gnu.org/file/zile-2.6.2.tar.gz/sha256/0hf788zadmwx0xp1dhrgqcfvhwnarh6h9b51va4dr2y9yfppvsvp",
    "https://tarballs.nixos.org/sha256/0hf788zadmwx0xp1dhrgqcfvhwnarh6h9b51va4dr2y9yfppvsvp"
  ],
  "integrity": "sha256-d+t9r/PJi9yI2qGsBA3MynK4HcMvwxZuB53Xpj5Cx0E="
},
--8<---------------cut here---------------end--------------->8---

How does that sound?

Thanks,
Ludo’.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 3205 bytes --]

diff --git a/hydra/build-package-metadata.scm b/hydra/build-package-metadata.scm
index 6fa2173..1ddb409 100755
--- a/hydra/build-package-metadata.scm
+++ b/hydra/build-package-metadata.scm
@@ -30,6 +30,7 @@
              (guix utils)
              (guix gexp)
              ((guix build download) #:select (maybe-expand-mirrors))
+             ((guix base32) #:select (bytevector->nix-base32-string))
              ((guix base64) #:select (base64-encode))
              ((guix describe) #:select (current-profile))
              ((guix config) #:select (%guix-version))
@@ -73,6 +74,27 @@ superseded packages."
 ;;; Required by 'origin->json' for 'computed-origin-method' corner cases
 (define gexp-references (@@ (guix gexp) gexp-references))
 
+(define %content-addressed-mirrors
+  ;; List of content-addressed mirrors.
+  ;; XXX: somewhat duplicated from (guix download)
+  (let ((guix-publish
+         (lambda (host)
+           (lambda (file hash)
+             ;; Files served by 'guix publish'.
+             (string-append "https://" host "/file/"
+                            file "/" (symbol->string
+                                      (content-hash-algorithm hash))
+                            "/" (bytevector->nix-base32-string
+                                 (content-hash-value hash)))))))
+
+    (list (guix-publish "bordeaux.guix.gnu.org")
+          (guix-publish "ci.guix.gnu.org")
+          (lambda (file hash)
+            (string-append "https://tarballs.nixos.org/"
+                           (symbol->string (content-hash-algorithm hash))
+                           "/" (bytevector->nix-base32-string
+                                (content-hash-value hash)))))))
+
 (define (origin->json origin)
   "Return a list of JSON representations (an alist) of ORIGIN."
   (define method
@@ -81,10 +103,17 @@ superseded packages."
   (define uri
     (origin-uri origin))
 
-  (define (resolve urls)
-    (map uri->string
-         (append-map (cut maybe-expand-mirrors <> %mirrors)
-                     (map string->uri urls))))
+  (define (resolve urls hash)
+    (append (map uri->string
+                 (append-map (cut maybe-expand-mirrors <> %mirrors)
+                             (map string->uri urls)))
+            (if hash
+                (let ((file (origin-actual-file-name origin))
+                      (hash (origin-hash origin)))
+                  (map (lambda (make-url)
+                         (make-url file hash))
+                       %content-addressed-mirrors))
+                '())))
 
   (if (eq? method (@@ (guix packages) computed-origin-method))
       ;; Packages in gnu/packages/gnuzilla.scm and gnu/packages/linux.scm
@@ -118,7 +147,8 @@ superseded packages."
                                 (resolve
                                  (match uri
                                    ((? string? url) (list url))
-                                   ((urls ...) urls)))))))
+                                   ((urls ...) urls))
+                                 (origin-hash origin))))))
                  ((eq? git-fetch method)
                   `(("git_url" . ,(git-reference-url uri))))
                  ((eq? svn-fetch method)

Re: Adding content-addressed URLs to https://guix.gnu.org/sources.json

[Simon Tournier]

Hi,

On Mon, 24 Apr 2023 at 18:41, Ludovic Courtès <ludovic.courtes@inria.fr> wrote:

> How does that sound?

The patch LGTM.

Cheers,
simon


PS: On a side note, building sources.json fails since ~1 month.

--8<---------------cut here---------------start------------->8---
$ guix repl -- build-package-metadata.scm /tmp/
guix repl: package metadata will be written to '/tmp/'
Backtrace:

[...]

In procedure %origin-patches-real: Wrong type argument: #<<computed-file> name: "ruby-sorbet-runtime-0.5.10610.20230106174520-1fa668010-checkout" gexp: #<gexp (begin (use-modules (guix build utils)) (copy-recursively (string-append #<gexp-input #<origin #<<git-reference> url: "https://github.com/sorbet/sorbet" commit: "0.5.10610.20230106174520-1fa668010" recursive?: #f> #<content-hash sha256:0f21dl06alxwn6xgdxyrkd58plmmsv04z2bcls9ld4cfzsrs5537> () 7fa67b7f6000>:out> "/gems/sorbet-" #<gexp-input "runtime":out>) #<gexp-output out>)) gnu/packages/ruby.scm:14078:5 7fa67b701c90> guile: #f options: (#:local-build? #t)>
--8<---------------cut here---------------end--------------->8---


Somehow, it reveals 3 currently uncovered cases: computed-file appearing
as,

 1. ’origin’ in source field (ruby-sorbet-runtime)
 2. ’inputs’ (racket-minimal)
 3. ’snippet’ in origin in source field (chromium)

And I think we should not try to fix these cases on
build-package-metadata.scm side but instead investigate why they are
needed in the first place.


The regression in build-package-metadata is introduced by commit
7405e0c83f8f4eee5c4c40e809a40f2e4407a38c:

  CommitDate: Tue Mar 28 22:22:24 2023 -0400
  Author:     Maxim Cournoyer <maxim.cournoyer@gmail.com>

  gnu: Add ruby-sorbet-runtime.

  * gnu/packages/ruby.scm (ruby-sorbet-runtime): New variable.
  (sorbet-version): New variable.
  (sorbet-monorepo): New variable.
  (make-sorbet-gem-source): New procedure.

    (define (make-sorbet-gem-source gem)
      "Return the source of GEM, a sub-directory."
      (computed-file
       (string-append "ruby-sorbet-" gem "-" sorbet-version "-checkout")
       (with-imported-modules (source-module-closure '((guix build utils)))
         #~(begin
             (use-modules (guix build utils))
             (copy-recursively (string-append #$sorbet-monorepo
                                              "/gems/sorbet-" #$gem)
                               #$output)))))
[...]

    (define-public ruby-sorbet-runtime
      (package
        (name "ruby-sorbet-runtime")
        (version sorbet-version)
        (source (make-sorbet-gem-source "runtime"))


This pattern appears to me wrong.  It should use ’snippet’.  And if the
point is to have a meaningful path in /gnu/store, as it is the case with
icecat or linux or emacs-company-box, the current way is
’computed-origin-method’ from (guix packages).

For ’ruby-sorbet-runtime’, the fix seems ’computed-origin-method’.

Well, some data behind ’computed-file’ appears in two other places:

    ./gnu/packages/racket.scm:505:  (computed-file
    ./gnu/packages/chromium.scm:348:  (computed-file

For ’chromium’, it appears to me somehow overcomplicated to extract all
the layers of patches; the work does not appears to me worth.  Somehow,
I would accept that “our way“ to build Chromium would be lost.

It remains the case of ’racket-minimal’ – which is somehow a corner case
of ’package-direct-sources’,

--8<---------------cut here---------------start------------->8---
scheme@(guix-user)> ,use(gnu packages racket)
scheme@(guix-user)> racket-minimal
$1 = #<package racket-minimal@8.7 gnu/packages/racket.scm:542 7f88dea4b630>
scheme@(guix-user)> (package-direct-sources racket-minimal)
$2 = ()
--8<---------------cut here---------------end--------------->8---

However, the case is indeed covered by the chain:

 + racket-minimal lists the input racket-vm-cs
 + racket-vm-cs inherits from racket-vm-bc
 + racket-vm-bc inherits from racket-vm-cgc
 + racket-vm-cgc has the source %racket-origin

--8<---------------cut here---------------start------------->8---
scheme@(guix-user)> (package-direct-sources (@@ (gnu packages racket) racket-vm-cgc))
$3 = (#<origin #<<git-reference> url: "https://github.com/racket/racket" commit: "v8.7" recursive?: #f> #<content-hash sha256:0agwa1nrv8mizkqg9nffjli00djyx1r9n6y6b6ry7k13pb6i7xnj> ("/gnu/store/g6xlx4ik4skazfdl0a9vbd5r0lmqnnd9-guix-module-union/share/guile/site/3.0/gnu/packages/patches/racket-backport-8.7-pkg-strip.patch" "/gnu/store/g6xlx4ik4skazfdl0a9vbd5r0lmqnnd9-guix-module-union/share/guile/site/3.0/gnu/packages/patches/racket-chez-scheme-bin-sh.patch" "/gnu/store/g6xlx4ik4skazfdl0a9vbd5r0lmqnnd9-guix-module-union/share/guile/site/3.0/gnu/packages/patches/racket-rktio-bin-sh.patch" "/gnu/store/g6xlx4ik4skazfdl0a9vbd5r0lmqnnd9-guix-module-union/share/guile/site/3.0/gnu/packages/patches/racket-zuo-bin-sh.patch") 7f88dea48720>)
--8<---------------cut here---------------end--------------->8---

Therefore the source data of ’racket-minimal’ is correctly archived.


WDYT to modify ’ruby-sorbet-runtime’ and let ’chromium’ uncovered?


Cheers,
simon

Re: Adding content-addressed URLs to https://guix.gnu.org/sources.json

[Ludovic Courtès]

Hi!

Simon Tournier <zimon.toutoune@gmail.com> skribis:

> PS: On a side note, building sources.json fails since ~1 month.
>
> $ guix repl -- build-package-metadata.scm /tmp/
> guix repl: package metadata will be written to '/tmp/'
> Backtrace:
>
> [...]
>
> In procedure %origin-patches-real: Wrong type argument: #<<computed-file> name: "ruby-sorbet-runtime-0.5.10610.20230106174520-1fa668010-checkout" gexp: #<gexp (begin (use-modules (guix build utils)) (copy-recursively (string-append #<gexp-input #<origin #<<git-reference> url: "https://github.com/sorbet/sorbet" commit: "0.5.10610.20230106174520-1fa668010" recursive?: #f> #<content-hash sha256:0f21dl06alxwn6xgdxyrkd58plmmsv04z2bcls9ld4cfzsrs5537> () 7fa67b7f6000>:out> "/gems/sorbet-" #<gexp-input "runtime":out>) #<gexp-output out>)) gnu/packages/ruby.scm:14078:5 7fa67b701c90> guile: #f options: (#:local-build? #t)>

I believe this is fixed by 1df54ceab38873dfc0fd2ec27313d7d17c79c350.
(It broke the ‘source’ and ‘disarchive’ jobsets as well.)

> Somehow, it reveals 3 currently uncovered cases: computed-file appearing
> as,
>
>  1. ’origin’ in source field (ruby-sorbet-runtime)
>  2. ’inputs’ (racket-minimal)
>  3. ’snippet’ in origin in source field (chromium)

I think #1 and #2 are okay: we can use any file-like object there, not
just origin/package.  Of course, <origin> is meant to be the best choice
for ‘source’, and <package> the best choice for ‘inputs’.  But I think
it’s fine to occasionally resort to some other abstraction when these
two are not adequate.

Ludo’.

Re: Adding content-addressed URLs to https://guix.gnu.org/sources.json

[Ludovic Courtès]

Hello,

(+Cc: guix-sysadmin)

Simon Tournier <zimon.toutoune@gmail.com> skribis:

> On Mon, 24 Apr 2023 at 18:41, Ludovic Courtès <ludovic.courtes@inria.fr> wrote:
>
>> How does that sound?
>
> The patch LGTM.

Applied in maintenance.git commit
b7af47ceb629b2267ea8c28dbe5750a456be0b8f.

I’ll reconfigure berlin soonish to it runs this updated script.

BTW, I noticed that many ci.guix.gnu.org/file URLs are 404, when those
are bordeaux.guix are not.  This may be due to a more aggressive GC
policy on berlin, since ‘guix publish’ serves /file URLs right from the
store.  We have an nginx cache for /file URLs (see
‘hydra/nginx/berlin.scm’) but it’s no good if files don’t get a change
to reach it before they’re removed from the store.  We should look more
closely into that!

Thanks,
Ludo’.

Re: Adding content-addressed URLs to https://guix.gnu.org/sources.json

[Maxim Cournoyer]

Hi Simon,

Simon Tournier <zimon.toutoune@gmail.com> writes:

[...]

> $ guix repl -- build-package-metadata.scm /tmp/
> guix repl: package metadata will be written to '/tmp/'
> Backtrace:
>
> [...]
>
> In procedure %origin-patches-real: Wrong type argument: #<<computed-file> name: "ruby-sorbet-runtime-0.5.10610.20230106174520-1fa668010-checkout" gexp: #<gexp (begin (use-modules (guix build utils)) (copy-recursively (string-append #<gexp-input #<origin #<<git-reference> url: "https://github.com/sorbet/sorbet" commit: "0.5.10610.20230106174520-1fa668010" recursive?: #f> #<content-hash sha256:0f21dl06alxwn6xgdxyrkd58plmmsv04z2bcls9ld4cfzsrs5537> () 7fa67b7f6000>:out> "/gems/sorbet-" #<gexp-input "runtime":out>) #<gexp-output out>)) gnu/packages/ruby.scm:14078:5 7fa67b701c90> guile: #f options: (#:local-build? #t)>
>
>
>
> Somehow, it reveals 3 currently uncovered cases: computed-file appearing
> as,
>
>  1. ’origin’ in source field (ruby-sorbet-runtime)
>  2. ’inputs’ (racket-minimal)
>  3. ’snippet’ in origin in source field (chromium)
>
> And I think we should not try to fix these cases on
> build-package-metadata.scm side but instead investigate why they are
> needed in the first place.
>
>
> The regression in build-package-metadata is introduced by commit
> 7405e0c83f8f4eee5c4c40e809a40f2e4407a38c:
>
>   CommitDate: Tue Mar 28 22:22:24 2023 -0400
>   Author:     Maxim Cournoyer <maxim.cournoyer@gmail.com>
>
>   gnu: Add ruby-sorbet-runtime.
>
>   * gnu/packages/ruby.scm (ruby-sorbet-runtime): New variable.
>   (sorbet-version): New variable.
>   (sorbet-monorepo): New variable.
>   (make-sorbet-gem-source): New procedure.
>
>     (define (make-sorbet-gem-source gem)
>       "Return the source of GEM, a sub-directory."
>       (computed-file
>        (string-append "ruby-sorbet-" gem "-" sorbet-version "-checkout")
>        (with-imported-modules (source-module-closure '((guix build utils)))
>          #~(begin
>              (use-modules (guix build utils))
>              (copy-recursively (string-append #$sorbet-monorepo
>                                               "/gems/sorbet-" #$gem)
>                                #$output)))))
> [...]
>
>     (define-public ruby-sorbet-runtime
>       (package
>         (name "ruby-sorbet-runtime")
>         (version sorbet-version)
>         (source (make-sorbet-gem-source "runtime"))
>
>
> This pattern appears to me wrong.  It should use ’snippet’.  And if the
> point is to have a meaningful path in /gnu/store, as it is the case with
> icecat or linux or emacs-company-box, the current way is
> ’computed-origin-method’ from (guix packages).
>
> For ’ruby-sorbet-runtime’, the fix seems ’computed-origin-method’.

Per the source, IIRC, 'computed-origin-method' is also considered a hack
or workaround around the fact that a snippet can't affect the name of a
source.  But it's a well established one.  That was indeed the rationale
here (to have meaningful top level directory names matching the source),
which is worth it in my opinion.

-- 
Thanks,
Maxim

Re: Adding content-addressed URLs to https://guix.gnu.org/sources.json

[Simon Tournier]

[-- Attachment #1: Type: text/plain, Size: 5203 bytes --]

Hi Ludo, Maxim, all,

On mar., 25 avril 2023 at 14:40, Ludovic Courtès <ludovic.courtes@inria.fr> wrote:

>> Somehow, it reveals 3 currently uncovered cases: computed-file appearing
>> as,
>>
>>  1. ’origin’ in source field (ruby-sorbet-runtime)
>>  2. ’inputs’ (racket-minimal)
>>  3. ’snippet’ in origin in source field (chromium)
>
> I think #1 and #2 are okay: we can use any file-like object there, not
> just origin/package.  Of course, <origin> is meant to be the best choice
> for ‘source’, and <package> the best choice for ‘inputs’.  But I think
> it’s fine to occasionally resort to some other abstraction when these
> two are not adequate.

I agree that any file-like object is nice.  Somehow, the issue is to
“unpack“ the information of this object.  For instance,

--8<---------------cut here---------------start------------->8---
scheme@(guix-user)> (define ruby-sorbet-runtime (@@ (gnu packages ruby) ruby-sorbet-runtime))
scheme@(guix-user)> (package-source ruby-sorbet-runtime)
$1 = #<<computed-file> name: "ruby-sorbet-runtime-0.5.10610.20230106174520-1fa668010-checkout" gexp: #<gexp (begin (use-modules (guix build utils)) (copy-recursively (string-append #<gexp-input #<origin #<<git-reference> url: "https://github.com/sorbet/sorbet" commit: "0.5.10610.20230106174520-1fa668010" recursive?: #f> #<content-hash sha256:0f21dl06alxwn6xgdxyrkd58plmmsv04z2bcls9ld4cfzsrs5537> () 7fd7ad6b81e0>:out> "/gems/sorbet-" #<gexp-input "runtime":out>) #<gexp-output out>)) gnu/packages/ruby.scm:14071:5 7fd7ae734480> guile: #f options: (#:local-build? #t)>
--8<---------------cut here---------------end--------------->8---

and as far as I understand, this case cannot be handled by some generic
code.  The extraction of the “real” origin needs manual and specific
extraction because of this ’computed-file’.

For sure, ’source’ can use any file-like object because some use-cases
require that.  However, I would be tempted to use an ’origin’ as a
preferred choice – i.e., when it’s possible and try to make it
possible. ;-) Because, somehow, it “normalizes“ the source information
and eases its extraction.


On mar., 25 avril 2023 at 09:52, Maxim Cournoyer <maxim.cournoyer@gmail.com> wrote:

>> This pattern appears to me wrong.  It should use ’snippet’.  And if the
>> point is to have a meaningful path in /gnu/store, as it is the case with
>> icecat or linux or emacs-company-box, the current way is
>> ’computed-origin-method’ from (guix packages).
>>
>> For ’ruby-sorbet-runtime’, the fix seems ’computed-origin-method’.
>
> Per the source, IIRC, 'computed-origin-method' is also considered a hack
> or workaround around the fact that a snippet can't affect the name of a
> source.  But it's a well established one.  That was indeed the rationale
> here (to have meaningful top level directory names matching the source),
> which is worth it in my opinion.

I agree that ’computed-origin-method’ had been considered as a hack.  I
guess, mainly because at the time, the need seemed singular and that
’snippet’ were maybe less powerful.  For what my opinion is worth,
instead of,

--8<---------------cut here---------------start------------->8---
(define (make-sorbet-gem-source gem)
  "Return the source of GEM, a sub-directory."
  (computed-file
   (string-append "ruby-sorbet-" gem "-" sorbet-version "-checkout")
   (with-imported-modules (source-module-closure '((guix build utils)))
     #~(begin
         (use-modules (guix build utils))
         (copy-recursively (string-append #$sorbet-monorepo
                                          "/gems/sorbet-" #$gem)
                           #$output)))))

(define-public ruby-sorbet-runtime
[...]
    (source (make-sorbet-gem-source "runtime"))
--8<---------------cut here---------------end--------------->8---

I would try to “normalize” this behaviour.  Well, it’s not clear for me
if ’computed-origin-method’ is suitable as a basis for that; somehow it
would appear to me the direction for these kind of use cases.

Anyway, from my point of view, considering this very specific use-case
of ruby-sorbet-runtime, the pattern using ’computed-file’ as ’source’ –
only for having the correct store pathname – is not something that I
would introduce when it is avoidable.

And, maybe I am missing a point, but it appears to me avoidable:

--8<---------------cut here---------------start------------->8---
$ ./pre-inst-env guix build ruby-sorbet-runtime -S
/gnu/store/ni5mz1j7lbdrdqsvdm5dq1d2ack8c8q6-ruby-sorbet-runtime-0.5.10610.20230106174520-1fa668010-checkout

$ guix hash -r $(guix build ruby-sorbet-runtime -S) $(./pre-inst-env guix build ruby-sorbet-runtime -S)
0agzz44qqq5pxqzzpxwhzlbwwc7x20jrmbmaxj6q8a5bq9ydzws7
0agzz44qqq5pxqzzpxwhzlbwwc7x20jrmbmaxj6q8a5bq9ydzws7
--8<---------------cut here---------------end--------------->8---

using a ’snippet’ – see below.  Basically, ’file-name’ does the job for
the correct name and the other part is just moving content around.
Well, I guess this snippet could be simplified.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: snippet.patch --]
[-- Type: text/x-diff, Size: 1659 bytes --]

diff --git a/gnu/packages/ruby.scm b/gnu/packages/ruby.scm
index 1dcd5f76a5..f61044f76b 100644
--- a/gnu/packages/ruby.scm
+++ b/gnu/packages/ruby.scm
@@ -14078,7 +14078,35 @@ (define-public ruby-sorbet-runtime
   (package
     (name "ruby-sorbet-runtime")
     (version sorbet-version)
-    (source (make-sorbet-gem-source "runtime"))
+    (source
+     (origin
+       (inherit sorbet-monorepo)
+       (file-name (git-file-name name version))
+       (modules '((guix build utils)
+                  (ice-9 ftw)))
+       (snippet
+        #~(begin
+            (use-modules (guix build utils))
+            (let* ((name "runtime")
+                   (gems "gems")
+                   (gem (string-append gems "/sorbet-" name)))
+              (for-each
+               (lambda (x)
+                 (unless (or (string=? "." x)
+                             (string=? ".." x))
+                   (delete-file-recursively x)))
+               (scandir "."
+                        (lambda (x)
+                          (and (eq? (stat:type (stat x)) 'directory)
+                               (not (string-prefix? gems x))))))
+              (for-each
+               delete-file
+               (find-files "."
+                           (lambda (x _)
+                             (not (string-prefix? (string-append "./" gems) x)))
+                           #:directories? #f))
+              (copy-recursively gem ".")
+              (delete-file-recursively gems))))))
     (build-system ruby-build-system)
     ;; 25 out of 841 tests currently fail, seemingly due to invalid
     ;; assumptions about file names in the build environment (see:

[-- Attachment #3: Type: text/plain, Size: 16 bytes --]



Cheers,
simon

Re: Adding content-addressed URLs to https://guix.gnu.org/sources.json

[Maxim Cournoyer]

Hi Simon,

Simon Tournier <zimon.toutoune@gmail.com> writes:

> Hi Ludo, Maxim, all,
>
> On mar., 25 avril 2023 at 14:40, Ludovic Courtès <ludovic.courtes@inria.fr> wrote:
>
>>> Somehow, it reveals 3 currently uncovered cases: computed-file appearing
>>> as,
>>>
>>>  1. ’origin’ in source field (ruby-sorbet-runtime)
>>>  2. ’inputs’ (racket-minimal)
>>>  3. ’snippet’ in origin in source field (chromium)
>>
>> I think #1 and #2 are okay: we can use any file-like object there, not
>> just origin/package.  Of course, <origin> is meant to be the best choice
>> for ‘source’, and <package> the best choice for ‘inputs’.  But I think
>> it’s fine to occasionally resort to some other abstraction when these
>> two are not adequate.
>
> I agree that any file-like object is nice.  Somehow, the issue is to
> “unpack“ the information of this object.  For instance,
>
> scheme@(guix-user)> (define ruby-sorbet-runtime (@@ (gnu packages ruby) ruby-sorbet-runtime))
> scheme@(guix-user)> (package-source ruby-sorbet-runtime)
> $1 = #<<computed-file> name: "ruby-sorbet-runtime-0.5.10610.20230106174520-1fa668010-checkout" gexp: #<gexp (begin (use-modules (guix build utils)) (copy-recursively (string-append #<gexp-input #<origin #<<git-reference> url: "https://github.com/sorbet/sorbet" commit: "0.5.10610.20230106174520-1fa668010" recursive?: #f> #<content-hash sha256:0f21dl06alxwn6xgdxyrkd58plmmsv04z2bcls9ld4cfzsrs5537> () 7fd7ad6b81e0>:out> "/gems/sorbet-" #<gexp-input "runtime":out>) #<gexp-output out>)) gnu/packages/ruby.scm:14071:5 7fd7ae734480> guile: #f options: (#:local-build? #t)>
>
>
> and as far as I understand, this case cannot be handled by some generic
> code.  The extraction of the “real” origin needs manual and specific
> extraction because of this ’computed-file’.
>
> For sure, ’source’ can use any file-like object because some use-cases
> require that.  However, I would be tempted to use an ’origin’ as a
> preferred choice – i.e., when it’s possible and try to make it
> possible. ;-) Because, somehow, it “normalizes“ the source information
> and eases its extraction.

I'm not sure I follow, perhaps because I lack context about how
Disarchiver use the source field of a package.  Would you mind
explaining a bit what the problem is or pointing me to a place it was
already explained?

-- 
Thanks,
Maxim

Re: Adding content-addressed URLs to https://guix.gnu.org/sources.json

[Ludovic Courtès]

Hello,

Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

> Simon Tournier <zimon.toutoune@gmail.com> writes:

[...]

>> I agree that any file-like object is nice.  Somehow, the issue is to
>> “unpack“ the information of this object.  For instance,
>>
>> scheme@(guix-user)> (define ruby-sorbet-runtime (@@ (gnu packages ruby) ruby-sorbet-runtime))
>> scheme@(guix-user)> (package-source ruby-sorbet-runtime)
>> $1 = #<<computed-file> name: "ruby-sorbet-runtime-0.5.10610.20230106174520-1fa668010-checkout" gexp: #<gexp (begin (use-modules (guix build utils)) (copy-recursively (string-append #<gexp-input #<origin #<<git-reference> url: "https://github.com/sorbet/sorbet" commit: "0.5.10610.20230106174520-1fa668010" recursive?: #f> #<content-hash sha256:0f21dl06alxwn6xgdxyrkd58plmmsv04z2bcls9ld4cfzsrs5537> () 7fd7ad6b81e0>:out> "/gems/sorbet-" #<gexp-input "runtime":out>) #<gexp-output out>)) gnu/packages/ruby.scm:14071:5 7fd7ae734480> guile: #f options: (#:local-build? #t)>
>>
>>
>> and as far as I understand, this case cannot be handled by some generic
>> code.  The extraction of the “real” origin needs manual and specific
>> extraction because of this ’computed-file’.
>>
>> For sure, ’source’ can use any file-like object because some use-cases
>> require that.  However, I would be tempted to use an ’origin’ as a
>> preferred choice – i.e., when it’s possible and try to make it
>> possible. ;-) Because, somehow, it “normalizes“ the source information
>> and eases its extraction.

Oh, got it.

> I'm not sure I follow, perhaps because I lack context about how
> Disarchiver use the source field of a package.  Would you mind
> explaining a bit what the problem is or pointing me to a place it was
> already explained?

The problem with the idiom used for ‘ruby-sorbet-runtime’ is that the
origin is hidden inside a gexp.

Thus, the machinery that produces <https://guix.gnu.org/sources.json>
(the file that SWH fetches periodically to ingest the source code
packages refer to) will not add it.  To put it differently, we have no
guarantee that the commit above will be archived and that we’ll
eventually be able to rebuild ‘ruby-sorbet-runtime’.

So I guess as a matter of policy, we should try and find other ways to
express this so we don’t lose track of origins.

In this particular case, we could do what Simon proposed or, even
simpler, just add a phase that calls ‘chdir’ (the advantage being that
we don’t have to create copies of subsets of ‘sorbet-monorepo’).

WDYT?

Ludo’.

Re: Adding content-addressed URLs to https://guix.gnu.org/sources.json

[Maxim Cournoyer]

Hi Ludo,

Ludovic Courtès <ludovic.courtes@inria.fr> writes:

> Hello,
>
> Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:
>
>> Simon Tournier <zimon.toutoune@gmail.com> writes:
>
> [...]
>
>>> I agree that any file-like object is nice.  Somehow, the issue is to
>>> “unpack“ the information of this object.  For instance,
>>>
>>> scheme@(guix-user)> (define ruby-sorbet-runtime (@@ (gnu packages ruby) ruby-sorbet-runtime))
>>> scheme@(guix-user)> (package-source ruby-sorbet-runtime)
>>> $1 = #<<computed-file> name:
>>> "ruby-sorbet-runtime-0.5.10610.20230106174520-1fa668010-checkout"
>>> gexp: #<gexp (begin (use-modules (guix build utils))
>>> (copy-recursively (string-append #<gexp-input #<origin
>>> #<<git-reference> url: "https://github.com/sorbet/sorbet" commit:
>>> "0.5.10610.20230106174520-1fa668010" recursive?: #f> #<content-hash
>>> sha256:0f21dl06alxwn6xgdxyrkd58plmmsv04z2bcls9ld4cfzsrs5537> ()
>>> 7fd7ad6b81e0>:out> "/gems/sorbet-" #<gexp-input "runtime":out>)
>>> #<gexp-output out>)) gnu/packages/ruby.scm:14071:5 7fd7ae734480>
>>> guile: #f options: (#:local-build? #t)>
>>>
>>>
>>> and as far as I understand, this case cannot be handled by some generic
>>> code.  The extraction of the “real” origin needs manual and specific
>>> extraction because of this ’computed-file’.
>>>
>>> For sure, ’source’ can use any file-like object because some use-cases
>>> require that.  However, I would be tempted to use an ’origin’ as a
>>> preferred choice – i.e., when it’s possible and try to make it
>>> possible. ;-) Because, somehow, it “normalizes“ the source information
>>> and eases its extraction.
>
> Oh, got it.
>
>> I'm not sure I follow, perhaps because I lack context about how
>> Disarchiver use the source field of a package.  Would you mind
>> explaining a bit what the problem is or pointing me to a place it was
>> already explained?
>
> The problem with the idiom used for ‘ruby-sorbet-runtime’ is that the
> origin is hidden inside a gexp.
>
> Thus, the machinery that produces <https://guix.gnu.org/sources.json>
> (the file that SWH fetches periodically to ingest the source code
> packages refer to) will not add it.  To put it differently, we have no
> guarantee that the commit above will be archived and that we’ll
> eventually be able to rebuild ‘ruby-sorbet-runtime’.
>
> So I guess as a matter of policy, we should try and find other ways to
> express this so we don’t lose track of origins.

Thanks for explaining, it makes sense.  One way to tip package writers
in the right direction would be to add a warning when such a situation
occurs upon running 'guix lint'.

> In this particular case, we could do what Simon proposed or, even
> simpler, just add a phase that calls ‘chdir’ (the advantage being that
> we don’t have to create copies of subsets of ‘sorbet-monorepo’).

I seem to recall there was more to it than simply having a
self-contained source correctly named, such as avoiding extra phases
doing chdir and having to delete extraneous .gems that'd get picked up
erroneously and build the wrong thing/fail the build.

Simon's solution is a bit more complicated/verbose than the gexp version
currently in used taste, but it works while preserving the origin and
avoids extra steps in each package that will be using sorbet-monorepo,
so I think I'd go with it.

-- 
Thanks,
Maxim

Re: Adding content-addressed URLs to https://guix.gnu.org/sources.json

[Simon Tournier]

Hi,

On Tue, 02 May 2023 at 08:52, Maxim Cournoyer <maxim.cournoyer@gmail.com> wrote:

> Thanks for explaining, it makes sense.  One way to tip package writers
> in the right direction would be to add a warning when such a situation
> occurs upon running 'guix lint'.

Somehow improve “guix lint -c archival”. ;-)


> Simon's solution is a bit more complicated/verbose than the gexp version
> currently in used taste, but it works while preserving the origin and
> avoids extra steps in each package that will be using sorbet-monorepo,
> so I think I'd go with it.

My point was to show that ’snippet’ could produce the exact same output
but exposing the ’origin’ usable by third-party (sources.json or else).
Maybe the snippet can be simplified. :-)

Another solution would to list ’sorbet-monorepo’ as an input and then
apply a generic build phase dealing with mono repo.  Hum, I think the
’snippet’ approach is better. :_

Note ’racket-packages-origin’ from (gnu packages racket) looks similar:

--8<---------------cut here---------------start------------->8---
  (computed-file
   (string-append "racket-pkg-" name "-sources")
   (with-imported-modules `((guix build utils))
     #~(begin
         (use-modules (guix build utils))
         (mkdir-p (string-append #$output "/share/racket/pkgs"))
         (chdir (string-append #$output "/share/racket/pkgs"))
         #$@(map (match-lambda
                   ((? string? name)
                    #~(copy-recursively #$(file-append origin (string-append "/" name))
                                        #$name))
                   ((name ".")
                    #~(copy-recursively #$origin #$name))
                   ((name path)
                    #~(copy-recursively #$(file-append origin (string-append "/" path))
                                        #$name)))
                 specs)))))
--8<---------------cut here---------------end--------------->8---

and that somehow uses the ’inputs’ approach.  Well, contrary to what I
said earlier in this thread, not all the sources for building Racket are
captured by sources.json:

--8<---------------cut here---------------start------------->8---

$ wget https://guix.gnu.org/sources.json
$ guix shell jq
$ cat sources.json | jq | grep sorbet

$ cat sources.json | jq | grep 'github.com/racket'
      "git_url": "https://github.com/racket/racket",
--8<---------------cut here---------------end--------------->8---

As you see, it requires,

    https://github.com/racket/2d

as input and that’s not listed.  I will give a look later if no one
beats me. :-)


Somehow, my initial point when mentioning ’computed-origin-method’ is
that maybe there is a need.  Maybe it’s an opportunity to rehash this
“method”.  Or another that provides 1. the structure we need for
extracting and 2. the flexibility we need for special use cases.

Cheers,
simon

Re: Adding content-addressed URLs to https://guix.gnu.org/sources.json

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 903 bytes --]

Hello!

Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

> Ludovic Courtès <ludovic.courtes@inria.fr> writes:

[...]

>> So I guess as a matter of policy, we should try and find other ways to
>> express this so we don’t lose track of origins.
>
> Thanks for explaining, it makes sense.  One way to tip package writers
> in the right direction would be to add a warning when such a situation
> occurs upon running 'guix lint'.

How about the attached patch?

--8<---------------cut here---------------start------------->8---
$ ./pre-inst-env guix lint -c archival ruby-sorbet-runtime racket
gnu/packages/ruby.scm:14081:12: ruby-sorbet-runtime@0.5.10610.20230106174520-1fa668010: source is not an origin, it cannot be archived
--8<---------------cut here---------------end--------------->8---

(The version string of this package looks way too long.)

Thanks,
Ludo’.


[-- Attachment #2: the patch --]
[-- Type: text/x-patch, Size: 3708 bytes --]

From 153cb8e5782946bd0c9de22022a475ae9fef70ea Mon Sep 17 00:00:00 2001
Message-Id: <153cb8e5782946bd0c9de22022a475ae9fef70ea.1683184269.git.ludo@gnu.org>
From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= <ludo@gnu.org>
Date: Thu, 4 May 2023 09:09:03 +0200
Subject: [PATCH] lint: archival: Warn against non-origin package sources.

Suggested by Maxim Cournoyer <maxim.cournoyer@gmail.com>
and Simon Tournier <zimon.toutoune@gmail.com>.

* guix/lint.scm (check-archival): Add 'local-file?' clause.  Clarify
message in case (package-source package) is not an origin.
* tests/lint.scm ("archival: not an origin"): New test.
---
 guix/lint.scm  | 11 +++++++----
 tests/lint.scm | 11 +++++++++--
 2 files changed, 16 insertions(+), 6 deletions(-)

diff --git a/guix/lint.scm b/guix/lint.scm
index 0ed5b8dc98..72b3f4e7b1 100644
--- a/guix/lint.scm
+++ b/guix/lint.scm
@@ -1610,11 +1610,11 @@ (define (check-archival package)
   (parameterize ((%allow-request? skip-when-limit-reached))
     (catch #t
       (lambda ()
-        (match (and (origin? (package-source package))
-                    (package-source package))
+        (match (package-source package)
           (#f                                     ;no source
            '())
-          ((= origin-uri (? git-reference? reference))
+          ((and (? origin?)
+                (= origin-uri (? git-reference? reference)))
            (define url
              (git-reference-url reference))
            (define commit
@@ -1680,9 +1680,12 @@ (define (check-archival package)
                    ((? content?)
                     '())))
                '()))
+          ((? local-file?)
+           '())
           (_
            (list (make-warning package
-                               (G_ "unsupported source type")
+                               (G_ "\
+source is not an origin, it cannot be archived")
                                #:field 'source)))))
       (match-lambda*
         (('swh-error url method response)
diff --git a/tests/lint.scm b/tests/lint.scm
index ce22e2355a..b91bd053c5 100644
--- a/tests/lint.scm
+++ b/tests/lint.scm
@@ -1,7 +1,7 @@
 ;;; GNU Guix --- Functional package management for GNU
 ;;; Copyright © 2012, 2013 Cyril Roelandt <tipecaml@gmail.com>
 ;;; Copyright © 2014, 2015, 2016 Eric Bavier <bavier@member.fsf.org>
-;;; Copyright © 2014-2022 Ludovic Courtès <ludo@gnu.org>
+;;; Copyright © 2014-2023 Ludovic Courtès <ludo@gnu.org>
 ;;; Copyright © 2015, 2016 Mathieu Lirzin <mthl@gnu.org>
 ;;; Copyright © 2016 Hartmut Goebel <h.goebel@crazy-compilers.com>
 ;;; Copyright © 2017 Alex Kost <alezost@gmail.com>
@@ -43,7 +43,8 @@ (define-module (test-lint)
   #:use-module (guix lint)
   #:use-module (guix ui)
   #:use-module (guix swh)
-  #:use-module ((guix gexp) #:select (gexp local-file gexp?))
+  #:use-module ((guix gexp)
+                #:select (gexp local-file computed-file gexp?))
   #:use-module ((guix utils) #:select (call-with-temporary-directory))
   #:use-module ((guix import hackage) #:select (%hackage-url))
   #:use-module ((guix import stackage) #:select (%stackage-url))
@@ -1298,6 +1299,12 @@ (define (package-with-phase-changes changes)
   '()
   (check-formatting (dummy-package "x")))
 
+(test-assert "archival: not an origin"
+  (warning-contains? "not an origin"
+                     (check-archival
+                      (dummy-package
+                       "x" (source (computed-file "x-src" #t))))))
+
 (test-assert "archival: missing content"
   (let* ((origin   (origin
                      (method url-fetch)

base-commit: 44d70440944aa47e5f37493346e560f38907d777
-- 
2.39.2

Re: Adding content-addressed URLs to https://guix.gnu.org/sources.json

[Ludovic Courtès]

Hi,

Ludovic Courtès <ludovic.courtes@inria.fr> skribis:

> How about the attached patch?
>
> $ ./pre-inst-env guix lint -c archival ruby-sorbet-runtime racket
> gnu/packages/ruby.scm:14081:12: ruby-sorbet-runtime@0.5.10610.20230106174520-1fa668010: source is not an origin, it cannot be archived
>
> (The version string of this package looks way too long.)
>
> Thanks,
> Ludo’.
>
> From 153cb8e5782946bd0c9de22022a475ae9fef70ea Mon Sep 17 00:00:00 2001
> Message-Id: <153cb8e5782946bd0c9de22022a475ae9fef70ea.1683184269.git.ludo@gnu.org>
> From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= <ludo@gnu.org>
> Date: Thu, 4 May 2023 09:09:03 +0200
> Subject: [PATCH] lint: archival: Warn against non-origin package sources.
>
> Suggested by Maxim Cournoyer <maxim.cournoyer@gmail.com>
> and Simon Tournier <zimon.toutoune@gmail.com>.
>
> * guix/lint.scm (check-archival): Add 'local-file?' clause.  Clarify
> message in case (package-source package) is not an origin.
> * tests/lint.scm ("archival: not an origin"): New test.

Pushed as 71fd35c1d55988c413a37c7d15006b4d38d7dde7.  Let me know if
anything’s amiss!

Ludo’.

Re: Adding content-addressed URLs to https://guix.gnu.org/sources.json

[Maxim Cournoyer]

Hi,

Ludovic Courtès <ludo@gnu.org> writes:

> Hi,
>
> Ludovic Courtès <ludovic.courtes@inria.fr> skribis:
>
>> How about the attached patch?
>>
>> $ ./pre-inst-env guix lint -c archival ruby-sorbet-runtime racket
>> gnu/packages/ruby.scm:14081:12:
>> ruby-sorbet-runtime@0.5.10610.20230106174520-1fa668010: source is
>> not an origin, it cannot be archived
>>
>> (The version string of this package looks way too long.)
>>
>> Thanks,
>> Ludo’.
>>
>> From 153cb8e5782946bd0c9de22022a475ae9fef70ea Mon Sep 17 00:00:00 2001
>> Message-Id: <153cb8e5782946bd0c9de22022a475ae9fef70ea.1683184269.git.ludo@gnu.org>
>> From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= <ludo@gnu.org>
>> Date: Thu, 4 May 2023 09:09:03 +0200
>> Subject: [PATCH] lint: archival: Warn against non-origin package sources.
>>
>> Suggested by Maxim Cournoyer <maxim.cournoyer@gmail.com>
>> and Simon Tournier <zimon.toutoune@gmail.com>.
>>
>> * guix/lint.scm (check-archival): Add 'local-file?' clause.  Clarify
>> message in case (package-source package) is not an origin.
>> * tests/lint.scm ("archival: not an origin"): New test.
>
> Pushed as 71fd35c1d55988c413a37c7d15006b4d38d7dde7.  Let me know if
> anything’s amiss!

Neat, thanks for tackling it.

-- 
Thanks,
Maxim