bug#62071: openjdk@9/10 sources not reproducible

bug#62071: openjdk@9/10 sources not reproducible

[Lars-Dominik Braun]

Hi,

it looks like the (auto-generated) tarballs for openjdk@9 and openjdk@10
changed their hash, causing a hash mismatch via

    guix build -S openjdk@9 --no-substitutes --no-grafts

I’m not sure why it uses these tarballs in the first place, since we
have a hg-download.

Lars

bug#62071: openjdk@9/10 sources not reproducible

[Björn Höfling]

[-- Attachment #1: Type: text/plain, Size: 625 bytes --]

On Thu, 9 Mar 2023 10:48:53 +0100
Lars-Dominik Braun <lars@6xq.net> wrote:

> Hi,
> 
> it looks like the (auto-generated) tarballs for openjdk@9 and
> openjdk@10 changed their hash, causing a hash mismatch via
> 
>     guix build -S openjdk@9 --no-substitutes --no-grafts
 

I can confirm this.

I found the old versions of openjdk 9 and 10 on a server of mine. I
will compare old/new tomorrow (oh, better say: later today).

> I’m not sure why it uses these tarballs in the first place, since we
> have a hg-download.

I don't know. Maybe there was no hg-download yet when we added OpenJDK9?

Björn

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

bug#62071: openjdk@9/10 sources not reproducible

[Björn Höfling]

[-- Attachment #1: Type: text/plain, Size: 873 bytes --]

On Thu, 9 Mar 2023 10:48:53 +0100
Lars-Dominik Braun <lars@6xq.net> wrote:

> Hi,
> 
> it looks like the (auto-generated) tarballs for openjdk@9 and
> openjdk@10 changed their hash, causing a hash mismatch via
> 
>     guix build -S openjdk@9 --no-substitutes --no-grafts
> 
> I’m not sure why it uses these tarballs in the first place, since we
> have a hg-download.

I compared for JDK9 the two tarballs (old and new hash) and there is no
difference in the content (according to diffoscope). Also, if I
hg-clone the repository/tag (and add the .hg_archival.txt file), all
three directory trees have the same hash value according to guix hash
-rx

Thus, it seams like their artifacts are not stable, as we saw it
for autogenerated artifacts on github.

I will check the same for JDK10 and will prepare a patch within the
next two days.

Björn

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

bug#62071: openjdk@9/10 sources not reproducible

[Simon Tournier]

Hi,

On dim., 12 mars 2023 at 22:00, Björn Höfling <bjoern.hoefling@bjoernhoefling.de> wrote:

> I compared for JDK9 the two tarballs (old and new hash) and there is no
> difference in the content (according to diffoscope). Also, if I
> hg-clone the repository/tag (and add the .hg_archival.txt file), all
> three directory trees have the same hash value according to guix hash
> -rx

So maybe it comes from some parameters of the compressor; maybe they
changed their level.  Well, I do not know how to check that.

Maybe the best is to switch from url-fetch to hg-fetch.  WDYT?

Cheers,
simon

bug#62071: openjdk@9/10 sources not reproducible

[Björn Höfling]

[-- Attachment #1: Type: text/plain, Size: 633 bytes --]

On Sun, 12 Mar 2023 22:00:21 +0100
Björn Höfling <bjoern.hoefling@bjoernhoefling.de> wrote:

> On Thu, 9 Mar 2023 10:48:53 +0100
> Lars-Dominik Braun <lars@6xq.net> wrote:
> 
> > Hi,
> > 
> > it looks like the (auto-generated) tarballs for openjdk@9 and
> > openjdk@10 changed their hash, causing a hash mismatch via
> > 
> >     guix build -S openjdk@9 --no-substitutes --no-grafts
> > 
> > I’m not sure why it uses these tarballs in the first place, since we
> > have a hg-download.  

Changed to hg-download in commit(s):

7636c49b45adb9870cf416c64bde032ec858a820

Thanks for pointing this out.

Björn

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

bug#62071: openjdk@9/10 sources not reproducible

[Ludovic Courtès]

Hi Björn,

Björn Höfling <bjoern.hoefling@bjoernhoefling.de> skribis:

> I will check the same for JDK10 and will prepare a patch within the
> next two days.

Thanks for 7636c49b45adb9870cf416c64bde032ec858a820 and its parent
commit!

For the record, there are two remaining issues:

  1. Reproducibility of past revisions.  If we lose copies of the
     auto-generated tarballs, then OpenJDK in past revisions of Guix is
     irreparably lost.  We should check whether/how to get them in
     Disarchive + SWH.

  2. Mercurial/SWH bridge.  While SWH has a one-to-one mapping with Git
     (you can ask it for a specific Git commit ID), that’s not true for
     hg.  This is a more general problem, but as things are today,
     there’s no automatic SWH fallback if the upstream hg server
     vanishes.

Thanks,
Ludo’.

bug#62071: openjdk@9/10 sources not reproducible

[Jonathan Brielmaier]

I’m not sure why it uses these tarballs in the first place, since we
have a hg-download.

-> I guess a reason could be that downloading via hg is quite slow.
Thats at least my impression when fetching the "comm" repository for
Thunderbird with mecurial. Tarballs and git checkout tend to be way
faster...

bug#62071: openjdk@9/10 sources not reproducible

[Björn Höfling]

[-- Attachment #1: Type: text/plain, Size: 1721 bytes --]

On Thu, 16 Mar 2023 12:48:19 +0100
Ludovic Courtès <ludovic.courtes@inria.fr> wrote:

> Hi Björn,
> 
> Björn Höfling <bjoern.hoefling@bjoernhoefling.de> skribis:
> 
> > I will check the same for JDK10 and will prepare a patch within the
> > next two days.  
> 
> Thanks for 7636c49b45adb9870cf416c64bde032ec858a820 and its parent
> commit!
> 
> For the record, there are two remaining issues:
> 
>   1. Reproducibility of past revisions.  If we lose copies of the
>      auto-generated tarballs, then OpenJDK in past revisions of Guix
> is irreparably lost.  We should check whether/how to get them in
>      Disarchive + SWH.

How do we do that? Adding git repos to SWH is something I can achieve,
but adding tarballs to SWH was always opaque to me.

I find no reference in the manual about Disarchive. Ideally, is there a
linter for just adding a package to the disarchive database?

 
>   2. Mercurial/SWH bridge.  While SWH has a one-to-one mapping with
> Git (you can ask it for a specific Git commit ID), that’s not true for
>      hg.  This is a more general problem, but as things are today,
>      there’s no automatic SWH fallback if the upstream hg server
>      vanishes.

For git, I know and appreciate that the linter can directly add a
missing repo to SWH. During linting, I recogniced this is missing for
hg.

I was thinking a second time about it and found that not only the newer
development of OpenJDK is on GitHub, but also the older versions are
available. So I could add another patch like this: 

+              (method git-fetch)
+              (uri (git-reference
+                    (url "https://github.com/openjdk/jdk9")

WDYT?

Björn

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

bug#62071: openjdk@9/10 sources not reproducible

[Ludovic Courtès]

Hello,

Björn Höfling <bjoern.hoefling@bjoernhoefling.de> skribis:

> On Thu, 16 Mar 2023 12:48:19 +0100
> Ludovic Courtès <ludovic.courtes@inria.fr> wrote:
>
>> Hi Björn,
>> 
>> Björn Höfling <bjoern.hoefling@bjoernhoefling.de> skribis:
>> 
>> > I will check the same for JDK10 and will prepare a patch within the
>> > next two days.  
>> 
>> Thanks for 7636c49b45adb9870cf416c64bde032ec858a820 and its parent
>> commit!
>> 
>> For the record, there are two remaining issues:
>> 
>>   1. Reproducibility of past revisions.  If we lose copies of the
>>      auto-generated tarballs, then OpenJDK in past revisions of Guix
>> is irreparably lost.  We should check whether/how to get them in
>>      Disarchive + SWH.
>
> How do we do that? Adding git repos to SWH is something I can achieve,
> but adding tarballs to SWH was always opaque to me.
>
> I find no reference in the manual about Disarchive. Ideally, is there a
> linter for just adding a package to the disarchive database?

SWH periodically ingests the contents of tarballs (not tarballs
themselves) that appear in <https://guix.gnu.org/sources.json>.  We’d
need to check whether it has the contents of those tarballs.

Then <https://disarchive.guix.gnu.org> is populated by the CI job at
<https://ci.guix.gnu.org/jobset/disarchive>.

Are the openjdk 9 and 10 tarballs archived?

Let’s look at their origins as of commit
1e6ddceb8318d413745ca1c9d91fde01b1e0364b.  We can construct their
Disarchive URL by first converting their SHA256 to hex:

--8<---------------cut here---------------start------------->8---
scheme@(guile-user)> ,use(guix base32)
scheme@(guile-user)> ,use(guix base16)
scheme@(guile-user)> (bytevector->base16-string (nix-base32-string->bytevector "01ihmyf7k5z17wbr7xig7y40l9f01d5zjgkcmawn1102hw5kchpq"))
$5 = "f842360b87028460b9aa6c3ef94b0bc0250a883f2ff693173fe197799caf3006"
--8<---------------cut here---------------end--------------->8---

That gives us:

  https://disarchive.guix.gnu.org/sha256/f842360b87028460b9aa6c3ef94b0bc0250a883f2ff693173fe197799caf3006
  https://disarchive.guix.gnu.org/sha256/249fd462bdd32571c6d0a45f3cb698a305c9e4e66b275d25e990ac0184c0dc7f

Both are 404.  But like I wrote, this is expected: they are bzip2
tarballs and Disarchive doesn’t support bzip2 (yet).

>>   2. Mercurial/SWH bridge.  While SWH has a one-to-one mapping with
>> Git (you can ask it for a specific Git commit ID), that’s not true for
>>      hg.  This is a more general problem, but as things are today,
>>      there’s no automatic SWH fallback if the upstream hg server
>>      vanishes.
>
> For git, I know and appreciate that the linter can directly add a
> missing repo to SWH. During linting, I recogniced this is missing for
> hg.
>
> I was thinking a second time about it and found that not only the newer
> development of OpenJDK is on GitHub, but also the older versions are
> available. So I could add another patch like this: 
>
> +              (method git-fetch)
> +              (uri (git-reference
> +                    (url "https://github.com/openjdk/jdk9")
>
> WDYT?

That’s a good idea.  It shouldn’t change the SHA256 (if it does,
something’s wrong) so it looks like an no-brainer.

Thanks!

Ludo’.

SWH: extend sources.json and Mercurial (or not Git and not tarball)

[Simon Tournier]

Hi,

On Thu, 16 Mar 2023 at 12:48, Ludovic Courtès <ludovic.courtes@inria.fr> wrote:

>   1. Reproducibility of past revisions.  If we lose copies of the
>      auto-generated tarballs, then OpenJDK in past revisions of Guix is
>      irreparably lost.  We should check whether/how to get them in
>      Disarchive + SWH.

The file sources.json that SWH ingests only contains original upstream
and not our copies.  One step forward would be to also list the URL of
our tarball substitutes as the last mirror in sources.json.

Any taker? :-)

>
>   2. Mercurial/SWH bridge.  While SWH has a one-to-one mapping with Git
>      (you can ask it for a specific Git commit ID), that’s not true for
>      hg.  This is a more general problem, but as things are today,
>      there’s no automatic SWH fallback if the upstream hg server
>      vanishes.

Since most git-fetch origins use label tags, the one-to-one mapping is
not guarantee and we rely on SWH resolver using URL + label tag to get
the content from SWH.  For instance, if the label tag is changed
in-place by upstream pointing then to one different commit, then SWH
creates another snapshot but our fallback will fail (known issue:
history of history, etc.)

If we would have a list of identifiers instead of only NAR+SHA256, and
we could have Git commit ID here (or SWHID or others), then it would
ease the fallback machinery.

SWH folk is currently adding NAR hashes; they store it as ’ExtID’ (see
[1] and merge request [2]), but it is not clear yet how they would
expose the API entry point or if they would do.

Extending ’origin’ with another optional field using other
content-address keys would robustify the preservation of Guix.  Yeah,
indeed we could also build the X-to-SWH bridge with the Disarchive
database (global bridge) but it would appear to me better to have some
“local” origin-based bridge.

1: https://gitlab.softwareheritage.org/swh/meta/-/issues/4979
2: https://gitlab.softwareheritage.org/swh/devel/swh-loader-core/-/merge_requests/459

Cheers,
simon

bug#62071: openjdk@9/10 sources not reproducible

[Simon Tournier]

Hi,

On Mon, 20 Mar 2023 at 10:08, Ludovic Courtès <ludovic.courtes@inria.fr> wrote:

>> I was thinking a second time about it and found that not only the newer
>> development of OpenJDK is on GitHub, but also the older versions are
>> available. So I could add another patch like this: 
>>
>> +              (method git-fetch)
>> +              (uri (git-reference
>> +                    (url "https://github.com/openjdk/jdk9")
>
> That’s a good idea.  It shouldn’t change the SHA256 (if it does,
> something’s wrong) so it looks like an no-brainer.

Well, by experience, it is rare that the released tarball contain the
exact same content as the tagged commit.  If it is the case (same SHA256
for both tarball and GitHub tagged release), indeed no-brainer. :-)

Else, some work still remain for the complete preservation of Guix. ;-)

Björn, what is the status of this SHA256?

Cheers,
simon

Adding content-addressed URLs to https://guix.gnu.org/sources.json

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 1588 bytes --]

Hi,

Simon Tournier <zimon.toutoune@gmail.com> skribis:

> On Thu, 16 Mar 2023 at 12:48, Ludovic Courtès <ludovic.courtes@inria.fr> wrote:
>
>>   1. Reproducibility of past revisions.  If we lose copies of the
>>      auto-generated tarballs, then OpenJDK in past revisions of Guix is
>>      irreparably lost.  We should check whether/how to get them in
>>      Disarchive + SWH.
>
> The file sources.json that SWH ingests only contains original upstream
> and not our copies.  One step forward would be to also list the URL of
> our tarball substitutes as the last mirror in sources.json.

The patch below (against maintenance.git) does that.  The result is
something like this:

--8<---------------cut here---------------start------------->8---
{
  "type": "url",
  "urls": [
    "https://ftpmirror.gnu.org/gnu/zile/zile-2.6.2.tar.gz",
    "ftp://ftp.cs.tu-berlin.de/pub/gnu/zile/zile-2.6.2.tar.gz",
    "ftp://ftp.funet.fi/pub/mirrors/ftp.gnu.org/gnu/zile/zile-2.6.2.tar.gz",
    "http://ftp.gnu.org/pub/gnu/zile/zile-2.6.2.tar.gz",
    "https://bordeaux.guix.gnu.org/file/zile-2.6.2.tar.gz/sha256/0hf788zadmwx0xp1dhrgqcfvhwnarh6h9b51va4dr2y9yfppvsvp",
    "https://ci.guix.gnu.org/file/zile-2.6.2.tar.gz/sha256/0hf788zadmwx0xp1dhrgqcfvhwnarh6h9b51va4dr2y9yfppvsvp",
    "https://tarballs.nixos.org/sha256/0hf788zadmwx0xp1dhrgqcfvhwnarh6h9b51va4dr2y9yfppvsvp"
  ],
  "integrity": "sha256-d+t9r/PJi9yI2qGsBA3MynK4HcMvwxZuB53Xpj5Cx0E="
},
--8<---------------cut here---------------end--------------->8---

How does that sound?

Thanks,
Ludo’.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 3205 bytes --]

diff --git a/hydra/build-package-metadata.scm b/hydra/build-package-metadata.scm
index 6fa2173..1ddb409 100755
--- a/hydra/build-package-metadata.scm
+++ b/hydra/build-package-metadata.scm
@@ -30,6 +30,7 @@
              (guix utils)
              (guix gexp)
              ((guix build download) #:select (maybe-expand-mirrors))
+             ((guix base32) #:select (bytevector->nix-base32-string))
              ((guix base64) #:select (base64-encode))
              ((guix describe) #:select (current-profile))
              ((guix config) #:select (%guix-version))
@@ -73,6 +74,27 @@ superseded packages."
 ;;; Required by 'origin->json' for 'computed-origin-method' corner cases
 (define gexp-references (@@ (guix gexp) gexp-references))
 
+(define %content-addressed-mirrors
+  ;; List of content-addressed mirrors.
+  ;; XXX: somewhat duplicated from (guix download)
+  (let ((guix-publish
+         (lambda (host)
+           (lambda (file hash)
+             ;; Files served by 'guix publish'.
+             (string-append "https://" host "/file/"
+                            file "/" (symbol->string
+                                      (content-hash-algorithm hash))
+                            "/" (bytevector->nix-base32-string
+                                 (content-hash-value hash)))))))
+
+    (list (guix-publish "bordeaux.guix.gnu.org")
+          (guix-publish "ci.guix.gnu.org")
+          (lambda (file hash)
+            (string-append "https://tarballs.nixos.org/"
+                           (symbol->string (content-hash-algorithm hash))
+                           "/" (bytevector->nix-base32-string
+                                (content-hash-value hash)))))))
+
 (define (origin->json origin)
   "Return a list of JSON representations (an alist) of ORIGIN."
   (define method
@@ -81,10 +103,17 @@ superseded packages."
   (define uri
     (origin-uri origin))
 
-  (define (resolve urls)
-    (map uri->string
-         (append-map (cut maybe-expand-mirrors <> %mirrors)
-                     (map string->uri urls))))
+  (define (resolve urls hash)
+    (append (map uri->string
+                 (append-map (cut maybe-expand-mirrors <> %mirrors)
+                             (map string->uri urls)))
+            (if hash
+                (let ((file (origin-actual-file-name origin))
+                      (hash (origin-hash origin)))
+                  (map (lambda (make-url)
+                         (make-url file hash))
+                       %content-addressed-mirrors))
+                '())))
 
   (if (eq? method (@@ (guix packages) computed-origin-method))
       ;; Packages in gnu/packages/gnuzilla.scm and gnu/packages/linux.scm
@@ -118,7 +147,8 @@ superseded packages."
                                 (resolve
                                  (match uri
                                    ((? string? url) (list url))
-                                   ((urls ...) urls)))))))
+                                   ((urls ...) urls))
+                                 (origin-hash origin))))))
                  ((eq? git-fetch method)
                   `(("git_url" . ,(git-reference-url uri))))
                  ((eq? svn-fetch method)

Re: Adding content-addressed URLs to https://guix.gnu.org/sources.json

[Simon Tournier]

Hi,

On Mon, 24 Apr 2023 at 18:41, Ludovic Courtès <ludovic.courtes@inria.fr> wrote:

> How does that sound?

The patch LGTM.

Cheers,
simon


PS: On a side note, building sources.json fails since ~1 month.

--8<---------------cut here---------------start------------->8---
$ guix repl -- build-package-metadata.scm /tmp/
guix repl: package metadata will be written to '/tmp/'
Backtrace:

[...]

In procedure %origin-patches-real: Wrong type argument: #<<computed-file> name: "ruby-sorbet-runtime-0.5.10610.20230106174520-1fa668010-checkout" gexp: #<gexp (begin (use-modules (guix build utils)) (copy-recursively (string-append #<gexp-input #<origin #<<git-reference> url: "https://github.com/sorbet/sorbet" commit: "0.5.10610.20230106174520-1fa668010" recursive?: #f> #<content-hash sha256:0f21dl06alxwn6xgdxyrkd58plmmsv04z2bcls9ld4cfzsrs5537> () 7fa67b7f6000>:out> "/gems/sorbet-" #<gexp-input "runtime":out>) #<gexp-output out>)) gnu/packages/ruby.scm:14078:5 7fa67b701c90> guile: #f options: (#:local-build? #t)>
--8<---------------cut here---------------end--------------->8---


Somehow, it reveals 3 currently uncovered cases: computed-file appearing
as,

 1. ’origin’ in source field (ruby-sorbet-runtime)
 2. ’inputs’ (racket-minimal)
 3. ’snippet’ in origin in source field (chromium)

And I think we should not try to fix these cases on
build-package-metadata.scm side but instead investigate why they are
needed in the first place.


The regression in build-package-metadata is introduced by commit
7405e0c83f8f4eee5c4c40e809a40f2e4407a38c:

  CommitDate: Tue Mar 28 22:22:24 2023 -0400
  Author:     Maxim Cournoyer <maxim.cournoyer@gmail.com>

  gnu: Add ruby-sorbet-runtime.

  * gnu/packages/ruby.scm (ruby-sorbet-runtime): New variable.
  (sorbet-version): New variable.
  (sorbet-monorepo): New variable.
  (make-sorbet-gem-source): New procedure.

    (define (make-sorbet-gem-source gem)
      "Return the source of GEM, a sub-directory."
      (computed-file
       (string-append "ruby-sorbet-" gem "-" sorbet-version "-checkout")
       (with-imported-modules (source-module-closure '((guix build utils)))
         #~(begin
             (use-modules (guix build utils))
             (copy-recursively (string-append #$sorbet-monorepo
                                              "/gems/sorbet-" #$gem)
                               #$output)))))
[...]

    (define-public ruby-sorbet-runtime
      (package
        (name "ruby-sorbet-runtime")
        (version sorbet-version)
        (source (make-sorbet-gem-source "runtime"))


This pattern appears to me wrong.  It should use ’snippet’.  And if the
point is to have a meaningful path in /gnu/store, as it is the case with
icecat or linux or emacs-company-box, the current way is
’computed-origin-method’ from (guix packages).

For ’ruby-sorbet-runtime’, the fix seems ’computed-origin-method’.

Well, some data behind ’computed-file’ appears in two other places:

    ./gnu/packages/racket.scm:505:  (computed-file
    ./gnu/packages/chromium.scm:348:  (computed-file

For ’chromium’, it appears to me somehow overcomplicated to extract all
the layers of patches; the work does not appears to me worth.  Somehow,
I would accept that “our way“ to build Chromium would be lost.

It remains the case of ’racket-minimal’ – which is somehow a corner case
of ’package-direct-sources’,

--8<---------------cut here---------------start------------->8---
scheme@(guix-user)> ,use(gnu packages racket)
scheme@(guix-user)> racket-minimal
$1 = #<package racket-minimal@8.7 gnu/packages/racket.scm:542 7f88dea4b630>
scheme@(guix-user)> (package-direct-sources racket-minimal)
$2 = ()
--8<---------------cut here---------------end--------------->8---

However, the case is indeed covered by the chain:

 + racket-minimal lists the input racket-vm-cs
 + racket-vm-cs inherits from racket-vm-bc
 + racket-vm-bc inherits from racket-vm-cgc
 + racket-vm-cgc has the source %racket-origin

--8<---------------cut here---------------start------------->8---
scheme@(guix-user)> (package-direct-sources (@@ (gnu packages racket) racket-vm-cgc))
$3 = (#<origin #<<git-reference> url: "https://github.com/racket/racket" commit: "v8.7" recursive?: #f> #<content-hash sha256:0agwa1nrv8mizkqg9nffjli00djyx1r9n6y6b6ry7k13pb6i7xnj> ("/gnu/store/g6xlx4ik4skazfdl0a9vbd5r0lmqnnd9-guix-module-union/share/guile/site/3.0/gnu/packages/patches/racket-backport-8.7-pkg-strip.patch" "/gnu/store/g6xlx4ik4skazfdl0a9vbd5r0lmqnnd9-guix-module-union/share/guile/site/3.0/gnu/packages/patches/racket-chez-scheme-bin-sh.patch" "/gnu/store/g6xlx4ik4skazfdl0a9vbd5r0lmqnnd9-guix-module-union/share/guile/site/3.0/gnu/packages/patches/racket-rktio-bin-sh.patch" "/gnu/store/g6xlx4ik4skazfdl0a9vbd5r0lmqnnd9-guix-module-union/share/guile/site/3.0/gnu/packages/patches/racket-zuo-bin-sh.patch") 7f88dea48720>)
--8<---------------cut here---------------end--------------->8---

Therefore the source data of ’racket-minimal’ is correctly archived.


WDYT to modify ’ruby-sorbet-runtime’ and let ’chromium’ uncovered?


Cheers,
simon

Re: Adding content-addressed URLs to https://guix.gnu.org/sources.json

[Ludovic Courtès]

Hi!

Simon Tournier <zimon.toutoune@gmail.com> skribis:

> PS: On a side note, building sources.json fails since ~1 month.
>
> $ guix repl -- build-package-metadata.scm /tmp/
> guix repl: package metadata will be written to '/tmp/'
> Backtrace:
>
> [...]
>
> In procedure %origin-patches-real: Wrong type argument: #<<computed-file> name: "ruby-sorbet-runtime-0.5.10610.20230106174520-1fa668010-checkout" gexp: #<gexp (begin (use-modules (guix build utils)) (copy-recursively (string-append #<gexp-input #<origin #<<git-reference> url: "https://github.com/sorbet/sorbet" commit: "0.5.10610.20230106174520-1fa668010" recursive?: #f> #<content-hash sha256:0f21dl06alxwn6xgdxyrkd58plmmsv04z2bcls9ld4cfzsrs5537> () 7fa67b7f6000>:out> "/gems/sorbet-" #<gexp-input "runtime":out>) #<gexp-output out>)) gnu/packages/ruby.scm:14078:5 7fa67b701c90> guile: #f options: (#:local-build? #t)>

I believe this is fixed by 1df54ceab38873dfc0fd2ec27313d7d17c79c350.
(It broke the ‘source’ and ‘disarchive’ jobsets as well.)

> Somehow, it reveals 3 currently uncovered cases: computed-file appearing
> as,
>
>  1. ’origin’ in source field (ruby-sorbet-runtime)
>  2. ’inputs’ (racket-minimal)
>  3. ’snippet’ in origin in source field (chromium)

I think #1 and #2 are okay: we can use any file-like object there, not
just origin/package.  Of course, <origin> is meant to be the best choice
for ‘source’, and <package> the best choice for ‘inputs’.  But I think
it’s fine to occasionally resort to some other abstraction when these
two are not adequate.

Ludo’.

Re: Adding content-addressed URLs to https://guix.gnu.org/sources.json

[Ludovic Courtès]

Hello,

(+Cc: guix-sysadmin)

Simon Tournier <zimon.toutoune@gmail.com> skribis:

> On Mon, 24 Apr 2023 at 18:41, Ludovic Courtès <ludovic.courtes@inria.fr> wrote:
>
>> How does that sound?
>
> The patch LGTM.

Applied in maintenance.git commit
b7af47ceb629b2267ea8c28dbe5750a456be0b8f.

I’ll reconfigure berlin soonish to it runs this updated script.

BTW, I noticed that many ci.guix.gnu.org/file URLs are 404, when those
are bordeaux.guix are not.  This may be due to a more aggressive GC
policy on berlin, since ‘guix publish’ serves /file URLs right from the
store.  We have an nginx cache for /file URLs (see
‘hydra/nginx/berlin.scm’) but it’s no good if files don’t get a change
to reach it before they’re removed from the store.  We should look more
closely into that!

Thanks,
Ludo’.

Re: Adding content-addressed URLs to https://guix.gnu.org/sources.json

[Maxim Cournoyer]

Hi Simon,

Simon Tournier <zimon.toutoune@gmail.com> writes:

[...]

> $ guix repl -- build-package-metadata.scm /tmp/
> guix repl: package metadata will be written to '/tmp/'
> Backtrace:
>
> [...]
>
> In procedure %origin-patches-real: Wrong type argument: #<<computed-file> name: "ruby-sorbet-runtime-0.5.10610.20230106174520-1fa668010-checkout" gexp: #<gexp (begin (use-modules (guix build utils)) (copy-recursively (string-append #<gexp-input #<origin #<<git-reference> url: "https://github.com/sorbet/sorbet" commit: "0.5.10610.20230106174520-1fa668010" recursive?: #f> #<content-hash sha256:0f21dl06alxwn6xgdxyrkd58plmmsv04z2bcls9ld4cfzsrs5537> () 7fa67b7f6000>:out> "/gems/sorbet-" #<gexp-input "runtime":out>) #<gexp-output out>)) gnu/packages/ruby.scm:14078:5 7fa67b701c90> guile: #f options: (#:local-build? #t)>
>
>
>
> Somehow, it reveals 3 currently uncovered cases: computed-file appearing
> as,
>
>  1. ’origin’ in source field (ruby-sorbet-runtime)
>  2. ’inputs’ (racket-minimal)
>  3. ’snippet’ in origin in source field (chromium)
>
> And I think we should not try to fix these cases on
> build-package-metadata.scm side but instead investigate why they are
> needed in the first place.
>
>
> The regression in build-package-metadata is introduced by commit
> 7405e0c83f8f4eee5c4c40e809a40f2e4407a38c:
>
>   CommitDate: Tue Mar 28 22:22:24 2023 -0400
>   Author:     Maxim Cournoyer <maxim.cournoyer@gmail.com>
>
>   gnu: Add ruby-sorbet-runtime.
>
>   * gnu/packages/ruby.scm (ruby-sorbet-runtime): New variable.
>   (sorbet-version): New variable.
>   (sorbet-monorepo): New variable.
>   (make-sorbet-gem-source): New procedure.
>
>     (define (make-sorbet-gem-source gem)
>       "Return the source of GEM, a sub-directory."
>       (computed-file
>        (string-append "ruby-sorbet-" gem "-" sorbet-version "-checkout")
>        (with-imported-modules (source-module-closure '((guix build utils)))
>          #~(begin
>              (use-modules (guix build utils))
>              (copy-recursively (string-append #$sorbet-monorepo
>                                               "/gems/sorbet-" #$gem)
>                                #$output)))))
> [...]
>
>     (define-public ruby-sorbet-runtime
>       (package
>         (name "ruby-sorbet-runtime")
>         (version sorbet-version)
>         (source (make-sorbet-gem-source "runtime"))
>
>
> This pattern appears to me wrong.  It should use ’snippet’.  And if the
> point is to have a meaningful path in /gnu/store, as it is the case with
> icecat or linux or emacs-company-box, the current way is
> ’computed-origin-method’ from (guix packages).
>
> For ’ruby-sorbet-runtime’, the fix seems ’computed-origin-method’.

Per the source, IIRC, 'computed-origin-method' is also considered a hack
or workaround around the fact that a snippet can't affect the name of a
source.  But it's a well established one.  That was indeed the rationale
here (to have meaningful top level directory names matching the source),
which is worth it in my opinion.

-- 
Thanks,
Maxim

Re: Adding content-addressed URLs to https://guix.gnu.org/sources.json

[Simon Tournier]

[-- Attachment #1: Type: text/plain, Size: 5203 bytes --]

Hi Ludo, Maxim, all,

On mar., 25 avril 2023 at 14:40, Ludovic Courtès <ludovic.courtes@inria.fr> wrote:

>> Somehow, it reveals 3 currently uncovered cases: computed-file appearing
>> as,
>>
>>  1. ’origin’ in source field (ruby-sorbet-runtime)
>>  2. ’inputs’ (racket-minimal)
>>  3. ’snippet’ in origin in source field (chromium)
>
> I think #1 and #2 are okay: we can use any file-like object there, not
> just origin/package.  Of course, <origin> is meant to be the best choice
> for ‘source’, and <package> the best choice for ‘inputs’.  But I think
> it’s fine to occasionally resort to some other abstraction when these
> two are not adequate.

I agree that any file-like object is nice.  Somehow, the issue is to
“unpack“ the information of this object.  For instance,

--8<---------------cut here---------------start------------->8---
scheme@(guix-user)> (define ruby-sorbet-runtime (@@ (gnu packages ruby) ruby-sorbet-runtime))
scheme@(guix-user)> (package-source ruby-sorbet-runtime)
$1 = #<<computed-file> name: "ruby-sorbet-runtime-0.5.10610.20230106174520-1fa668010-checkout" gexp: #<gexp (begin (use-modules (guix build utils)) (copy-recursively (string-append #<gexp-input #<origin #<<git-reference> url: "https://github.com/sorbet/sorbet" commit: "0.5.10610.20230106174520-1fa668010" recursive?: #f> #<content-hash sha256:0f21dl06alxwn6xgdxyrkd58plmmsv04z2bcls9ld4cfzsrs5537> () 7fd7ad6b81e0>:out> "/gems/sorbet-" #<gexp-input "runtime":out>) #<gexp-output out>)) gnu/packages/ruby.scm:14071:5 7fd7ae734480> guile: #f options: (#:local-build? #t)>
--8<---------------cut here---------------end--------------->8---

and as far as I understand, this case cannot be handled by some generic
code.  The extraction of the “real” origin needs manual and specific
extraction because of this ’computed-file’.

For sure, ’source’ can use any file-like object because some use-cases
require that.  However, I would be tempted to use an ’origin’ as a
preferred choice – i.e., when it’s possible and try to make it
possible. ;-) Because, somehow, it “normalizes“ the source information
and eases its extraction.


On mar., 25 avril 2023 at 09:52, Maxim Cournoyer <maxim.cournoyer@gmail.com> wrote:

>> This pattern appears to me wrong.  It should use ’snippet’.  And if the
>> point is to have a meaningful path in /gnu/store, as it is the case with
>> icecat or linux or emacs-company-box, the current way is
>> ’computed-origin-method’ from (guix packages).
>>
>> For ’ruby-sorbet-runtime’, the fix seems ’computed-origin-method’.
>
> Per the source, IIRC, 'computed-origin-method' is also considered a hack
> or workaround around the fact that a snippet can't affect the name of a
> source.  But it's a well established one.  That was indeed the rationale
> here (to have meaningful top level directory names matching the source),
> which is worth it in my opinion.

I agree that ’computed-origin-method’ had been considered as a hack.  I
guess, mainly because at the time, the need seemed singular and that
’snippet’ were maybe less powerful.  For what my opinion is worth,
instead of,

--8<---------------cut here---------------start------------->8---
(define (make-sorbet-gem-source gem)
  "Return the source of GEM, a sub-directory."
  (computed-file
   (string-append "ruby-sorbet-" gem "-" sorbet-version "-checkout")
   (with-imported-modules (source-module-closure '((guix build utils)))
     #~(begin
         (use-modules (guix build utils))
         (copy-recursively (string-append #$sorbet-monorepo
                                          "/gems/sorbet-" #$gem)
                           #$output)))))

(define-public ruby-sorbet-runtime
[...]
    (source (make-sorbet-gem-source "runtime"))
--8<---------------cut here---------------end--------------->8---

I would try to “normalize” this behaviour.  Well, it’s not clear for me
if ’computed-origin-method’ is suitable as a basis for that; somehow it
would appear to me the direction for these kind of use cases.

Anyway, from my point of view, considering this very specific use-case
of ruby-sorbet-runtime, the pattern using ’computed-file’ as ’source’ –
only for having the correct store pathname – is not something that I
would introduce when it is avoidable.

And, maybe I am missing a point, but it appears to me avoidable:

--8<---------------cut here---------------start------------->8---
$ ./pre-inst-env guix build ruby-sorbet-runtime -S
/gnu/store/ni5mz1j7lbdrdqsvdm5dq1d2ack8c8q6-ruby-sorbet-runtime-0.5.10610.20230106174520-1fa668010-checkout

$ guix hash -r $(guix build ruby-sorbet-runtime -S) $(./pre-inst-env guix build ruby-sorbet-runtime -S)
0agzz44qqq5pxqzzpxwhzlbwwc7x20jrmbmaxj6q8a5bq9ydzws7
0agzz44qqq5pxqzzpxwhzlbwwc7x20jrmbmaxj6q8a5bq9ydzws7
--8<---------------cut here---------------end--------------->8---

using a ’snippet’ – see below.  Basically, ’file-name’ does the job for
the correct name and the other part is just moving content around.
Well, I guess this snippet could be simplified.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: snippet.patch --]
[-- Type: text/x-diff, Size: 1659 bytes --]

diff --git a/gnu/packages/ruby.scm b/gnu/packages/ruby.scm
index 1dcd5f76a5..f61044f76b 100644
--- a/gnu/packages/ruby.scm
+++ b/gnu/packages/ruby.scm
@@ -14078,7 +14078,35 @@ (define-public ruby-sorbet-runtime
   (package
     (name "ruby-sorbet-runtime")
     (version sorbet-version)
-    (source (make-sorbet-gem-source "runtime"))
+    (source
+     (origin
+       (inherit sorbet-monorepo)
+       (file-name (git-file-name name version))
+       (modules '((guix build utils)
+                  (ice-9 ftw)))
+       (snippet
+        #~(begin
+            (use-modules (guix build utils))
+            (let* ((name "runtime")
+                   (gems "gems")
+                   (gem (string-append gems "/sorbet-" name)))
+              (for-each
+               (lambda (x)
+                 (unless (or (string=? "." x)
+                             (string=? ".." x))
+                   (delete-file-recursively x)))
+               (scandir "."
+                        (lambda (x)
+                          (and (eq? (stat:type (stat x)) 'directory)
+                               (not (string-prefix? gems x))))))
+              (for-each
+               delete-file
+               (find-files "."
+                           (lambda (x _)
+                             (not (string-prefix? (string-append "./" gems) x)))
+                           #:directories? #f))
+              (copy-recursively gem ".")
+              (delete-file-recursively gems))))))
     (build-system ruby-build-system)
     ;; 25 out of 841 tests currently fail, seemingly due to invalid
     ;; assumptions about file names in the build environment (see:

[-- Attachment #3: Type: text/plain, Size: 16 bytes --]



Cheers,
simon

Re: Adding content-addressed URLs to https://guix.gnu.org/sources.json

[Maxim Cournoyer]

Hi Simon,

Simon Tournier <zimon.toutoune@gmail.com> writes:

> Hi Ludo, Maxim, all,
>
> On mar., 25 avril 2023 at 14:40, Ludovic Courtès <ludovic.courtes@inria.fr> wrote:
>
>>> Somehow, it reveals 3 currently uncovered cases: computed-file appearing
>>> as,
>>>
>>>  1. ’origin’ in source field (ruby-sorbet-runtime)
>>>  2. ’inputs’ (racket-minimal)
>>>  3. ’snippet’ in origin in source field (chromium)
>>
>> I think #1 and #2 are okay: we can use any file-like object there, not
>> just origin/package.  Of course, <origin> is meant to be the best choice
>> for ‘source’, and <package> the best choice for ‘inputs’.  But I think
>> it’s fine to occasionally resort to some other abstraction when these
>> two are not adequate.
>
> I agree that any file-like object is nice.  Somehow, the issue is to
> “unpack“ the information of this object.  For instance,
>
> scheme@(guix-user)> (define ruby-sorbet-runtime (@@ (gnu packages ruby) ruby-sorbet-runtime))
> scheme@(guix-user)> (package-source ruby-sorbet-runtime)
> $1 = #<<computed-file> name: "ruby-sorbet-runtime-0.5.10610.20230106174520-1fa668010-checkout" gexp: #<gexp (begin (use-modules (guix build utils)) (copy-recursively (string-append #<gexp-input #<origin #<<git-reference> url: "https://github.com/sorbet/sorbet" commit: "0.5.10610.20230106174520-1fa668010" recursive?: #f> #<content-hash sha256:0f21dl06alxwn6xgdxyrkd58plmmsv04z2bcls9ld4cfzsrs5537> () 7fd7ad6b81e0>:out> "/gems/sorbet-" #<gexp-input "runtime":out>) #<gexp-output out>)) gnu/packages/ruby.scm:14071:5 7fd7ae734480> guile: #f options: (#:local-build? #t)>
>
>
> and as far as I understand, this case cannot be handled by some generic
> code.  The extraction of the “real” origin needs manual and specific
> extraction because of this ’computed-file’.
>
> For sure, ’source’ can use any file-like object because some use-cases
> require that.  However, I would be tempted to use an ’origin’ as a
> preferred choice – i.e., when it’s possible and try to make it
> possible. ;-) Because, somehow, it “normalizes“ the source information
> and eases its extraction.

I'm not sure I follow, perhaps because I lack context about how
Disarchiver use the source field of a package.  Would you mind
explaining a bit what the problem is or pointing me to a place it was
already explained?

-- 
Thanks,
Maxim

Re: Adding content-addressed URLs to https://guix.gnu.org/sources.json

[Ludovic Courtès]

Hello,

Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

> Simon Tournier <zimon.toutoune@gmail.com> writes:

[...]

>> I agree that any file-like object is nice.  Somehow, the issue is to
>> “unpack“ the information of this object.  For instance,
>>
>> scheme@(guix-user)> (define ruby-sorbet-runtime (@@ (gnu packages ruby) ruby-sorbet-runtime))
>> scheme@(guix-user)> (package-source ruby-sorbet-runtime)
>> $1 = #<<computed-file> name: "ruby-sorbet-runtime-0.5.10610.20230106174520-1fa668010-checkout" gexp: #<gexp (begin (use-modules (guix build utils)) (copy-recursively (string-append #<gexp-input #<origin #<<git-reference> url: "https://github.com/sorbet/sorbet" commit: "0.5.10610.20230106174520-1fa668010" recursive?: #f> #<content-hash sha256:0f21dl06alxwn6xgdxyrkd58plmmsv04z2bcls9ld4cfzsrs5537> () 7fd7ad6b81e0>:out> "/gems/sorbet-" #<gexp-input "runtime":out>) #<gexp-output out>)) gnu/packages/ruby.scm:14071:5 7fd7ae734480> guile: #f options: (#:local-build? #t)>
>>
>>
>> and as far as I understand, this case cannot be handled by some generic
>> code.  The extraction of the “real” origin needs manual and specific
>> extraction because of this ’computed-file’.
>>
>> For sure, ’source’ can use any file-like object because some use-cases
>> require that.  However, I would be tempted to use an ’origin’ as a
>> preferred choice – i.e., when it’s possible and try to make it
>> possible. ;-) Because, somehow, it “normalizes“ the source information
>> and eases its extraction.

Oh, got it.

> I'm not sure I follow, perhaps because I lack context about how
> Disarchiver use the source field of a package.  Would you mind
> explaining a bit what the problem is or pointing me to a place it was
> already explained?

The problem with the idiom used for ‘ruby-sorbet-runtime’ is that the
origin is hidden inside a gexp.

Thus, the machinery that produces <https://guix.gnu.org/sources.json>
(the file that SWH fetches periodically to ingest the source code
packages refer to) will not add it.  To put it differently, we have no
guarantee that the commit above will be archived and that we’ll
eventually be able to rebuild ‘ruby-sorbet-runtime’.

So I guess as a matter of policy, we should try and find other ways to
express this so we don’t lose track of origins.

In this particular case, we could do what Simon proposed or, even
simpler, just add a phase that calls ‘chdir’ (the advantage being that
we don’t have to create copies of subsets of ‘sorbet-monorepo’).

WDYT?

Ludo’.

Re: Adding content-addressed URLs to https://guix.gnu.org/sources.json

[Maxim Cournoyer]

Hi Ludo,

Ludovic Courtès <ludovic.courtes@inria.fr> writes:

> Hello,
>
> Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:
>
>> Simon Tournier <zimon.toutoune@gmail.com> writes:
>
> [...]
>
>>> I agree that any file-like object is nice.  Somehow, the issue is to
>>> “unpack“ the information of this object.  For instance,
>>>
>>> scheme@(guix-user)> (define ruby-sorbet-runtime (@@ (gnu packages ruby) ruby-sorbet-runtime))
>>> scheme@(guix-user)> (package-source ruby-sorbet-runtime)
>>> $1 = #<<computed-file> name:
>>> "ruby-sorbet-runtime-0.5.10610.20230106174520-1fa668010-checkout"
>>> gexp: #<gexp (begin (use-modules (guix build utils))
>>> (copy-recursively (string-append #<gexp-input #<origin
>>> #<<git-reference> url: "https://github.com/sorbet/sorbet" commit:
>>> "0.5.10610.20230106174520-1fa668010" recursive?: #f> #<content-hash
>>> sha256:0f21dl06alxwn6xgdxyrkd58plmmsv04z2bcls9ld4cfzsrs5537> ()
>>> 7fd7ad6b81e0>:out> "/gems/sorbet-" #<gexp-input "runtime":out>)
>>> #<gexp-output out>)) gnu/packages/ruby.scm:14071:5 7fd7ae734480>
>>> guile: #f options: (#:local-build? #t)>
>>>
>>>
>>> and as far as I understand, this case cannot be handled by some generic
>>> code.  The extraction of the “real” origin needs manual and specific
>>> extraction because of this ’computed-file’.
>>>
>>> For sure, ’source’ can use any file-like object because some use-cases
>>> require that.  However, I would be tempted to use an ’origin’ as a
>>> preferred choice – i.e., when it’s possible and try to make it
>>> possible. ;-) Because, somehow, it “normalizes“ the source information
>>> and eases its extraction.
>
> Oh, got it.
>
>> I'm not sure I follow, perhaps because I lack context about how
>> Disarchiver use the source field of a package.  Would you mind
>> explaining a bit what the problem is or pointing me to a place it was
>> already explained?
>
> The problem with the idiom used for ‘ruby-sorbet-runtime’ is that the
> origin is hidden inside a gexp.
>
> Thus, the machinery that produces <https://guix.gnu.org/sources.json>
> (the file that SWH fetches periodically to ingest the source code
> packages refer to) will not add it.  To put it differently, we have no
> guarantee that the commit above will be archived and that we’ll
> eventually be able to rebuild ‘ruby-sorbet-runtime’.
>
> So I guess as a matter of policy, we should try and find other ways to
> express this so we don’t lose track of origins.

Thanks for explaining, it makes sense.  One way to tip package writers
in the right direction would be to add a warning when such a situation
occurs upon running 'guix lint'.

> In this particular case, we could do what Simon proposed or, even
> simpler, just add a phase that calls ‘chdir’ (the advantage being that
> we don’t have to create copies of subsets of ‘sorbet-monorepo’).

I seem to recall there was more to it than simply having a
self-contained source correctly named, such as avoiding extra phases
doing chdir and having to delete extraneous .gems that'd get picked up
erroneously and build the wrong thing/fail the build.

Simon's solution is a bit more complicated/verbose than the gexp version
currently in used taste, but it works while preserving the origin and
avoids extra steps in each package that will be using sorbet-monorepo,
so I think I'd go with it.

-- 
Thanks,
Maxim

Re: Adding content-addressed URLs to https://guix.gnu.org/sources.json

[Simon Tournier]

Hi,

On Tue, 02 May 2023 at 08:52, Maxim Cournoyer <maxim.cournoyer@gmail.com> wrote:

> Thanks for explaining, it makes sense.  One way to tip package writers
> in the right direction would be to add a warning when such a situation
> occurs upon running 'guix lint'.

Somehow improve “guix lint -c archival”. ;-)


> Simon's solution is a bit more complicated/verbose than the gexp version
> currently in used taste, but it works while preserving the origin and
> avoids extra steps in each package that will be using sorbet-monorepo,
> so I think I'd go with it.

My point was to show that ’snippet’ could produce the exact same output
but exposing the ’origin’ usable by third-party (sources.json or else).
Maybe the snippet can be simplified. :-)

Another solution would to list ’sorbet-monorepo’ as an input and then
apply a generic build phase dealing with mono repo.  Hum, I think the
’snippet’ approach is better. :_

Note ’racket-packages-origin’ from (gnu packages racket) looks similar:

--8<---------------cut here---------------start------------->8---
  (computed-file
   (string-append "racket-pkg-" name "-sources")
   (with-imported-modules `((guix build utils))
     #~(begin
         (use-modules (guix build utils))
         (mkdir-p (string-append #$output "/share/racket/pkgs"))
         (chdir (string-append #$output "/share/racket/pkgs"))
         #$@(map (match-lambda
                   ((? string? name)
                    #~(copy-recursively #$(file-append origin (string-append "/" name))
                                        #$name))
                   ((name ".")
                    #~(copy-recursively #$origin #$name))
                   ((name path)
                    #~(copy-recursively #$(file-append origin (string-append "/" path))
                                        #$name)))
                 specs)))))
--8<---------------cut here---------------end--------------->8---

and that somehow uses the ’inputs’ approach.  Well, contrary to what I
said earlier in this thread, not all the sources for building Racket are
captured by sources.json:

--8<---------------cut here---------------start------------->8---

$ wget https://guix.gnu.org/sources.json
$ guix shell jq
$ cat sources.json | jq | grep sorbet

$ cat sources.json | jq | grep 'github.com/racket'
      "git_url": "https://github.com/racket/racket",
--8<---------------cut here---------------end--------------->8---

As you see, it requires,

    https://github.com/racket/2d

as input and that’s not listed.  I will give a look later if no one
beats me. :-)


Somehow, my initial point when mentioning ’computed-origin-method’ is
that maybe there is a need.  Maybe it’s an opportunity to rehash this
“method”.  Or another that provides 1. the structure we need for
extracting and 2. the flexibility we need for special use cases.

Cheers,
simon

Re: Adding content-addressed URLs to https://guix.gnu.org/sources.json

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 903 bytes --]

Hello!

Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

> Ludovic Courtès <ludovic.courtes@inria.fr> writes:

[...]

>> So I guess as a matter of policy, we should try and find other ways to
>> express this so we don’t lose track of origins.
>
> Thanks for explaining, it makes sense.  One way to tip package writers
> in the right direction would be to add a warning when such a situation
> occurs upon running 'guix lint'.

How about the attached patch?

--8<---------------cut here---------------start------------->8---
$ ./pre-inst-env guix lint -c archival ruby-sorbet-runtime racket
gnu/packages/ruby.scm:14081:12: ruby-sorbet-runtime@0.5.10610.20230106174520-1fa668010: source is not an origin, it cannot be archived
--8<---------------cut here---------------end--------------->8---

(The version string of this package looks way too long.)

Thanks,
Ludo’.


[-- Attachment #2: the patch --]
[-- Type: text/x-patch, Size: 3708 bytes --]

From 153cb8e5782946bd0c9de22022a475ae9fef70ea Mon Sep 17 00:00:00 2001
Message-Id: <153cb8e5782946bd0c9de22022a475ae9fef70ea.1683184269.git.ludo@gnu.org>
From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= <ludo@gnu.org>
Date: Thu, 4 May 2023 09:09:03 +0200
Subject: [PATCH] lint: archival: Warn against non-origin package sources.

Suggested by Maxim Cournoyer <maxim.cournoyer@gmail.com>
and Simon Tournier <zimon.toutoune@gmail.com>.

* guix/lint.scm (check-archival): Add 'local-file?' clause.  Clarify
message in case (package-source package) is not an origin.
* tests/lint.scm ("archival: not an origin"): New test.
---
 guix/lint.scm  | 11 +++++++----
 tests/lint.scm | 11 +++++++++--
 2 files changed, 16 insertions(+), 6 deletions(-)

diff --git a/guix/lint.scm b/guix/lint.scm
index 0ed5b8dc98..72b3f4e7b1 100644
--- a/guix/lint.scm
+++ b/guix/lint.scm
@@ -1610,11 +1610,11 @@ (define (check-archival package)
   (parameterize ((%allow-request? skip-when-limit-reached))
     (catch #t
       (lambda ()
-        (match (and (origin? (package-source package))
-                    (package-source package))
+        (match (package-source package)
           (#f                                     ;no source
            '())
-          ((= origin-uri (? git-reference? reference))
+          ((and (? origin?)
+                (= origin-uri (? git-reference? reference)))
            (define url
              (git-reference-url reference))
            (define commit
@@ -1680,9 +1680,12 @@ (define (check-archival package)
                    ((? content?)
                     '())))
                '()))
+          ((? local-file?)
+           '())
           (_
            (list (make-warning package
-                               (G_ "unsupported source type")
+                               (G_ "\
+source is not an origin, it cannot be archived")
                                #:field 'source)))))
       (match-lambda*
         (('swh-error url method response)
diff --git a/tests/lint.scm b/tests/lint.scm
index ce22e2355a..b91bd053c5 100644
--- a/tests/lint.scm
+++ b/tests/lint.scm
@@ -1,7 +1,7 @@
 ;;; GNU Guix --- Functional package management for GNU
 ;;; Copyright © 2012, 2013 Cyril Roelandt <tipecaml@gmail.com>
 ;;; Copyright © 2014, 2015, 2016 Eric Bavier <bavier@member.fsf.org>
-;;; Copyright © 2014-2022 Ludovic Courtès <ludo@gnu.org>
+;;; Copyright © 2014-2023 Ludovic Courtès <ludo@gnu.org>
 ;;; Copyright © 2015, 2016 Mathieu Lirzin <mthl@gnu.org>
 ;;; Copyright © 2016 Hartmut Goebel <h.goebel@crazy-compilers.com>
 ;;; Copyright © 2017 Alex Kost <alezost@gmail.com>
@@ -43,7 +43,8 @@ (define-module (test-lint)
   #:use-module (guix lint)
   #:use-module (guix ui)
   #:use-module (guix swh)
-  #:use-module ((guix gexp) #:select (gexp local-file gexp?))
+  #:use-module ((guix gexp)
+                #:select (gexp local-file computed-file gexp?))
   #:use-module ((guix utils) #:select (call-with-temporary-directory))
   #:use-module ((guix import hackage) #:select (%hackage-url))
   #:use-module ((guix import stackage) #:select (%stackage-url))
@@ -1298,6 +1299,12 @@ (define (package-with-phase-changes changes)
   '()
   (check-formatting (dummy-package "x")))
 
+(test-assert "archival: not an origin"
+  (warning-contains? "not an origin"
+                     (check-archival
+                      (dummy-package
+                       "x" (source (computed-file "x-src" #t))))))
+
 (test-assert "archival: missing content"
   (let* ((origin   (origin
                      (method url-fetch)

base-commit: 44d70440944aa47e5f37493346e560f38907d777
-- 
2.39.2

Re: Adding content-addressed URLs to https://guix.gnu.org/sources.json

[Ludovic Courtès]

Hi,

Ludovic Courtès <ludovic.courtes@inria.fr> skribis:

> How about the attached patch?
>
> $ ./pre-inst-env guix lint -c archival ruby-sorbet-runtime racket
> gnu/packages/ruby.scm:14081:12: ruby-sorbet-runtime@0.5.10610.20230106174520-1fa668010: source is not an origin, it cannot be archived
>
> (The version string of this package looks way too long.)
>
> Thanks,
> Ludo’.
>
> From 153cb8e5782946bd0c9de22022a475ae9fef70ea Mon Sep 17 00:00:00 2001
> Message-Id: <153cb8e5782946bd0c9de22022a475ae9fef70ea.1683184269.git.ludo@gnu.org>
> From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= <ludo@gnu.org>
> Date: Thu, 4 May 2023 09:09:03 +0200
> Subject: [PATCH] lint: archival: Warn against non-origin package sources.
>
> Suggested by Maxim Cournoyer <maxim.cournoyer@gmail.com>
> and Simon Tournier <zimon.toutoune@gmail.com>.
>
> * guix/lint.scm (check-archival): Add 'local-file?' clause.  Clarify
> message in case (package-source package) is not an origin.
> * tests/lint.scm ("archival: not an origin"): New test.

Pushed as 71fd35c1d55988c413a37c7d15006b4d38d7dde7.  Let me know if
anything’s amiss!

Ludo’.

Re: Adding content-addressed URLs to https://guix.gnu.org/sources.json

[Maxim Cournoyer]

Hi,

Ludovic Courtès <ludo@gnu.org> writes:

> Hi,
>
> Ludovic Courtès <ludovic.courtes@inria.fr> skribis:
>
>> How about the attached patch?
>>
>> $ ./pre-inst-env guix lint -c archival ruby-sorbet-runtime racket
>> gnu/packages/ruby.scm:14081:12:
>> ruby-sorbet-runtime@0.5.10610.20230106174520-1fa668010: source is
>> not an origin, it cannot be archived
>>
>> (The version string of this package looks way too long.)
>>
>> Thanks,
>> Ludo’.
>>
>> From 153cb8e5782946bd0c9de22022a475ae9fef70ea Mon Sep 17 00:00:00 2001
>> Message-Id: <153cb8e5782946bd0c9de22022a475ae9fef70ea.1683184269.git.ludo@gnu.org>
>> From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= <ludo@gnu.org>
>> Date: Thu, 4 May 2023 09:09:03 +0200
>> Subject: [PATCH] lint: archival: Warn against non-origin package sources.
>>
>> Suggested by Maxim Cournoyer <maxim.cournoyer@gmail.com>
>> and Simon Tournier <zimon.toutoune@gmail.com>.
>>
>> * guix/lint.scm (check-archival): Add 'local-file?' clause.  Clarify
>> message in case (package-source package) is not an origin.
>> * tests/lint.scm ("archival: not an origin"): New test.
>
> Pushed as 71fd35c1d55988c413a37c7d15006b4d38d7dde7.  Let me know if
> anything’s amiss!

Neat, thanks for tackling it.

-- 
Thanks,
Maxim