Preservation of Guix (PoG) report 2023-03-13

Preservation of Guix (PoG) report 2023-03-13

[Timothy Sample]

[-- Attachment #1: Type: text/plain, Size: 5115 bytes --]

Hi Guix,

It’s been a while!  :)

Allow me to present to you a long-overdue update to the Preservation of
Guix (PoG) report: <https://ngyro.com/pog-reports/2023-03-13/>.  🎉

Note that you can link to the most recent version of the report using
<https://ngyro.com/pog-reports/latest/>.

What is this?  Well, I added a description to the report itself, but
here’s a brief teaser.  The PoG report shows what we know about the
archival status of the approximately 54K sources (and counting) Guix has
linked to since around the time of the 1.0 release.

For this edition, I took a bit of time to fix the contrast and colours
to be a bit more accessible.  They’re about half as garish as they used
to be, too.

Over the whole set, 77.1% are known to be safely tucked away in the
Software Heritage archive.  But it’s actually much better than that.  If
we only look at the most recent sampled commit (from Sunday the 5th),
that number becomes 87.4%, which is starting to look pretty good!

I have a few more notes on the report, but I want to put this near the
top of the message so that people will see it.  :)  I wrote a script
(see attached) that uses the PoG database to find missing sources on a
packge-by-package basis.  That is, you can run

    guix repl specification-to-swhids.scm pog.db bash

and it will print a table of all of the transitive sources needed to
build Bash, along with their preservation status.  Here’s a (heavily
edited and snipped to fit an email message) sample of its output:

[... many “stored” inputs]
sha256 0r5p. swh:1:dir:02f7. stored  /gnu/store/.-gmp-6.0.0a.tar.xz
sha256 0c3k. swh:1:dir:6027. stored  /gnu/store/.-mescc-....tar.xz
sha256 1r1z. swh:1:dir:6087. stored  /gnu/store/.-bash-2.05b.tar.gz
sha256 14l0. unknown         unknown /gnu/store/.-gcc-4.9.4.tar.bz2
sha256 0m2y. unknown         unknown /gnu/store/.-ed-1.17.tar.lz
[... more “unknown” inputs]

(I had to pipe the output to “sort -k 4” to have it sorted by status.)

The first two columns are the Guix hash.  The next two columns are the
SWHID (if known) and whether SWH has it (if known).  That last column is
the store filename (which is nice because it usually tells you what it
is we are looking at).  In this sample, you can see that GMP, MesCC
Tools, and Bash are all safe.  However, we don’t know about GCC 4 and
ed.  This is kinda like an automated version of Simon’s recent
investigation [1].  The “unknown” two are due to Disarchive’s lack of
support for those compression formats.  I just wrote this script today
(mind the rough edges), and I’ve learned a lot from trying it on a few
packages.  It’s a little like a terrifying robotic TODO list, since it
shows a lot of problems, but it’s also exiting because solving all the
problems for the Guix package, say, would be a massive leap forward.
Here’s a rough road map for that based on a glance at the script’s
output:

    • Subversion support (for TeX-based documentation stuff, I guess)
    • bzip2 support for Disarchive (there are 45 bzip2 tarballs)
    • ZIP support for Disarchive (for the 8 ZIP files)
    • lzip support for Disarchive (or a workaround for ed)
    • Fix some issues (gettext is .tar.gz, but something went wrong)
    • Do something with the static bootstrap binaries

[1] https://lists.gnu.org/archive/html/guix-devel/2023-02/msg00398.html

If you want to try it out for yourself, you’ll need to download the
database <https://ngyro.com/pog-reports/2023-03-13/pog.db>.  Heads up:
it’s just over 200M, and my server can be pretty slow.

One other stray thought: the script should work with the time machine,
so you can check on packages from the past.  I didn’t test it, but I bet
it’s fine.

Okay.  Here are the rest of my notes about the report itself.

One thing that jumps out at me is 189 Git sources that SWH does not
have.  Usually they have basically all of the non-recursive Git sources.
It’s something to look into.

I also took a quick peek at the 1.9K “unknown” tar-gz sources.  About
39% percent of them are old Rust crates.  It’s a known problem with
Disarchive.  However, 42% of them are old Bioconductor packages.  They
seem to be lost.  It looks like Bioconductor now stores multiple package
versions per Bioconductor version [2], but before version 3.15 that was
not the case.  As an example, take “ggcyto” from Bioconductor 3.10 [3].
We packaged version 1.14.0, and then at some point Bioconductor 3.10
switched to version 1.14.1.  We packaged that, too, but now 1.14.0 is
gone.  I know it’s been discussed before, but I can’t remember what the
conclusion was.  Are these just gone forever?  I’m doing another pass
through all of them and recovering a few from the bordeaux substitute
server, but only a handful.

[2] https://bioconductor.org/packages/3.15/bioc/src/contrib/Archive/DiffBind/
[3] https://bioconductor.org/packages/3.10/bioc/html/ggcyto.html

That’s all for now.  Enjoy the update and the script!


-- Tim


[-- Attachment #2: specification-to-swhids.scm --]
[-- Type: text/plain, Size: 6548 bytes --]

;;; specification-to-swhids.scm
;;; Copyright © 2023 Timothy Sample <samplet@ngyro.com>
;;;
;;; This program is free software: you can redistribute it and/or modify
;;; it under the terms of the GNU General Public License as published by
;;; the Free Software Foundation, either version 3 of the License, or (at
;;; your option) any later version.
;;;
;;; This program is distributed in the hope that it will be useful, but
;;; WITHOUT ANY WARRANTY; without even the implied warranty of
;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
;;; General Public License for more details.
;;;
;;; You should have received a copy of the GNU General Public License
;;; along with this program. If not, see <https://www.gnu.org/licenses/>.

(use-modules (gnu packages)
             (guix base32)
             (guix derivations)
             (guix gexp)
             (guix monads)
             (guix store)
             (ice-9 format)
             (ice-9 getopt-long)
             (ice-9 match)
             (sqlite3)
             (srfi srfi-9 gnu))

\f
;;; Database stuff

(define (call-with-sqlite-db filename proc)
  "Open the SQLite database at FILENAME and pass the resulting
connection to PROC.  The connection will only be open during the dynamic
extent of PROC.  If that dynamic extent is re-entered (using a
continuation, say), the database connection will be re-established."
  (let ((db #f))
    (dynamic-wind
      (lambda ()
        (set! db (sqlite-open filename)))
      (lambda ()
        (proc db))
      (lambda ()
        (sqlite-close db)
        (set! db #f)))))

(define (database-lookup db query params converter)
  "Using the SQLite database connection DB, run QUERY with PARAMS, and
map CONVERTER over the resulting rows."
  (let* ((stmt (sqlite-prepare db query))
         (_ (unless (null? params)
              (apply sqlite-bind-arguments stmt params)))
         (result (sqlite-fold (lambda (x acc)
                                (cons (converter x) acc))
                              '()
                              stmt)))
    (sqlite-finalize stmt)
    result))

(define (lookup-swh-status db algorithm hash)
  "Using the SQLite database connection DB, lookup the SWHID of the
fixed-output derivation with the ALGORITHM-computed checksum HASH.
Here, both ALGORITHM and HASH are strings, the latter being the Nix
base-32 representation of the hash value."
  (define query "\
SELECT swhid, is_in_swh
FROM fods
WHERE algorithm = ?
    AND hash = ?")
  (define (converter row) row)
  (and=> (database-lookup db query (list algorithm hash) converter) car))

\f
;;; Guix stuff

(define (derivation-transitive-fixed-output-inputs drv)
  "Compute the list of all fixed-output derivations in the transitive
inputs of the derivation DRV."
  (define seen (make-hash-table))
  (define fod-hashes (make-hash-table))

  (define (seen? drv)
    (hashq-ref seen drv))

  (let loop ((queue (list drv)))
    (match queue
      (() (hash-map->list cons fod-hashes))
      ((drv . rest)
       (hashq-set! seen drv #t)
       (when (fixed-output-derivation? drv)
         (let* ((out (assoc-ref (derivation-outputs drv) "out"))
                (algo (derivation-output-hash-algo out))
                (hash (derivation-output-hash out))
                (filename (derivation-output-path out)))
           (hash-set! fod-hashes (cons algo hash) filename)))
       (loop (append (filter (negate seen?)
                             (map derivation-input-derivation
                                  (derivation-inputs drv)))
                     rest))))))

(define (lookup-object-hashes obj)
  "Get the list of Guix hashes needed for the lowerable object OBJ."
  (let ((drv (run-with-store (open-connection)
               (lower-object obj))))
    (derivation-transitive-fixed-output-inputs drv)))

\f
;;; Glue

(define-immutable-record-type <source>
  (make-source algorithm hash filename swhid in-swh?)
  source?
  (algorithm source-algorithm)
  (hash source-hash)
  (filename source-filename)
  (swhid source-swhid)
  (in-swh? source-in-swh?))

(define (guix-hash->source db hash-obj)
  "Using the SQLite database connection DB, convert HASH-OBJ to a source
record.  HASH-OBJ should be a result from 'lookup-object-hashes'."
  (match-let* ((((algorithm . hash) . filename) hash-obj)
               (algorithm (symbol->string algorithm))
               (hash (bytevector->nix-base32-string hash))
               (#(swhid in-swh?) (lookup-swh-status db algorithm hash)))
    (make-source algorithm hash filename swhid in-swh?)))

(define (object-sources db obj)
  "Using the SQLite database connection DB, get the list of source
records for the lowerable object OBJ."
  (let ((hashes (lookup-object-hashes obj)))
    (map (lambda (hash) (guix-hash->source db hash)) hashes)))

\f
;; Shell interface

(define (print-source src)
  (format #t "~a\t~a\t~50a\t~a\t~a~%"
          (source-algorithm src)
          (source-hash src)
          (or (source-swhid src) "unknown")
          (cond
           ((source-in-swh? src) "stored")
           ((source-swhid src) "missing")
           (else "unknown"))
          (source-filename src)))

(define version "2023-03-13-0")

(define version-message (string-append "\
specification-to-swhids.scm " version "
"))

(define help-message "\
Usage: guix repl specification-to-swhids.scm DB-FILENAME SPECIFICATION

Print a table of Guix hashes, SWHIDs, and store filenames for the Guix
package SPECIFICATION using the Preservation of Guix database at
DB-FILENAME.  See <https://ngyro.com/pog-reports/latest>.
")

(define options-grammar
  `((help (single-char #\h))
    (version (single-char #\V))))

(define (main args)
  (let ((options (getopt-long args options-grammar)))
    (when (option-ref options 'help #f)
      (display help-message)
      (exit EXIT_SUCCESS))
    (when (option-ref options 'version #f)
      (display version-message)
      (exit EXIT_SUCCESS))
    (match (option-ref options '() #f)
      ((db-filename specification)
       (for-each print-source
                 (let ((obj (specification->package specification)))
                   (call-with-sqlite-db db-filename
                     (lambda (db)
                       (object-sources db obj)))))
       (exit EXIT_SUCCESS))
      (_
       (display help-message)
       (exit EXIT_FAILURE)))))

(main (command-line))

Re: Preservation of Guix (PoG) report 2023-03-13

[Simon Tournier]

Hi,

On Mon, 13 Mar 2023 at 19:37, Timothy Sample <samplet@ngyro.com> wrote:

> Note that you can link to the most recent version of the report using
> <https://ngyro.com/pog-reports/latest/>.

Awesome! \o/

Well, I do not remember if you consider also the ’origin’
(fixed-outputs) as ’inputs’ or ’patches’.  Do you?

Basically, ’package-direct-sources’ from (guix packages).

For instance, see the package ’ntp’,

--8<---------------cut here---------------start------------->8---
(source
     (origin
       (method url-fetch)
       (uri (list (string-append
                   "https://www.eecis.udel.edu/~ntp/ntp_spool/ntp4/ntp-"
                   (version-major+minor version)
[...]
       (sha256
        (base32 "06cwhimm71safmwvp6nhxp6hvxsg62whnbgbgiflsqb8mgg40n7n"))
       ;; Add an upstream patch to fix build with GCC 10.  Taken from
       ;; <https://bugs.ntp.org/show_bug.cgi?id=3688>.
       (patches (list (origin
                        (method url-fetch)
                        (uri "https://bugs.ntp.org/attachment.cgi?id=1760\
&action=diff&context=patch&collapsed=&headers=1&format=raw")
                        (file-name "ntp-gcc-compat.patch")
                        (sha256
                         (base32
                          "13d28sg45rflc7kqiv30asrhna8n69wlpwx16l65rravgpvp90h2")))
--8<---------------cut here---------------end--------------->8---

or see the package ’tensorflow’,

--8<---------------cut here---------------start------------->8---
    (native-inputs
     `(("pkg-config" ,pkg-config)
[...]
       ("boringssl-src"
        ,(let ((commit "ee7aa02")
               (revision "1"))
           (origin
             (method git-fetch)
             (uri (git-reference
                   (url "https://boringssl.googlesource.com/boringssl")
                   (commit commit)))
             (file-name (string-append "boringssl-0-" revision
                                       (string-take commit 7)
                                       "-checkout"))
             (sha256
              (base32
               "1jf693q0nw0adsic6cgmbdx6g7wr4rj4vxa8j1hpn792fqhd8wgw")))))
--8<---------------cut here---------------end--------------->8---


> Over the whole set, 77.1% are known to be safely tucked away in the
> Software Heritage archive.  But it’s actually much better than that.  If
> we only look at the most recent sampled commit (from Sunday the 5th),
> that number becomes 87.4%, which is starting to look pretty good!

Just to be point the new nixguix loader [1] is still in SWH staging and
not yet deployed, IIRC.  It will not change much the coverage on our
side but it should be fix some corner-cases.

1: <https://gitlab.softwareheritage.org/swh/meta/-/issues/4662>


>      This is kinda like an automated version of Simon’s recent
> investigation.

Neat!  Note that I also wanted to check the SWH capacity for cooking,
not only checking the end points.  For instance, it allowed to discover
mismatch due to uncovered CR/LF normalization; now fixed with:
58f20fa8181bdcd4269671e1d3cef1268947af3a.


> Here’s a rough road map for that based on a glance at the script’s
> output:
>
>     • Subversion support (for TeX-based documentation stuff, I guess)

For the interested reader, details for helping in the implementation:

    https://issues.guix.gnu.org/issue/43442#9
    https://issues.guix.gnu.org/issue/43442#11

However, it would ease all the dance if SWH would consider to store and
expose NAR hashes on their side.  As discussed here:

    https://gitlab.softwareheritage.org/swh/meta/-/issues/4538


>              However, 42% of them are old Bioconductor packages.  They
> seem to be lost.  It looks like Bioconductor now stores multiple package
> versions per Bioconductor version [2], but before version 3.15 that was
> not the case.  As an example, take “ggcyto” from Bioconductor 3.10 [3].
> We packaged version 1.14.0, and then at some point Bioconductor 3.10
> switched to version 1.14.1.  We packaged that, too, but now 1.14.0 is
> gone.

Well, I have not investigated much because it is between December 2019
and March 2020 thus “guix time-machine” is not smooth for this old time.

First question, does we have the source tarball in Berlin or Bordeaux or
somewhere else?  If yes, there is a hope. :-) Else, it is probably gone
forever.

The hope is: https://git.bioconductor.org/packages/ggcyto

If we have the tarball with the correct checksum from commit
f5f440312d848e12463f0c6f7510a86b623a9e27

--8<---------------cut here---------------start------------->8---
+    (version "1.14.0")
+    (source
+     (origin
+       (method url-fetch)
+       (uri (bioconductor-uri "ggcyto" version))
+       (sha256
+        (base32
+         "165qszvy5z176h1l3dnjb5dcm279b6bjl5n5gzz8wfn4xpn8anc8"))))
--8<---------------cut here---------------end--------------->8---

then we can disassemble it and then using the Git repository, we can try
to assemble the content from SWH and the meta from Disarchive DB.

For sure, it is again another example why we should augment by intrinsic
identifiers the Guix way for fetching.  See:

    https://lists.gnu.org/archive/html/guix-devel/2023-03/msg00025.html


>        I know it’s been discussed before, but I can’t remember what the
> conclusion was.  Are these just gone forever?

Discussed here:

    https://issues.guix.gnu.org/issue/39885
    https://issues.guix.gnu.org/issue/54787    



Cheers,
simon

Re: Preservation of Guix (PoG) report 2023-03-13

[Ludovic Courtès]

Hi Timothy,

Timothy Sample <samplet@ngyro.com> skribis:

> Allow me to present to you a long-overdue update to the Preservation of
> Guix (PoG) report: <https://ngyro.com/pog-reports/2023-03-13/>.  🎉

Yay, thank you!!

Do you think this could be turned into a Guix System service, with an
eye towards making it run on project infrastructure?

> Over the whole set, 77.1% are known to be safely tucked away in the
> Software Heritage archive.  But it’s actually much better than that.  If
> we only look at the most recent sampled commit (from Sunday the 5th),
> that number becomes 87.4%, which is starting to look pretty good!

Truly impressive!

> I have a few more notes on the report, but I want to put this near the
> top of the message so that people will see it.  :)  I wrote a script
> (see attached) that uses the PoG database to find missing sources on a
> packge-by-package basis.  That is, you can run
>
>     guix repl specification-to-swhids.scm pog.db bash

Neat!

>     • Subversion support (for TeX-based documentation stuff, I guess)
>     • bzip2 support for Disarchive (there are 45 bzip2 tarballs)
>     • ZIP support for Disarchive (for the 8 ZIP files)
>     • lzip support for Disarchive (or a workaround for ed)
>     • Fix some issues (gettext is .tar.gz, but something went wrong)
>     • Do something with the static bootstrap binaries

Subversion support should probably be high-priority because TeX Live is
deep down in the dependency graph so if we ever lose it, we’re doomed.

Thanks for the update!

Ludo’.

Re: Preservation of Guix (PoG) report 2023-03-13

[Timothy Sample]

Hey,

Simon Tournier <zimon.toutoune@gmail.com> writes:

> Well, I do not remember if you consider also the ’origin’
> (fixed-outputs) as ’inputs’ or ’patches’.  Do you?

I’m quite confident I’m getting everything.  I’ll describe my approach,
because I’m happy with it.  :)

The Guix package graph exists twice, essentially.  There’s the
high-level representation made up of packages, origins, gexps, etc.
Then, there is the low-level representation which is just derivations.
The high-level representation has nice metadata and makes sense to
humans, while the low-level representation is easy to traverse.

AFAICT, there’s no generic way to traverse the high-level
representation.  Every lowerable object has complete control over how it
references other lowerable objects, and is not obliged provide any means
of listing those references.  That is, there’s no ‘lowerable-inputs’
procedure or anything like that.  (We have ‘bag-node-edges’ in ‘(guix
scripts graph)’, but it doesn’t cover everything.)

What I do for the report is traverse (as best I can) the high-level
representation and construct a map from derivations to origin objects.
Then, I traverse the low-level representation to find all the
fixed-output derivations.  Finally, I use the map to look up origin
objects for each fixed-output derivation.  If I miss an origin object,
the fixed-output derivation still gets recorded.  It will show up in the
report as “unknown” until I investigate why it’s missing and correct it.

There’s currently 56 (out of 54K) fixed-output derivations that are
missing metadata in my database.  A fair few of them have to do with
Telegram, Thunderbird, and UBlock Origin.  All it means is that those
packages have sneaky ways of referencing origins that my code can’t
handle.  It’s harmless and easy to fix as time permits.

>> Over the whole set, 77.1% are known to be safely tucked away in the
>> Software Heritage archive.  But it’s actually much better than that.  If
>> we only look at the most recent sampled commit (from Sunday the 5th),
>> that number becomes 87.4%, which is starting to look pretty good!
>
> Just to be point the new nixguix loader [1] is still in SWH staging and
> not yet deployed, IIRC.  It will not change much the coverage on our
> side but it should be fix some corner-cases.
>
> 1: <https://gitlab.softwareheritage.org/swh/meta/-/issues/4662>

Good to know!

>>      This is kinda like an automated version of Simon’s recent
>> investigation.
>
> Neat!  Note that I also wanted to check the SWH capacity for cooking,
> not only checking the end points.  For instance, it allowed to discover
> mismatch due to uncovered CR/LF normalization; now fixed with:
> 58f20fa8181bdcd4269671e1d3cef1268947af3a.

Maybe we need a “chaos monkey mode” for Guix.  It could randomly select
packages to build, randomly pick source code fallback methods, and also
test reproducibility (like “--check”).  You could have a blocklist for
browsers, etc., but otherwise it could pick the odd package to test
thoroughly.  Those of us with the time and inclination could crank up
that knob and get interesting feedback about reproducibility at the cost
of doing a few package builds here and there.

>> Here’s a rough road map for that based on a glance at the script’s
>> output:
>>
>>     • Subversion support (for TeX-based documentation stuff, I guess)
>
> For the interested reader, details for helping in the implementation:
>
>     https://issues.guix.gnu.org/issue/43442#9
>     https://issues.guix.gnu.org/issue/43442#11

Fantastic.  That looks very promising!

> However, it would ease all the dance if SWH would consider to store and
> expose NAR hashes on their side.  As discussed here:
>
>     https://gitlab.softwareheritage.org/swh/meta/-/issues/4538

This would be nice, yes.

>>              However, 42% of them are old Bioconductor packages.  They
>> seem to be lost.  It looks like Bioconductor now stores multiple package
>> versions per Bioconductor version [2], but before version 3.15 that was
>> not the case.  As an example, take “ggcyto” from Bioconductor 3.10 [3].
>> We packaged version 1.14.0, and then at some point Bioconductor 3.10
>> switched to version 1.14.1.  We packaged that, too, but now 1.14.0 is
>> gone.
>
> Well, I have not investigated much because it is between December 2019
> and March 2020 thus “guix time-machine” is not smooth for this old time.
>
> First question, does we have the source tarball in Berlin or Bordeaux or
> somewhere else?  If yes, there is a hope. :-) Else, it is probably gone
> forever.

Like I wrote, I picked up a handful from Bordeaux, but not much.

> The hope is: https://git.bioconductor.org/packages/ggcyto
>
> If we have the tarball with the correct checksum from commit
> f5f440312d848e12463f0c6f7510a86b623a9e27
>
> +    (version "1.14.0")
> +    (source
> +     (origin
> +       (method url-fetch)
> +       (uri (bioconductor-uri "ggcyto" version))
> +       (sha256
> +        (base32
> +         "165qszvy5z176h1l3dnjb5dcm279b6bjl5n5gzz8wfn4xpn8anc8"))))
>
> then we can disassemble it and then using the Git repository, we can try
> to assemble the content from SWH and the meta from Disarchive DB.

I played around with this approach a bit, but it’s extremely tedious,
and I’m not hopeful it will work.  Even if it does, it will be hard to
automate.  I never fully tested the idea, just decided the effort was
too high for such a low probability of success.  I’m putting these in
the “low priority” bin for now.


-- Tim

Re: Preservation of Guix (PoG) report 2023-03-13

[Timothy Sample]

Hi Ludo,

Ludovic Courtès <ludo@gnu.org> writes:

> Do you think this could be turned into a Guix System service, with an
> eye towards making it run on project infrastructure?

I need to revisit what you did with Disarchive and Cuirass.  The process
for the PoG report is very similar.  I can’t jump into it right away,
but I agree that it would be better and I hope to work on it eventually.

> Subversion support should probably be high-priority because TeX Live is
> deep down in the dependency graph so if we ever lose it, we’re doomed.

I will start processing Subversion references on my end.  That way, we
will at least know how many of them are in the SWH archive.


-- Tim

Re: Preservation of Guix (PoG) report 2023-03-13

[Ludovic Courtès]

Hello!

Timothy Sample <samplet@ngyro.com> skribis:

[...]

>>>     • Subversion support (for TeX-based documentation stuff, I guess)
>>
>> For the interested reader, details for helping in the implementation:
>>
>>     https://issues.guix.gnu.org/issue/43442#9
>>     https://issues.guix.gnu.org/issue/43442#11
>
> Fantastic.  That looks very promising!
>
>> However, it would ease all the dance if SWH would consider to store and
>> expose NAR hashes on their side.  As discussed here:
>>
>>     https://gitlab.softwareheritage.org/swh/meta/-/issues/4538
>
> This would be nice, yes.

Good news is progress is happening on these fronts…

Ludo’.