ImageMagick from 2020?

ImageMagick from 2020?

[zimoun]

Hi,

Trying to get 'hs-to-coq' working, I am travelling back in 2020 with
commit: 1ac4004502.  Then, I hit ImageMagick mismtach.

--8<---------------cut here---------------start------------->8---
$ guix time-machine --commit=1ac4004502 -- build imagemagick
guile: warning: failed to install locale
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 100.0%
building /gnu/store/ngslsnxgy9syjs97cb7yv4744hllr6vx-ImageMagick-6.9.10-68.tar.xz.drv...

Starting download of /gnu/store/5qw3rrbk7679kd11147ks6jjcffmvv8m-ImageMagick-6.9.10-68.tar.xz
From https://sunsite.icm.edu.pl/packages/ImageMagick/ImageMagick-6.9.10-68.tar.xz...
download failed "https://sunsite.icm.edu.pl/packages/ImageMagick/ImageMagick-6.9.10-68.tar.xz" 404 "Not Found"

Starting download of /gnu/store/5qw3rrbk7679kd11147ks6jjcffmvv8m-ImageMagick-6.9.10-68.tar.xz
From http://mirrors-usa.go-parts.com/mirrors/ImageMagick/ImageMagick-6.9.10-68.tar.xz...
following redirection to `https://go-parts.com/mirrors/ImageMagick/ImageMagick-6.9.10-68.tar.xz'...
downloading from http://mirrors-usa.go-parts.com/mirrors/ImageMagick/ImageMagick-6.9.10-68.tar.xz...
 ImageMagick-6.9.10-68.tar.xz                                                       70.7MiB/s 00:00 | 48KiB transferred
sha256 hash mismatch for /gnu/store/5qw3rrbk7679kd11147ks6jjcffmvv8m-ImageMagick-6.9.10-68.tar.xz:
  expected hash: 1jxjxhnvznpbdigry2cgxx94cx6k6y3rs40a464n5yln29s1qlz1
  actual hash:   1j2r0l2j8yglhnbhi7pj14fv38zaxvxs6v6wx462fn6x1cq4xr00
hash mismatch for store item '/gnu/store/5qw3rrbk7679kd11147ks6jjcffmvv8m-ImageMagick-6.9.10-68.tar.xz'
build of /gnu/store/ngslsnxgy9syjs97cb7yv4744hllr6vx-ImageMagick-6.9.10-68.tar.xz.drv failed
View build log at '/var/log/guix/drvs/ng/slsnxgy9syjs97cb7yv4744hllr6vx-ImageMagick-6.9.10-68.tar.xz.drv.bz2'.
cannot build derivation `/gnu/store/xg2zdql0gpyavx2s4kv408gdvmpn18c5-imagemagick-6.9.10-68.drv': 1 dependencies couldn't be built
guix build: error: build of `/gnu/store/xg2zdql0gpyavx2s4kv408gdvmpn18c5-imagemagick-6.9.10-68.drv' failed
--8<---------------cut here---------------end--------------->8---


We probably do not have this item.  Since it is 'xz', it is somehow
expected because it is works in progress.

Can someone confirm we do not effectively have this item in our infra?

(It would mean that 1818 dependent packages thus not "lost".)

Cheers,
simon

Re: ImageMagick from 2020?

[Timothy Sample]

Hi zimoun,

zimoun <zimon.toutoune@gmail.com> writes:

> We probably do not have this item.  Since it is 'xz', it is somehow
> expected because it is works in progress.

XZ support is working in Disarchive now.  I’m hosting quite a few XZ
specifications at <https://disarchive.ngyro.com>.

> Can someone confirm we do not effectively have this item in our infra?

According to my logs, I tried to get it a month ago and it failed then,
too.  If you track it down, let me know and I’ll ingest it.


-- Tim

Re: ImageMagick from 2020?

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 1487 bytes --]

Timothy Sample schreef op ma 10-01-2022 om 13:01 [-0500]:
> Hi zimoun,
> 
> zimoun <zimon.toutoune@gmail.com> writes:
> 
> > We probably do not have this item.  Since it is 'xz', it is somehow
> > expected because it is works in progress.
> 
> XZ support is working in Disarchive now.  I’m hosting quite a few XZ
> specifications at <https://disarchive.ngyro.com>.
> 
> > Can someone confirm we do not effectively have this item in our infra?
> 
> According to my logs, I tried to get it a month ago and it failed then,
> too.  If you track it down, let me know and I’ll ingest it.

From repology, I found that CentOS c7 has ImageMagick 6.9.10-68:

https://git.centos.org/rpms/ImageMagick/blob/c7/f/SPECS/ImageMagick.spec

CentOS keeps its copy of the source code at

https://git.centos.org/sources/ImageMagick/c7/

there are two files there, seems like they are named after some kind of
hash?

To determine the hash, you can look at
https://git.centos.org/rpms/ImageMagick/blob/c7/f/.ImageMagick.metadata

It's this file you need:

https://git.centos.org/sources/ImageMagick/c7/ed27964d872abc81dc2388d333874766058a9b5d

The hash checks out:

guix download https://git.centos.org/sources/ImageMagick/c7/ed27964d872abc81dc2388d333874766058a9b5d
[downloading]
1jxjxhnvznpbdigry2cgxx94cx6k6y3rs40a464n5yln29s1qlz1

It might be interesting to teach (guix build download) to fallback to
the archives of CentOS, Debian, ...

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: ImageMagick from 2020?

[zimoun]

Hi Maxime,

Thanks!

On Mon, 10 Jan 2022 at 19:36, Maxime Devos <maximedevos@telenet.be> wrote:

> guix download https://git.centos.org/sources/ImageMagick/c7/ed27964d872abc81dc2388d333874766058a9b5d
> [downloading]
> 1jxjxhnvznpbdigry2cgxx94cx6k6y3rs40a464n5yln29s1qlz1

Then after,

    cp /gnu/store/qv9adcfsvhm79sinv36dznvzzvw0db5y-ed27964d872abc81dc2388d333874766058a9b5d
/tmp/ImageMagick-6.9.10-68.tar.xz
    guix download file:///tmp/ImageMagick-6.9.10-68.tar.xz

and life can be back as usual. :-)

> It might be interesting to teach (guix build download) to fallback to
> the archives of CentOS, Debian, ...

I think it could be a nice idea.


Cheers,
simon

Re: ImageMagick from 2020?

[Ludovic Courtès]

Hi,

Maxime Devos <maximedevos@telenet.be> skribis:

> From repology, I found that CentOS c7 has ImageMagick 6.9.10-68:
>
> https://git.centos.org/rpms/ImageMagick/blob/c7/f/SPECS/ImageMagick.spec
>
> CentOS keeps its copy of the source code at
>
> https://git.centos.org/sources/ImageMagick/c7/
>
> there are two files there, seems like they are named after some kind of
> hash?

It’s now on ci.guix too, though not GC-protected.

Timothy, will your machinery be able to pick it up?

BTW, related to that, could we rsync https://disarchive.ngyro.com to
https://disarchive.guix.gnu.org?  The latter only contains data starting
from ‘master’ revisions ca. November 2021.

Thanks,
Ludo’.

Re: ImageMagick from 2020?

[zimoun]

Hi Ludo,

On Tue, 18 Jan 2022 at 15:39, Ludovic Courtès <ludo@gnu.org> wrote:

> It’s now on ci.guix too, though not GC-protected.
>
> Timothy, will your machinery be able to pick it up?

Hum, the issue is the ingestion by SWH, no?  The file 'sources.json'
only contains the last sources of the last revision.  And even, the
file 'sources.json' only contains the upstream URL, not the substitute
ones.


Cheers,
simon

Re: ImageMagick from 2020?

[Ludovic Courtès]

Hi,

zimoun <zimon.toutoune@gmail.com> skribis:

> On Tue, 18 Jan 2022 at 15:39, Ludovic Courtès <ludo@gnu.org> wrote:
>
>> It’s now on ci.guix too, though not GC-protected.
>>
>> Timothy, will your machinery be able to pick it up?
>
> Hum, the issue is the ingestion by SWH, no?  The file 'sources.json'
> only contains the last sources of the last revision.  And even, the
> file 'sources.json' only contains the upstream URL, not the substitute
> ones.

Oh right, so we’ll need to feed them historical ‘sources.json’ files
eventually, I think Timothy was planning to do that eventually.

Ludo’.

Re: ImageMagick from 2020?

[zimoun]

Hi,

On Wed, 19 Jan 2022 at 11:36, Ludovic Courtès <ludo@gnu.org> wrote:

> > Hum, the issue is the ingestion by SWH, no?  The file 'sources.json'
> > only contains the last sources of the last revision.  And even, the
> > file 'sources.json' only contains the upstream URL, not the substitute
> > ones.
>
> Oh right, so we’ll need to feed them historical ‘sources.json’ files
> eventually, I think Timothy was planning to do that eventually.

From my side, what I would like to achieve soon:

 1- compute sources.json by CI, which means turn the current code as derivation;
 2- add the Guix fixed-output substitutes in the list of upstream.

But how to do #1 is not clear for me.  And I have never tried this #2.

Then what is also missing is:

 3- have a collection of sources.json per Guix revision -- at least
some determined by us;
 4- make the SWH nixguix loader points to this collection.

Currently, the file 'sources.json' contains the field 'revision' which
is the Guix revision (commit).  What is not clear is if the collection
is by appending or via different files.  It is a pending discussion
with SWH folks, IIRC.


Cheers,
simon

Re: ImageMagick from 2020?

[Timothy Sample]

[-- Attachment #1: Type: text/plain, Size: 2074 bytes --]

Hey,

zimoun <zimon.toutoune@gmail.com> writes:

> On Wed, 19 Jan 2022 at 11:36, Ludovic Courtès <ludo@gnu.org> wrote:
>
>> Oh right, so we’ll need to feed them historical ‘sources.json’ files
>> eventually, I think Timothy was planning to do that eventually.
>
> From my side, what I would like to achieve soon:
>
> [...]
>
> Then what is also missing is:
>
>  3- have a collection of sources.json per Guix revision -- at least
> some determined by us;

I’ve attached a script that makes a “sources.json” per commit from the
PoG database.  It only lists regularly downloaded sources (no VCS
sources), since that’s all the SWH loader supports so far.

I also played around with it and came up with

    https://ngyro.com/pog-reports/2022-01-16/missing-sources.json

This is a “sources.json” that only lists the “missing” and “unknown”
sources from the PoG report.  It lists sources across all commits (since
1.0.0).  This might be the easiest thing for SWH to handle, since it
omits nearly 20k sources that they definitely already have.  Since they
don’t have the tarball hashes, they have no way to skip downloading and
processing tarballs that they already have by hash.  Hence, filtering it
with the extra data we have through the PoG projects should be something
that they welcome!

If they want, they could point a loader task at

   https://ngyro.com/pog-reports/latest/missing-sources.json

and I could publish updates when I publish new PoG reports.

There’s one other thing to think about.  Some of our sources are
arguably unsuitable for SWH.  For instance, our bootstrap binaries.  I
bet we have a bunch of other borderline things, too, like game assets.
Of course, if they are indiscriminately ingesting Github, I’m sure
they’ve loaded plenty of garbage.  Mostly, I think about these things
because I believe it’s important to maintain the Guix-SWH relationship.

Here’s the per-commit script.  You can run it like this:

    $ guile sources.scm pog.db output-directory


[-- Attachment #2: sources.scm --]
[-- Type: text/plain, Size: 3721 bytes --]

(use-modules (gcrypt base64)
             (guix base32)
             (guix build download)
             ((guix download) #:select (%mirrors))
             (ice-9 match)
             (json)
             (sqlite3)
             (srfi srfi-1)
             (srfi srfi-9 gnu)
             (srfi srfi-19)
             (web uri))

(define-immutable-record-type <commit>
  (make-commit push-time hash)
  commit?
  (push-time commit-push-time)
  (hash commit-hash))

(define lookup-commits-query "\
SELECT c.push_time,
    c.hash
FROM commits c
ORDER BY c.push_time DESC")

(define (lookup-commits db)
  (define (record->commit rec)
    (match-let ((#(push-time hash) rec))
      (make-commit (and push-time (make-time time-utc 0 push-time))
                   hash)))
  (define (kons rec acc)
    (cons (record->commit rec) acc))
  (let* ((stmt (sqlite-prepare db lookup-commits-query))
         (commits (sqlite-fold kons '() stmt)))
    (sqlite-finalize stmt)
    commits))

(define lookup-sources-query "\
SELECT f.hash,
    fr.reference
FROM commits c
    JOIN fod_commit_links fcl USING (commit_id)
    JOIN fods f USING (fod_id)
    JOIN fod_references fr USING (fod_id)
WHERE c.hash = ?
    AND f.algorithm = 'sha256'
    AND (fr.reference LIKE '\"%'
        OR fr.reference LIKE '(\"%')
    AND NOT fr.is_error")

(define (nix-base32-sha256->subresource-integrity digest)
  "Convert the Nix-style base32-encoded SHA-256 hash DIGEST into a
Subresource Integrity metadata value."
  (define bv (nix-base32-string->bytevector digest))
  (define b64 (base64-encode bv))
  (string-append "sha256-" b64))

(define (web-reference-urls reference)
  (define uris
    (match (call-with-input-string reference read)
      ((urls ...) (map string->uri urls))
      (url (list (string->uri url)))))
  (append-map (lambda (uri)
                (map uri->string
                     (maybe-expand-mirrors uri %mirrors)))
              uris))

(define (lookup-sources db commit)
  (define (record->url-source rec)
    (match-let ((#(digest reference) rec))
      (let ((urls (web-reference-urls reference))
            (integrity (nix-base32-sha256->subresource-integrity digest)))
        `(("type" . "url")
          ("urls" . ,(list->vector urls))
          ("integrity" . ,integrity)))))
  (define (kons rec acc)
    (cons (record->url-source rec) acc))
  (let* ((stmt (sqlite-prepare db lookup-sources-query))
         (_ (sqlite-bind-arguments stmt commit))
         (sources (sqlite-fold kons '() stmt)))
    (sqlite-finalize stmt)
    sources))

(define (commit-sources-name directory commit)
  (string-append directory
                 "/"
                 (date->string (time-utc->date (commit-push-time commit))
                               "~Y-~m-~d")
                 "-"
                 (string-take (commit-hash commit) 7)
                 "-sources.json"))

(match (program-arguments)
  ((_ db-file directory)
   (mkdir directory)
   (let* ((db (sqlite-open db-file))
          (commits (lookup-commits db)))
     (for-each (lambda (commit)
                 (call-with-output-file (commit-sources-name directory commit)
                   (lambda (out)
                     (let* ((hash (commit-hash commit))
                            (sources (lookup-sources db hash)))
                       (scm->json `(("version" . "1")
                                    ("revision" . ,hash)
                                    ("sources" . ,(list->vector sources)))
                                  out)
                       (newline out)))))
               (list (car commits)))
     (sqlite-close db)
     (exit EXIT_SUCCESS)))
  (_ (display "usage: sources.scm DB-FILE\n" (current-error-port))
     (exit EXIT_FAILURE)))

[-- Attachment #3: Type: text/plain, Size: 9 bytes --]



-- Tim

Re: ImageMagick from 2020?

[Ludovic Courtès]

Hi!

Timothy Sample <samplet@ngyro.com> skribis:

> I also played around with it and came up with
>
>     https://ngyro.com/pog-reports/2022-01-16/missing-sources.json
>
> This is a “sources.json” that only lists the “missing” and “unknown”
> sources from the PoG report.  It lists sources across all commits (since
> 1.0.0).  This might be the easiest thing for SWH to handle, since it
> omits nearly 20k sources that they definitely already have.  Since they
> don’t have the tarball hashes, they have no way to skip downloading and
> processing tarballs that they already have by hash.  Hence, filtering it
> with the extra data we have through the PoG projects should be something
> that they welcome!
>
> If they want, they could point a loader task at
>
>    https://ngyro.com/pog-reports/latest/missing-sources.json
>
> and I could publish updates when I publish new PoG reports.

Sounds great!  Could you ask them whether they could do that?  It may be
that it’s going to be a one-time thing, if everything goes well.

> There’s one other thing to think about.  Some of our sources are
> arguably unsuitable for SWH.  For instance, our bootstrap binaries.  I
> bet we have a bunch of other borderline things, too, like game assets.
> Of course, if they are indiscriminately ingesting Github, I’m sure
> they’ve loaded plenty of garbage.  Mostly, I think about these things
> because I believe it’s important to maintain the Guix-SWH relationship.

Right, it would be nice to filter them out somehow, even though it’s a
drop in the ocean of binaries that SWH ingests routinely (for instance
that’s the reason why we find some tarballs, as is, via
‘lookup-content’).

Thanks,
Ludo’.