guix weather -m etc/sources-manifest.scm and CI?

guix weather -m etc/sources-manifest.scm and CI?

[zimoun]

Hi,

Playing with the new 'etc/sources-manifest.scm', using fb32a38, I get:

--8<---------------cut here---------------start------------->8---
$ guix weather -m ~/src/guix/guix/etc/source-manifest.scm
computing 16,831 package derivations for x86_64-linux...
looking for 16,831 store items on https://ci.guix.gnu.org...
https://ci.guix.gnu.org
  74.6% substitutes available (12,556 out of 16,831)
  at least 65,367.1 MiB of nars (compressed)
  81,988.1 MiB on disk (uncompressed)
  0.095 seconds per request (1,606.8 seconds in total)
  10.5 requests per second

  0.0% (0 out of 4,275) of the missing items are queued
  5 queued builds
      aarch64-linux: 4 (80.0%)
      powerpc64le-linux: 1 (20.0%)
  build rate: .00 builds per hour
      powerpc64le-linux: 0.00 builds per hour
      aarch64-linux: 0.00 builds per hour
      i686-linux: 0.00 builds per hour
      x86_64-linux: 0.00 builds per hour
looking for 16,831 store items on https://bordeaux.guix.gnu.org...
https://bordeaux.guix.gnu.org
  99.8% substitutes available (16,804 out of 16,831)
  62,195.0 MiB of nars (compressed)
  108,212.7 MiB on disk (uncompressed)
  0.049 seconds per request (829.2 seconds in total)
  20.3 requests per second
  (continuous integration information unavailable)
--8<---------------cut here---------------end--------------->8---

The questions are:

Why ci.guix.gnu.org contains only 75%?  And bordeaux almost everything?
(I guess the missing ones on bordeaux are corner cases as icecat, linux-libre).

Does it make sense to duplicate the storage of all these origins?

Using extensively "guix time-machine", I note that a lot of
derivations are missing and thus they are built locally, which is
costly on poor machine.  Could we reduce the duplication and so save
some space in order to systematically keep these derivations?  It
would greatly ease the Guix experience for "guix time-machine" users.
:-)

Cheers,
simon

PS: about etc/disarchive-manifest.scm, I guess 'all-origins' is missing:

--8<---------------cut here---------------start------------->8---
(let ((origins (all-origins)))
  (manifest
   (list (manifest-entry
           (name "disarchive-collection")
           (version (length origins))
           (item (disarchive-collection origins))))))
--8<---------------cut here---------------end--------------->8---

Re: guix weather -m etc/sources-manifest.scm and CI?

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 8583 bytes --]


zimoun <zimon.toutoune@gmail.com> writes:

> Playing with the new 'etc/sources-manifest.scm', using fb32a38, I get:
>
> $ guix weather -m ~/src/guix/guix/etc/source-manifest.scm
> computing 16,831 package derivations for x86_64-linux...
> looking for 16,831 store items on https://ci.guix.gnu.org...
> https://ci.guix.gnu.org
>   74.6% substitutes available (12,556 out of 16,831)
>   at least 65,367.1 MiB of nars (compressed)
>   81,988.1 MiB on disk (uncompressed)
>   0.095 seconds per request (1,606.8 seconds in total)
>   10.5 requests per second
>
>   0.0% (0 out of 4,275) of the missing items are queued
>   5 queued builds
>       aarch64-linux: 4 (80.0%)
>       powerpc64le-linux: 1 (20.0%)
>   build rate: .00 builds per hour
>       powerpc64le-linux: 0.00 builds per hour
>       aarch64-linux: 0.00 builds per hour
>       i686-linux: 0.00 builds per hour
>       x86_64-linux: 0.00 builds per hour
> looking for 16,831 store items on https://bordeaux.guix.gnu.org...
> https://bordeaux.guix.gnu.org
>   99.8% substitutes available (16,804 out of 16,831)
>   62,195.0 MiB of nars (compressed)
>   108,212.7 MiB on disk (uncompressed)
>   0.049 seconds per request (829.2 seconds in total)
>   20.3 requests per second
>   (continuous integration information unavailable)
>
>
> The questions are:
>
> Why ci.guix.gnu.org contains only 75%?  And bordeaux almost everything?
> (I guess the missing ones on bordeaux are corner cases as icecat, linux-libre).

bordeaux.guix.gnu.org is hopefully only missing substitutes for things
where there's actually an issue building them. It's possible not to
guess though and instead ask guix weather what is missing.

I used time-machine as well for the latest commit processed by
data.guix.gnu.org, just so that the list doesn't include builds which
haven't started yet.

→ guix time-machine --commit=33bc3fb2a5f30a6e21f1b8d6d43867d921bd951c -- weather --substitute-urls=https://bordeaux.guix.gnu.org --display-missing -m ./etc/source-manifest.scm
computing 16,850 package derivations for x86_64-linux...
looking for 16,850 store items on https://bordeaux.guix.gnu.org...
https://bordeaux.guix.gnu.org
  99.8% substitutes available (16,822 out of 16,850)
  62,377.9 MiB of nars (compressed)
  108,593.9 MiB on disk (uncompressed)
  0.160 seconds per request (4.5 seconds in total)
  6.3 requests per second
  (continuous integration information unavailable)

Substitutes are missing for the following items:
  /gnu/store/42knh9b75m6kc30m8v247sswhdfqnn8i-xpp3-1.1.4_src.tgz                                                                                         i686-linux
  /gnu/store/9fgglvi13vhpc63knf15dipzmmck6ia9-mini-os-git-checkout                                                                                       i586-gnu
  /gnu/store/pb5jmi9zalg6xylzsjmrskwxs0kar97l-fossil-src-2.11.tar.gz                                                                                     armhf-linux
  /gnu/store/8s5d2hjd6nhzf6m9dlxrykk8ijkf62pi-texlive-marginnote-51265-checkout                                                                          armhf-linux
  /gnu/store/cp3ka40bhb28rrmyj4mzf9xjhi0ssxjx-CombBLAS_beta_16_2.tgz                                                                                     x86_64-linux
  /gnu/store/f1h94axw82id0k8c2lippg73sqlibqs8-dovecot-trees-2.1.0.tar.gz                                                                                 aarch64-linux
  /gnu/store/f12l7dg2z1xyf2wcw9g3v7jwpgd8m5zv-tla2tools-1.8.0-checkout                                                                                   i686-linux
  /gnu/store/53j9996hdgnmhgzswjjggdz9wnv29p5b-jpegsrc.v9d.tar.gz                                                                                         i586-gnu
  /gnu/store/80s1m4q2hnjfbqzw3fhywvsyim2b00cd-gcc-4.7.4.tar.bz2                                                                                          x86_64-linux
  /gnu/store/csv5xca0p8w5jqqx53szy2dja8lwxma2-unicode-blocks.txt                                                                                         i686-linux
  /gnu/store/xn62dzq9hw3qnvmbxyxjkvhlacs72rz7-canada1500.zip                                                                                             armhf-linux
  /gnu/store/6vqin3by3nkn0sxhgwnzi9l7gflpfw1q-gcc-vc4-6.5.0-checkout                                                                                     x86_64-linux
  /gnu/store/zxfkf2bzq7pp7nhmbdgzvmjp0iv46wds-propeller-gcc-b4f45a4725e0b6d0af59e594c4e3e35ca4105867-checkout                                            i686-linux
  /gnu/store/g98l0m8qsfdqybak2jz3ma2miv9bki4j-emacs-danneskjold-theme-0.0.0-2.e4d1f2c-checkout                                                           x86_64-linux
  /gnu/store/bznj81ls02r7kwld6338dhba0pql7nik-Rserve_1.8-6.tar.gz                                                                                        i686-linux
  /gnu/store/fb0rqvv38pf02qv3zjvjsindjdhlbpca-rda_1.0.2-2.1.tar.gz                                                                                       x86_64-linux
  /gnu/store/m5jn626l55nwxhgs405byka7pzkg1arz-d3.v6.js                                                                                                   powerpc64le-linux
  /gnu/store/bf496ni94ik19gdxkgp0vddkfxbid2vf-d3.v5.js                                                                                                   mips64el-linux
  /gnu/store/c0kig783lv56vv0yc6i8rsvhaxdp52sd-paml4.9e.tgz                                                                                               mips64el-linux
  /gnu/store/17pi83jai4wlnnv1dilqwv1xv6g35d7x-plink-1.07-src.zip                                                                                         aarch64-linux
  /gnu/store/c3k6gxx7yha95f4avia11ls71ndzfx34-ggtree_3.0.3.tar.gz                                                                                        x86_64-linux
  /gnu/store/g2fhcc16yk5asszzmih0cpkwqb3b9bb3-clusterProfiler_4.0.4.tar.gz                                                                               x86_64-linux
  /gnu/store/7n2rk9n46am8qx03ppycvkcdbnh6qzkq-eye-1.4.1.tar.gz                                                                                           aarch64-linux
  /gnu/store/izhvfxx0iiwv8p1xr8ag5mpvsi86x9kx-weightwatcher-1.12.tar.gz                                                                                  mips64el-linux
  /gnu/store/l3r5j8r4f6fh6bxa9va3681n3bclxpdq-stuff-1.26.0.tar.gz                                                                                        i686-linux
  /gnu/store/ymip20qmlz320r2zx0hhfxn70428n76z-skymaker-3.10.5.tar.gz                                                                                     armhf-linux
  /gnu/store/xqc7n8fsq5zxag37a2ay88cx6c3zsm51-mpfrcx-0.6.3.tar.gz                                                                                        x86_64-linux
  /gnu/store/yf2dgn6jcdwvbnzcjwcadc8lwr9ks0wr-eclib-20190909-checkout                                                                                    mips64el-linux


If you visit https://data.guix.gnu.org and add the store path to the
URL, you can get links to the failed builds ([1] for example).

1: https://data.guix.gnu.org/gnu/store/42knh9b75m6kc30m8v247sswhdfqnn8i-xpp3-1.1.4_src.tgz

> Does it make sense to duplicate the storage of all these origins?
>
> Using extensively "guix time-machine", I note that a lot of
> derivations are missing and thus they are built locally, which is
> costly on poor machine.  Could we reduce the duplication and so save
> some space in order to systematically keep these derivations?  It
> would greatly ease the Guix experience for "guix time-machine" users.
> :-)

In terms of duplication, I still think that an argument can be made that
bordeaux.guix.gnu.org is providing value, even if it's doing some of the
same stuff as ci.guix.gnu.org.

As for keeping build results, everything that's ever been built for
bordeaux.guix.gnu.org (that's only ~337739 things totalling ~1.4TiBs) is
still around. In some ways, this is because deleting things is a bit
more difficult, as the files aren't in the store, you can't just do a
guix gc.

However, I do want to try to never delete anything. That's going to
require a bit of work as the local storage on the bayfront machine will
be all used up at some point, but I do have the beginnings of a plan to
avoid this an keep building things.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: guix weather -m etc/sources-manifest.scm and CI?

[zimoun]

Hi Chris,

On Fri, 17 Sep 2021 at 10:49, Christopher Baines <mail@cbaines.net> wrote:
> zimoun <zimon.toutoune@gmail.com> writes:

>> https://ci.guix.gnu.org
>>   74.6% substitutes available (12,556 out of 16,831)

[...]

>> https://bordeaux.guix.gnu.org
>>   99.8% substitutes available (16,804 out of 16,831)

>> The questions are:
>>
>> Why ci.guix.gnu.org contains only 75%?  And bordeaux almost everything?
>> (I guess the missing ones on bordeaux are corner cases as icecat, linux-libre).
>
> bordeaux.guix.gnu.org is hopefully only missing substitutes for things
> where there's actually an issue building them. It's possible not to
> guess though and instead ask guix weather what is missing.

[...]

> If you visit https://data.guix.gnu.org and add the store path to the
> URL, you can get links to the failed builds ([1] for example).
>
> 1:
> https://data.guix.gnu.org/gnu/store/42knh9b75m6kc30m8v247sswhdfqnn8i-xpp3-1.1.4_src.tgz

Cool!  Thanks for explaining.  Let try to systematically address this
failures about the source. :-)


>> Does it make sense to duplicate the storage of all these origins?
>>
>> Using extensively "guix time-machine", I note that a lot of
>> derivations are missing and thus they are built locally, which is
>> costly on poor machine.  Could we reduce the duplication and so save
>> some space in order to systematically keep these derivations?  It
>> would greatly ease the Guix experience for "guix time-machine" users.
>> :-)
>
> In terms of duplication, I still think that an argument can be made that
> bordeaux.guix.gnu.org is providing value, even if it's doing some of the
> same stuff as ci.guix.gnu.org.

My question with duplication is about “long term storage of all the
sources”.  Not about building twice (it is easy to find justification
via reproducibility checker for instance :-)).

Now Guix have 2 build farms so let consider it is a strength. :-) The
question is how to exploit such strength by sharing the workload and
have a better global scaling up (and not 2 independent scaling).  For
what my opinion is worth here. :-)

> As for keeping build results, everything that's ever been built for
> bordeaux.guix.gnu.org (that's only ~337739 things totalling ~1.4TiBs) is
> still around. In some ways, this is because deleting things is a bit
> more difficult, as the files aren't in the store, you can't just do a
> guix gc.
>
> However, I do want to try to never delete anything. That's going to
> require a bit of work as the local storage on the bayfront machine will
> be all used up at some point, but I do have the beginnings of a plan to
> avoid this an keep building things.

From my understanding, the project does not have a lot of resources;
especially about storage.  And my question is: do we have a “coherent”
plan between the machines behind Bordeaux and the ones behind Berlin?

I mean, IMHO, it does not seem a correct usage of project resource to
back twice all from v1.0 to now (maybe from earlier to later).  And I
also have in mind energy consumption which is becoming more and more a
precious value.

For instance, we could imagine the both build farms build and challenge
the outputs.  Then, on Berlin the identical outputs are deleted after
say 6 months and only live on Bordeaux.  Bordeaux being the backup
time-machine.  That’s only an idea to open the discussion. :-) Not a
concrete plan.

Moreover, Disarchive is also something that should be added; probably to
machines behind Bordeaux in the above plan.

From my point of view, what is really missing are the derivations for
*all* the commits; for example module-import.drv, guix-33bc3fb2a.drv,
guix-command.drv, guix-module-union.drv, guix-33bc3fb2a-modules.drv,
guix-packages-modules.drv, guix-system-tests-modules.drv,
guix-packages-base-modules.drv, etc.  And the backup time-machine should
systematically build them for all the reachable commits.


Cheers,
simon

Re: guix weather -m etc/sources-manifest.scm and CI?

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 3769 bytes --]


zimoun <zimon.toutoune@gmail.com> writes:

>> As for keeping build results, everything that's ever been built for
>> bordeaux.guix.gnu.org (that's only ~337739 things totalling ~1.4TiBs) is
>> still around. In some ways, this is because deleting things is a bit
>> more difficult, as the files aren't in the store, you can't just do a
>> guix gc.
>>
>> However, I do want to try to never delete anything. That's going to
>> require a bit of work as the local storage on the bayfront machine will
>> be all used up at some point, but I do have the beginnings of a plan to
>> avoid this an keep building things.
>
> From my understanding, the project does not have a lot of resources;
> especially about storage.  And my question is: do we have a “coherent”
> plan between the machines behind Bordeaux and the ones behind Berlin?

Not that I'm following at least. The machines behind
bordeaux.guix.gnu.org were just ones that I could find to use when the
plan of dpeloying behind ci.guix.gnu.org was looking less and less
likely.

> I mean, IMHO, it does not seem a correct usage of project resource to
> back twice all from v1.0 to now (maybe from earlier to later).  And I
> also have in mind energy consumption which is becoming more and more a
> precious value.
>
> For instance, we could imagine the both build farms build and challenge
> the outputs.  Then, on Berlin the identical outputs are deleted after
> say 6 months and only live on Bordeaux.  Bordeaux being the backup
> time-machine.  That’s only an idea to open the discussion. :-) Not a
> concrete plan.

Some duplication of stored data is going to be necessary for redundancy
I think, for bordeaux.guix.gnu.org I was thinking of at least trying to
store all the substitute data in two different places.

As you've mentioned, building the same things multiple times is also
valuable for testing reproduciblity.

> Moreover, Disarchive is also something that should be added; probably to
> machines behind Bordeaux in the above plan.

I haven't been following Disarchive too closely. I think I remember
thinking it might be interesting for it to listen to data from
data.guix.gnu.org, to find out about new source tarballs (which isn't
really possible yet).

> From my point of view, what is really missing are the derivations for
> *all* the commits; for example module-import.drv, guix-33bc3fb2a.drv,
> guix-command.drv, guix-module-union.drv, guix-33bc3fb2a-modules.drv,
> guix-packages-modules.drv, guix-system-tests-modules.drv,
> guix-packages-base-modules.drv, etc.  And the backup time-machine should
> systematically build them for all the reachable commits.

If you're just talking about the derivations, data.guix.gnu.org does
store these derivations for the commits it processes (which are just the
states of the master branch, and that data is also missing for some
older commits that were processed a while ago).

If you're talking about substitutes for the outputs of these
derivations, the situation on bordeaux.guix.gnu.org isn't great for
x86_64-linux currently, since building guile@3.0.7 is blocking building
these things for recent revisions [1]. The underlying reason is a
problem with the testsuite I think, but it's hard to work around for
some other reasons as well. Once it can be built, all the blocked builds
will be attempted too.

1: https://data.guix.gnu.org/gnu/store/5z950sh1wpi59mfh53qdjdn8qnlcwp8g-guile-3.0.7.drv

I am interested in what can be done to support using guix time-machine
though, I think there's a range of things like storing substitutes,
testing things better (so they don't start failing to build in the
future), and helping users find the right commits to use.

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: guix weather -m etc/sources-manifest.scm and CI?

[Ludovic Courtès]

Hello!

zimoun <zimon.toutoune@gmail.com> skribis:

> Playing with the new 'etc/sources-manifest.scm', using fb32a38, I get:
>
> $ guix weather -m ~/src/guix/guix/etc/source-manifest.scm
> computing 16,831 package derivations for x86_64-linux...
> looking for 16,831 store items on https://ci.guix.gnu.org...
> https://ci.guix.gnu.org
>   74.6% substitutes available (12,556 out of 16,831)
>   at least 65,367.1 MiB of nars (compressed)
>   81,988.1 MiB on disk (uncompressed)
>   0.095 seconds per request (1,606.8 seconds in total)
>   10.5 requests per second
>
>   0.0% (0 out of 4,275) of the missing items are queued
>   5 queued builds
>       aarch64-linux: 4 (80.0%)
>       powerpc64le-linux: 1 (20.0%)
>   build rate: .00 builds per hour
>       powerpc64le-linux: 0.00 builds per hour
>       aarch64-linux: 0.00 builds per hour
>       i686-linux: 0.00 builds per hour
>       x86_64-linux: 0.00 builds per hour
> looking for 16,831 store items on https://bordeaux.guix.gnu.org...
> https://bordeaux.guix.gnu.org
>   99.8% substitutes available (16,804 out of 16,831)
>   62,195.0 MiB of nars (compressed)
>   108,212.7 MiB on disk (uncompressed)
>   0.049 seconds per request (829.2 seconds in total)
>   20.3 requests per second
>   (continuous integration information unavailable)
>
>
> The questions are:
>
> Why ci.guix.gnu.org contains only 75%?  And bordeaux almost everything?
> (I guess the missing ones on bordeaux are corner cases as icecat, linux-libre).

I wonder too!

BTW, I meant to send a message here about these two manifests, and then
got caught in a crazy patch-review/bug-fix/work loop.  The goal of this
manifest is to run ‘weather’ and ‘build’ so we can get all the source
and check the status at once, as you found out.

I set up a Cuirass job the other day¹, but due to my aforementioned
disorganization, I haven’t yet taken the time to investigate its
failure.

¹ https://ci.guix.gnu.org/jobset/source

> Does it make sense to duplicate the storage of all these origins?

It surely does.

> PS: about etc/disarchive-manifest.scm, I guess 'all-origins' is missing:
>
> (let ((origins (all-origins)))
>   (manifest
>    (list (manifest-entry
>            (name "disarchive-collection")
>            (version (length origins))
>            (item (disarchive-collection origins))))))

No, there’s:

  (include "source-manifest.scm")

at the top.

So the goal here was to move forward on this front by (1) setting up a
CI jobset, and (2) having an mcron job or something that would
periodically copy the “disarchives” in a directory served by nginx.  We
have had all the code and most of the infrastructure for several months,
so I thought it’s high time to get our act together.  Help welcome!

Thanks,
Ludo’.

Re: guix weather -m etc/sources-manifest.scm and CI?

[zimoun]

Hi,

On Thu, 23 Sep 2021 at 22:18, Ludovic Courtès <ludo@gnu.org> wrote:

> BTW, I meant to send a message here about these two manifests

Sorry if I shot your surprise effect announcement. :-)

> I set up a Cuirass job the other day¹, but due to my aforementioned
> disorganization, I haven’t yet taken the time to investigate its
> failure.
>
> ¹ https://ci.guix.gnu.org/jobset/source

Cool!

(Find my stuff with Cuirass always requires magic for me :-))


>> Does it make sense to duplicate the storage of all these origins?
>
> It surely does.

Do you mean having all twice: on Berlin and Bordeaux.  In addition to SWH
(although it is not ready yet).


>> PS: about etc/disarchive-manifest.scm, I guess 'all-origins' is missing:

> No, there’s:
>
>   (include "source-manifest.scm")
>
> at the top.

Oops!


> So the goal here was to move forward on this front by (1) setting up a
> CI jobset, and (2) having an mcron job or something that would
> periodically copy the “disarchives” in a directory served by nginx.  We
> have had all the code and most of the infrastructure for several months,
> so I thought it’s high time to get our act together.  Help welcome!

Thanks for pushing forward. :-)

Cheers,
simon

Re: guix weather -m etc/sources-manifest.scm and CI?

[Ludovic Courtès]

Hello!

zimoun <zimon.toutoune@gmail.com> skribis:

> On Thu, 23 Sep 2021 at 22:18, Ludovic Courtès <ludo@gnu.org> wrote:

[...]

>> I set up a Cuirass job the other day¹, but due to my aforementioned
>> disorganization, I haven’t yet taken the time to investigate its
>> failure.
>>
>> ¹ https://ci.guix.gnu.org/jobset/source
>
> Cool!

It turns out that at first sight I can’t see what I’m doing wrong here.
Evaluation fails with:

--8<---------------cut here---------------start------------->8---
Computing Guix derivation for 'x86_64-linux'...  
ice-9/boot-9.scm:1685:16: In procedure raise-exception:
#<unknown port>:16:103: Unknown # object: "#<"
--8<---------------cut here---------------end--------------->8---

Mathieu, could it have to do with the fact that the manifest is within
the ‘guix’ channel?

Thanks,
Ludo’.

Re: guix weather -m etc/sources-manifest.scm and CI?

[Mathieu Othacehe]

[-- Attachment #1: Type: text/plain, Size: 1846 bytes --]


Hey!

> Mathieu, could it have to do with the fact that the manifest is within
> the ‘guix’ channel?

Using the attached little reproducer, I get a more interesting
backtrace:

--8<---------------cut here---------------start------------->8---
Backtrace:
In ice-9/boot-9.scm:
  1752:10 11 (with-exception-handler _ _ #:unwind? _ # _)
In unknown file:
          10 (apply-smob/0 #<thunk 7f84f88adf60>)
In ice-9/boot-9.scm:
    724:2  9 (call-with-prompt _ _ #<procedure default-prompt-handle…>)
In ice-9/eval.scm:
    619:8  8 (_ #(#(#<directory (guile-user) 7f84f88a7c80>)))
In ice-9/boot-9.scm:
   2835:4  7 (save-module-excursion _)
  4380:12  6 (_)
In gnu/ci.scm:
    496:4  5 (cuirass-jobs _ _)
In srfi/srfi-1.scm:
   673:15  4 (append-map _ _ . _)
   586:17  3 (map1 (list "x86_64-linux"))
   586:17  2 (map1 (#<origin "https://www.brain-dump.org/projects…> …))
In gnu/ci.scm:
   575:38  1 (_ #<origin "https://www.brain-dump.org/projects/abduco…>)
In guix/packages.scm:
    441:0  0 (%package-name-procedure #<origin "https://www.brain-du…>)

guix/packages.scm:441:0: In procedure %package-name-procedure:
In procedure package-name: Wrong type argument: #<origin "https://www.brain-dump.org/projects/abduco/abduco-0.6.tar.gz" #<content-hash sha256:1x1m58ckwsprljgmdy93mvgjyg9x3cqrzdf3mysp0mx97zhhj2f9> () 7f84f252ec00>
--8<---------------cut here---------------end--------------->8---

The issue is that (gnu ci) expects manifests of packages and not
manifests of origins. Hence, the "job-name" procedure that calls
"package-name" on an origin fails.

We could maybe generalize the manifests->packages to support packages
and origins. The package-job procedure would also need some adjustment
to call origin->derivation instead of package-derivation.

WDYT?

Thanks,

Mathieu

[-- Attachment #2: cuirass.scm --]
[-- Type: application/octet-stream, Size: 443 bytes --]

(use-modules (gnu)
             (guix)
             (gnu ci))

(define s (open-connection))
(define args
  '((subset . (manifests "/etc/source-manifest.scm"))
    (systems . (list "x86_64-linux"))
    (channels
     (repository
      (version 0)
      (url "/gnu/store/22av86m87c4x574cc2070ijbd5mcbhz6-guix-source")
      (branch "master")
      (commit "0000000000000000000000000000000000000000")
      (name guix)))))

(cuirass-jobs s args)

Re: guix weather -m etc/sources-manifest.scm and CI?

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 586 bytes --]

Hi,

Mathieu Othacehe <othacehe@gnu.org> skribis:

> The issue is that (gnu ci) expects manifests of packages and not
> manifests of origins. Hence, the "job-name" procedure that calls
> "package-name" on an origin fails.

Ah, silly me.

> We could maybe generalize the manifests->packages to support packages
> and origins. The package-job procedure would also need some adjustment
> to call origin->derivation instead of package-derivation.

How about this change?  That will create jobs for manifests containing
any kind of file-like object.

Thanks!

Ludo’.


[-- Attachment #2: Type: text/x-patch, Size: 3524 bytes --]

diff --git a/gnu/ci.scm b/gnu/ci.scm
index ceb1b94af9..e1011355db 100644
--- a/gnu/ci.scm
+++ b/gnu/ci.scm
@@ -1,5 +1,5 @@
 ;;; GNU Guix --- Functional package management for GNU
-;;; Copyright © 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020 Ludovic Courtès <ludo@gnu.org>
+;;; Copyright © 2012-2021 Ludovic Courtès <ludo@gnu.org>
 ;;; Copyright © 2017, 2020 Jan (janneke) Nieuwenhuizen <janneke@gnu.org>
 ;;; Copyright © 2018, 2019 Clément Lassieur <clement@lassieur.org>
 ;;; Copyright © 2020 Julien Lepiller <julien@lepiller.eu>
@@ -86,7 +86,7 @@
 (define* (derivation->job name drv
                           #:key
                           (max-silent-time 3600)
-                          (timeout 3600))
+                          (timeout (* 5 3600)))
   "Return a Cuirass job called NAME and describing DRV.
 
 MAX-SILENT-TIME and TIMEOUT are build options passed to the daemon when
@@ -443,19 +443,40 @@ valid."
               (map channel-url channels)))
        arguments))
 
-(define (manifests->packages store manifests)
-  "Return the list of packages found in MANIFESTS."
+(define (manifests->jobs store manifests)
+  "Return the list of jobs for the entries in MANIFESTS, a list of file
+names."
   (define (load-manifest manifest)
     (save-module-excursion
      (lambda ()
        (set-current-module (make-user-module '((guix profiles) (gnu))))
        (primitive-load manifest))))
 
-  (delete-duplicates!
-   (map manifest-entry-item
-        (append-map (compose manifest-entries
-                             load-manifest)
-                    manifests))))
+  (define (manifest-entry-job-name entry)
+    (string-append (manifest-entry-name entry) "-"
+                   (manifest-entry-version entry)))
+
+  (define (manifest-entry->job entry)
+    (let* ((obj (manifest-entry-item entry))
+           (drv (parameterize ((%graft? #f))
+                  (run-with-store store
+                    (lower-object obj))))
+           (max-silent-time (or (and (package? obj)
+                                     (assoc-ref (package-properties obj)
+                                                'max-silent-time))
+                                3600))
+           (timeout (or (and (package? obj)
+                             (assoc-ref (package-properties obj) 'timeout))
+                        (* 5 3600))))
+      (derivation->job (manifest-entry-job-name entry) drv
+                       #:max-silent-time max-silent-time
+                       #:timeout timeout)))
+
+  (map manifest-entry->job
+       (delete-duplicates
+        (append-map (compose manifest-entries load-manifest)
+                    manifests)
+        manifest-entry=?)))
 
 (define (arguments->systems arguments)
   "Return the systems list from ARGUMENTS."
@@ -568,12 +589,8 @@ valid."
                  packages)))
          (('manifests . rest)
           ;; Build packages in the list of manifests.
-          (let* ((manifests (arguments->manifests rest channels))
-                 (packages (manifests->packages store manifests)))
-            (map (lambda (package)
-                   (package-job store (job-name package)
-                                package system))
-                 packages)))
+          (let ((manifests (arguments->manifests rest channels)))
+            (manifests->jobs store manifests)))
          (else
           (error "unknown subset" subset))))
      systems)))

Re: guix weather -m etc/sources-manifest.scm and CI?

[Mathieu Othacehe]

Hey Ludo,

> -                                package system))
> -                 packages)))
> +          (let ((manifests (arguments->manifests rest channels)))
> +            (manifests->jobs store manifests)))

This sounds great to me :)

Thanks,

Mathieu