Building guix-modular with cuirass

Building guix-modular with cuirass

[Mathieu Othacehe]

Hi Guix,

I setup my own cuirass server to build guix-modular, here's how:

--8<---------------cut here---------------start------------->8---
(define (build-guix-modular store arguments)
  (let* ((source (assq-ref arguments 'file-name))
         (revision (assq-ref arguments 'revision))
         (build (primitive-load (string-append source "/build-aux/build-self.scm")))
         (res (lambda ()
                `((#:job-name . ,(string-append "guix-" revision "-job"))
                  (#:derivation . ,(derivation-file-name
                                    (run-with-store store
                                      (build source
                                             #:version revision
                                             #:guile-version "2.2"))))))))
    (format (current-error-port) "---------> ~a|~a~%" arguments (res))
    res))
--8<---------------cut here---------------end--------------->8---

This is directly inspired by what I found in build-aux/ directory. While
everything seems great (thanks Ludo for this new feature :p), my other
machines do not use derivations builded by cuirass when running 'guix
pull'.

However, on the machine running cuirass, I have no build error an when
running 'guix pull', nothing is builded and everything works fine.

Any idea how to debug this issue ?

Thanks,

Mathieu

Re: Building guix-modular with cuirass

[Ludovic Courtès]

Hello Mathieu,

Mathieu Othacehe <m.othacehe@gmail.com> skribis:

> I setup my own cuirass server to build guix-modular, here's how:
>
> (define (build-guix-modular store arguments)

You shouldn’t need to do anything like that.  Here’s what we have on
berlin.guixsd.org:

--8<---------------cut here---------------start------------->8---
$ sudo sqlite3 /var/run/cuirass/cuirass.db 
SQLite version 3.19.3 2017-06-08 14:26:16
Enter ".help" for usage hints.
sqlite> select * from specifications where repo_name='guix-modular';
guix-modular|https://git.savannah.gnu.org/git/guix.git||build-aux/cuirass/guix-modular.scm|cuirass-jobs|((systems "x86_64-linux" "i686-linux" "aarch64-linux"))|master|||1
--8<---------------cut here---------------end--------------->8---

If you add a similar entry on your server, it should be fine.

Can you try and report back?

If Cuirass doesn’t seem to do what you want, I recommend browsing
/var/log/cuirass.log to see what’s going on.

HTH!

Ludo’.

Re: Building guix-modular with cuirass

[Mathieu Othacehe]

Hi Ludo,

> If you add a similar entry on your server, it should be fine.
>
> Can you try and report back?

Yes you're right it works fine, thanks :)

However, some of my machines are not picking up the substitutes builded
while running 'guix pull'.

I found why, on machines that I didn't reconfigure recently,
'default-guile' is guile-2.2.2, while on the cuirass machine it is
guile-2.2.3. When reconfiguring them before, substitutes are picked-up
correctly.

Thanks,

Mathieu

Re: Building guix-modular with cuirass

[Ludovic Courtès]

Hello,

Mathieu Othacehe <m.othacehe@gmail.com> skribis:

>> If you add a similar entry on your server, it should be fine.
>>
>> Can you try and report back?
>
> Yes you're right it works fine, thanks :)
>
> However, some of my machines are not picking up the substitutes builded
> while running 'guix pull'.
>
> I found why, on machines that I didn't reconfigure recently,
> 'default-guile' is guile-2.2.2, while on the cuirass machine it is
> guile-2.2.3. When reconfiguring them before, substitutes are picked-up
> correctly.

Oh indeed.

In the near future, I want ‘guix pull’ to install a ‘guix’ binary as
opposed to simply dropping a bunch of modules in ~/.config/guix/latest.
At that point we won’t have this kind of problem anymore.

Ludo’.

Re: Building guix-modular with cuirass

[Mathieu Othacehe]

Hey,

> In the near future, I want ‘guix pull’ to install a ‘guix’ binary as
> opposed to simply dropping a bunch of modules in ~/.config/guix/latest.
> At that point we won’t have this kind of problem anymore.

On the same topic, as I explained in a previous email
(https://lists.gnu.org/archive/html/guix-devel/2017-03/msg00222.html) my
main use-case for cuirass is to evaluate my manifest and build the
corresponding packages.

I could use 'cuirass-jobs' procedure and set arguments to (#:arguments
(subset . "my-package-1" "my-package-2" ...)). The drawback of this
approach is that everytime the manifest is updated, a reconfigure of
cuirass service is needed to update the package list.

I'd like to have an upstream mecanism were cuirass evaluates a local
manifest file (or even better take it from a git repository), an build
all the corresponding packages. The only configuration input from the
user would be the path of his manifest.

My first idea would be to add a piece of code in
build-aux/hydra/build-manifest.scm that would pull a manifest specified
as an argument, evaluates the packages it contains and feed it to
cuirass but that sounds a bit hacky.

Any better idea ?

Thanks,

Mathieu

Re: Building guix-modular with cuirass

[Nils Gillmann]

Ludovic Courtès transcribed 800 bytes:
> Hello,
> 
> Mathieu Othacehe <m.othacehe@gmail.com> skribis:
> 
> >> If you add a similar entry on your server, it should be fine.
> >>
> >> Can you try and report back?
> >
> > Yes you're right it works fine, thanks :)
> >
> > However, some of my machines are not picking up the substitutes builded
> > while running 'guix pull'.
> >
> > I found why, on machines that I didn't reconfigure recently,
> > 'default-guile' is guile-2.2.2, while on the cuirass machine it is
> > guile-2.2.3. When reconfiguring them before, substitutes are picked-up
> > correctly.
> 
> Oh indeed.
> 
> In the near future, I want ‘guix pull’ to install a ‘guix’ binary as
> opposed to simply dropping a bunch of modules in ~/.config/guix/latest.
> At that point we won’t have this kind of problem anymore.
> 
> Ludo’.
> 

Not trying to derail this thread too much, but could you explain that a
bit more Ludovic? I'm curious.
This is moving beyond the current change with modular guix (which still
drops a bunch of modules into the store and compiles them), correct?

Re: Building guix-modular with cuirass

[Ludovic Courtès]

Hi!

Mathieu Othacehe <m.othacehe@gmail.com> skribis:

> On the same topic, as I explained in a previous email
> (https://lists.gnu.org/archive/html/guix-devel/2017-03/msg00222.html) my
> main use-case for cuirass is to evaluate my manifest and build the
> corresponding packages.
>
> I could use 'cuirass-jobs' procedure and set arguments to (#:arguments
> (subset . "my-package-1" "my-package-2" ...)). The drawback of this
> approach is that everytime the manifest is updated, a reconfigure of
> cuirass service is needed to update the package list.
>
> I'd like to have an upstream mecanism were cuirass evaluates a local
> manifest file (or even better take it from a git repository), an build
> all the corresponding packages. The only configuration input from the
> user would be the path of his manifest.
>
> My first idea would be to add a piece of code in
> build-aux/hydra/build-manifest.scm that would pull a manifest specified
> as an argument, evaluates the packages it contains and feed it to
> cuirass but that sounds a bit hacky.
>
> Any better idea ?

Perhaps we can make the package selection mechanism in
build-aux/hydra/gnu-system.scm a bit more flexible and export the
relevant procedures so you can write your own build-manifest.scm that
would just be a few lines and otherwise reusing what gnu-system.scm
already provides.

How does that sound?

Ludo’.

Re: Building guix-modular with cuirass

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 2216 bytes --]

Nils Gillmann <ng0@n0.is> writes:

> Ludovic Courtès transcribed 800 bytes:
>>
>> In the near future, I want ‘guix pull’ to install a ‘guix’ binary as
>> opposed to simply dropping a bunch of modules in ~/.config/guix/latest.
>> At that point we won’t have this kind of problem anymore.
>> 
>> Ludo’.
>> 
>
> Not trying to derail this thread too much, but could you explain that a
> bit more Ludovic? I'm curious.
> This is moving beyond the current change with modular guix (which still
> drops a bunch of modules into the store and compiles them), correct?

I think he's referring to his latest comments here:

https://debbugs.gnu.org/cgi/bugreport.cgi?bug=22629

Today, the "guix" command is a thin wrapper, as explained here:

https://lists.gnu.org/archive/html/help-guix/2017-09/msg00092.html

Going forward, I think Ludo is suggesting that we should no longer rely
on the thin wrapper to delegate business logic to whatever is installed
in ~/.config/guix/latest; instead, "guix pull" would basically build a
new Guix (including the "guix" command) in a profile located at
~/.config/guix/current.  Each user would use their own "guix" command.

Currently, when a user invokes a command like "guix package -i foo", the
path of execution is roughly like this:

1. /run/current-system/profile/bin/guix (or, if it's a foreign
distribution and the user has installed Guix according to the manual,
this will be /usr/local/bin/guix, which points to
/var/guix/profiles/per-user/root/guix-profile/bin/guix)
2. ~/.config/guix/latest/${whatever_module_was_invoked}

But after the proposed improvement, the path of execution would be:

1. ~/.config/guix/current/bin/guix
2. ~/.config/guix/current/${whatever_module_was_invoked}

So, every Guix installation would truly be self-contained, unlike the
current situation, where the same thin "guix" command is shared by every
user.  And since ~/.config/guix/current is basically just a profile, I
think we'd also get roll-back for free, which is nice because currently
roll-back after a "guix pull" isn't as easy as it could be

Ludo, please feel free to correct me if I'm misrepresenting anything.

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Building guix-modular with cuirass

[swedebugia]

On 2018-05-13 04:17, Chris Marusich wrote:
> Nils Gillmann <ng0@n0.is> writes:
>
>> Ludovic Courtès transcribed 800 bytes:
>>> In the near future, I want ‘guix pull’ to install a ‘guix’ binary as
>>> opposed to simply dropping a bunch of modules in ~/.config/guix/latest.
>>> At that point we won’t have this kind of problem anymore.
>>>
>>> Ludo’.
>>>
>> Not trying to derail this thread too much, but could you explain that a
>> bit more Ludovic? I'm curious.
>> This is moving beyond the current change with modular guix (which still
>> drops a bunch of modules into the store and compiles them), correct?
> I think he's referring to his latest comments here:
>
> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=22629
>
> Today, the "guix" command is a thin wrapper, as explained here:
>
> https://lists.gnu.org/archive/html/help-guix/2017-09/msg00092.html
>
> Going forward, I think Ludo is suggesting that we should no longer rely
> on the thin wrapper to delegate business logic to whatever is installed
> in ~/.config/guix/latest; instead, "guix pull" would basically build a
> new Guix (including the "guix" command) in a profile located at
> ~/.config/guix/current.  Each user would use their own "guix" command.
>
> Currently, when a user invokes a command like "guix package -i foo", the
> path of execution is roughly like this:
>
> 1. /run/current-system/profile/bin/guix (or, if it's a foreign
> distribution and the user has installed Guix according to the manual,
> this will be /usr/local/bin/guix, which points to
> /var/guix/profiles/per-user/root/guix-profile/bin/guix)
> 2. ~/.config/guix/latest/${whatever_module_was_invoked}
>
> But after the proposed improvement, the path of execution would be:
>
> 1. ~/.config/guix/current/bin/guix
> 2. ~/.config/guix/current/${whatever_module_was_invoked}
>
> So, every Guix installation would truly be self-contained, unlike the
> current situation, where the same thin "guix" command is shared by every
> user.  And since ~/.config/guix/current is basically just a profile, I
> think we'd also get roll-back for free, which is nice because currently
> roll-back after a "guix pull" isn't as easy as it could be
>
> Ludo, please feel free to correct me if I'm misrepresenting anything.
Sounds like nice improvement to me. :)

Guix pull seems still to be the least robust and least user friendly 
part of Guix according to my experience.