bug#65225: [PATCH] environment: Build the profile for the requested system.

bug#65225: [PATCH] environment: Build the profile for the requested system.

[Tobias Geerinckx-Rice via Bug reports for GNU Guix]

Previously, ‘--system=’ did not affect profile hooks, meaning that all
packages would be built for both the host and requested systems.

* guix/scripts/environment.scm (guix-environment*): Parameterize
%current-system to match the requested ‘--system=’.

Reported by ekaitz in #guix.
---
 guix/scripts/environment.scm | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/guix/scripts/environment.scm b/guix/scripts/environment.scm
index 9712389842..27f7e53549 100644
--- a/guix/scripts/environment.scm
+++ b/guix/scripts/environment.scm
@@ -1146,7 +1146,8 @@ (define (guix-environment* opts)
           (warning (G_ "no packages specified; creating an empty environment~%")))
 
         ;; Use the bootstrap Guile when requested.
-        (parameterize ((%graft? (assoc-ref opts 'graft?))
+        (parameterize ((%current-system system)
+                       (%graft? (assoc-ref opts 'graft?))
                        (%guile-for-build
                         (and store-needed?
                              (package-derivation

base-commit: 9e71d4fd6b3893ae87cb079b57d7a8fe6e9e7914
-- 
2.41.0

bug#65225: ‘guix shell --system=ALIEN’ builds for both systems

[Tobias Geerinckx-Rice via Bug reports for GNU Guix]

Hi,

ekaitz noticed this when building a mes-derived package that fails on 
x86_64.  Here's a reproducer that uses upstream Guix:

$ guix shell --system=riscv64-linux drawterm --no-grafts --rebuild-cache 
--dry-run | grep drv$
   /gnu/store/4c02fgswkbldys93w4vgj0gwax2ly4bh-profile.drv

$ guix build --dry-run 
/gnu/store/4c02fgswkbldys93w4vgj0gwax2ly4bh-profile.drv | grep drawterm
   
/gnu/store/mkxyhjq117wdalc0gbz4468blqxih1kn-drawterm-20210628-1.c97fe46
   
/gnu/store/h8dk35aw2n8rj1hki6dkdrknfly26vq8-drawterm-20210628-1.c97fe46

/gnu/store/fl4iimlcdnlkarjm8m6z3392wss6b8yr-drawterm-20210628-1.c97fe46.drv
→ 
/gnu/store/mkxyhjq117wdalc0gbz4468blqxih1kn-drawterm-20210628-1.c97fe46
is riscv64-linux.

/gnu/store/9a1pji59hzacrmpb65nk3pp3m01niqf3-drawterm-20210628-1.c97fe46.drv
→ 
/gnu/store/h8dk35aw2n8rj1hki6dkdrknfly26vq8-drawterm-20210628-1.c97fe46
is x86_64-linux.

Eventually the ‘correct’ riscv64 drawterm ends up in the shell, but Guix 
should never have built the x86_64 version.

Kind regards,

T G-R

Sent from a Web browser.  Excuse or enjoy my brevity.

bug#65225: ‘guix shell --system=ALIEN’ builds for both systems

[Tobias Geerinckx-Rice via Bug reports for GNU Guix]

Oh, OK.

Parts of scripts/shell.scm (and perhaps environment.scm) assume that

$ guix build --system={x86_64,riscv64}-linux foo

is equivalent to

$ guix build --system=riscv64-linux foo

It's not.  This includes and causes things like commit 
9c513303156b418567b9d2cde9f8df66190051ac.  There's a lot of fast & loose 
assoc-reffing going on here that makes me nervous.

Anyway, I'll get me to a PC and then fix the code that's injecting a 
spurious extra ‘(system . (%current-system))’ that's causing these 
double builds.  Simple™.

Kind regards,

T G-R

Sent from a Web browser.  Excuse or enjoy my brevity.

bug#65225: ‘guix shell --system=ALIEN’ builds for both systems

[Tobias Geerinckx-Rice via Bug reports for GNU Guix]

Oh OK.

> the code that's injecting a spurious extra ‘(system . 
> (%current-system))’ that's causing these double builds

is not actually the problem.  Instead, the profile hooks are 
unconditionally built for the host's (%current-system).  Forcing them to 
match ‘--system=’ does the trick.  Simple™!

Kind regards,

T G-R

Sent from a Web browser.  Excuse or enjoy my brevity.

bug#65225: [PATCH] environment: Build the profile for the requested system.

[Josselin Poiret via Bug reports for GNU Guix]

[-- Attachment #1: Type: text/plain, Size: 5051 bytes --]

Hi Tobias,

Tobias Geerinckx-Rice <me@tobias.gr> writes:

> Previously, ‘--system=’ did not affect profile hooks, meaning that all
> packages would be built for both the host and requested systems.
>
> * guix/scripts/environment.scm (guix-environment*): Parameterize
> %current-system to match the requested ‘--system=’.
>
> Reported by ekaitz in #guix.
> ---
>  guix/scripts/environment.scm | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
>
> diff --git a/guix/scripts/environment.scm b/guix/scripts/environment.scm
> index 9712389842..27f7e53549 100644
> --- a/guix/scripts/environment.scm
> +++ b/guix/scripts/environment.scm
> @@ -1146,7 +1146,8 @@ (define (guix-environment* opts)
>            (warning (G_ "no packages specified; creating an empty environment~%")))
>  
>          ;; Use the bootstrap Guile when requested.
> -        (parameterize ((%graft? (assoc-ref opts 'graft?))
> +        (parameterize ((%current-system system)
> +                       (%graft? (assoc-ref opts 'graft?))
>                         (%guile-for-build
>                          (and store-needed?
>                               (package-derivation
>
> base-commit: 9e71d4fd6b3893ae87cb079b57d7a8fe6e9e7914
> -- 
> 2.41.0

So, I've looked into this deeper because this fix didn't seem satisfying
to me: it suggests that the implementation of profile-derivation itself
is wrong, and I wanted to fix it instead.  Here's what I uncovered:

%current-system (applies mutatis mutandis to %current-target-system)
is a Guile parameter.  That means that it is accessed through a function
call, and its values really depends on where that function call occurs.
Now, this interacts with the store monad in a cumbersome way: monadic
values in this case are functions (lambda (store) ...) returning two
values, the actual output and the store.  These functions are run only
at the run-with-store call.

Now, there are two non-equivalent ways of getting the current system in
a monadic context.  You can either do

(mlet ((system -> (%current-system))
  ...)

or

(mlet ((system (current-system))
  ...)

The former directly evaluates (%current-system), while the latter only
evaluates (%current-system) when the monadic value is run!

What does this mean for our case here?  Well, the problem lies with how
the hooks are lowered: they use (gexp->derivation ...) without the
optional #:system keyword.  This looks up the current system at call
time with the mlet -> construct, so everything should be okay, right?
Well, the hooks are run through a mapm/accumulate-builds call, which
puts everything in a monadic value, effectively delaying the look-up
until monadic run time.

--8<---------------cut here---------------start------------->8---
(mlet* %store-monad (...
                     (extras (if (null? (manifest-entries manifest))
                                 (return '())
                                 (mapm/accumulate-builds (lambda (hook)
                                                           (hook manifest))
                                                         hooks))))
                     ...)
--8<---------------cut here---------------end--------------->8---

At this point, I thought: “Well, I could just parameterize
%current-system inside the (lambda (hook) ...), and all would be well,
right?  Well, it didn't seem to work and I was pretty confused by it.  I
tested a bit and noticed that actually, contrary to what was intended
(there even is a comment in gexp-derivation about it), gexp-derivation
looks up the system at monadic run time!  It looks like this:

--8<---------------cut here---------------start------------->8---
(mlet* %store-monad ( ;; The following binding forces '%current-system' and
                     ;; '%current-target-system' to be looked up at >>=
                     ;; time.
                     (graft?    (set-grafting graft?))

                     (system -> (or system (%current-system)))
                     (target -> (if (eq? target 'current)
                                    (%current-target-system)
                                    target))
                     ...)
  ...)
--8<---------------cut here---------------end--------------->8---

Well, the issue here is that such an mlet starts by translating the
graft? binding into a >>= call, which ends up putting the rest of the
translation into a function call that will *not* be called until the
monadic value is run.  That means that the system and target bindings
afterwards are *not* looked up at call time but at monadic run time!
And sure enough, moving the (graft? ...) binding after the (target ->
...) one does solve this!

IMO, this is way too complicated to keep in mind at all times, and there
are bugs lurking under the surface absolutely everywhere, waiting for a
corner case to be uncovered.

I'll send a new patch once I've fixed and tested this further.

Best,
-- 
Josselin Poiret

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 682 bytes --]

bug#65225: [PATCH] environment: Build the profile for the requested system.

[Josselin Poiret via Bug reports for GNU Guix]

[-- Attachment #1: Type: text/plain, Size: 2582 bytes --]

Hi everyone,

Actually, let me add a big erratum, since it appears I misread the
intention of gexp->derivation (and realized 2 minutes after writing the
previous email).

Josselin Poiret <dev@jpoiret.xyz> writes:
> --8<---------------cut here---------------start------------->8---
> (mlet* %store-monad ( ;; The following binding forces '%current-system' and
>                      ;; '%current-target-system' to be looked up at >>=
>                      ;; time.
>                      (graft?    (set-grafting graft?))
>
>                      (system -> (or system (%current-system)))
>                      (target -> (if (eq? target 'current)
>                                     (%current-target-system)
>                                     target))
>                      ...)
>   ...)
> --8<---------------cut here---------------end--------------->8---
>
> Well, the issue here is that such an mlet starts by translating the
> graft? binding into a >>= call, which ends up putting the rest of the
> translation into a function call that will *not* be called until the
> monadic value is run.  That means that the system and target bindings
> afterwards are *not* looked up at call time but at monadic run time!

This is actually what the comment above hints at, I misunderstood its
meaning.  It seems that this piece of code used to be (before 2015)

--8<---------------cut here---------------start------------->8---
(mlet* %store-monad ( ;; The following binding forces '%current-system' and
                     ;; '%current-target-system' to be looked up at >>=
                     ;; time.
                     (unused    (return #f)

                     (system -> (or system (%current-system)))
                     (target -> (if (eq? target 'current)
                                    (%current-target-system)
                                    target))
                     ...)
  ...)
--8<---------------cut here---------------end--------------->8---

probably at a time when (current-system) didn't exist.  In turn, this
means that gexp->derivation intentionally delays getting the current
system to monadic run time.  Thus, we probably need to pass an optional
#:system argument to the hooks that they can forward to
gexp->derivation to fix this “properly”

> IMO, this is way too complicated to keep in mind at all times, and there
> are bugs lurking under the surface absolutely everywhere, waiting for a
> corner case to be uncovered.

My comment still stands.

Best,
-- 
Josselin Poiret

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 682 bytes --]

bug#65225: [PATCH] environment: Build the profile for the requested system.

[Ludovic Courtès]

Hi,

Josselin Poiret <dev@jpoiret.xyz> skribis:

> IMO, this is way too complicated to keep in mind at all times, and there
> are bugs lurking under the surface absolutely everywhere, waiting for a
> corner case to be uncovered.

Great writeup!  Yeah, that’s the sad situation of mixing parameters and
monads; it’s a longstanding issue and as you write, there have been
bugs.  The solution to these has been primarily to avoid relying on
default values and pass explicit values.  The other trick is what
‘lower-gexp’ does, with the comment you quoted, or what
‘gexp->derivation’ does.

The good news is that the monadic interface is kept internal
and not really exposed to users, who should stick to file-like objects
as much as possible.

Back to profile hooks: it seems that passing #:system #f to
‘gexp->derivation’ in each of these hooks would solve the problem, no?

Alternatively, each hook could take ‘system’ as a second argument.

Willing to give it a shot?

Thanks,
Ludo’.

bug#65225: ‘guix shell --system=ALIEN’ builds for both systems

[Maxim Cournoyer]

Hi,

Ludovic Courtès <ludo@gnu.org> writes:

> Hello Guix,
>
> This fixes <https://issues.guix.gnu.org/65225>, ensuring that profile
> hooks are built for the right system.  It does so by passing each
> profile hook the targeted system.
>
> Thoughts?
>
> Thanks,
> Ludo'.
>
> Ludovic Courtès (2):
>   packages: Add ‘system’ parameter for ‘set-guile-for-build’.
>   profiles: Hooks honor the #:system parameter of ‘profile-derivation’.
>
>  gnu/bootloader.scm |  5 +++--
>  guix/channels.scm  |  3 ++-
>  guix/packages.scm  |  7 ++++---
>  guix/profiles.scm  | 49 ++++++++++++++++++++++++++++++----------------
>  tests/profiles.scm | 24 ++++++++++++++++++++++-
>  5 files changed, 64 insertions(+), 24 deletions(-)

I see the cover letter, but not the patches :-).  Where did they go?

-- 
Thanks,
Maxim

bug#65225: ‘guix shell --system=ALIEN’ builds for both systems

[Simon Tournier]

Hi Maxim,

On Fri, 20 Oct 2023 at 18:46, Maxim Cournoyer <maxim.cournoyer@gmail.com> wrote:

> I see the cover letter, but not the patches :-).  Where did they go?

They are in #66640 [1].  The cover letter had been CC:
65225@debbugs.gnu.org, I guess.

1: https://issues.guix.gnu.org/issue/66640

Cheers,
simon

bug#65225: ‘guix shell --system=ALIEN’ builds for both systems

[Maxim Cournoyer]

Hi Simon,

Simon Tournier <zimon.toutoune@gmail.com> writes:

> Hi Maxim,
>
> On Fri, 20 Oct 2023 at 18:46, Maxim Cournoyer <maxim.cournoyer@gmail.com> wrote:
>
>> I see the cover letter, but not the patches :-).  Where did they go?
>
> They are in #66640 [1].  The cover letter had been CC:
> 65225@debbugs.gnu.org, I guess.

Thanks!

-- 
Maxim