Getting the Guix Build Coordinator agent working on the Hurd

Getting the Guix Build Coordinator agent working on the Hurd

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 1469 bytes --]

Hey,

This has been on my todo list ever since the childhurd service came in
to existence. I spent some more time looking at it yesterday, and did
manage to get the agent working in a childhurd VM, at least somewhat.

Firstly, guile-lib needed tweaking so that it actually cross compiles,
I've sent a patch [1].

1: https://lists.gnu.org/archive/html/guile-devel/2021-02/msg00004.html

Adding the agent service also required this tweak [2].

2: https://issues.guix.gnu.org/46506

Then I faced two problems with the guix-build-coordinator
package. Firstly, wrap-program picks bash for Linux for the wrapper
script, which isn't very useful. I hacked around this by setting the
PATH such that it picked bash for the Hurd. In terms of properly fixing
this, I guess that needs to somehow be able to find the right bash, I'm
not sure how though?

The second issue is that I'm not sure capturing the build time
GUILE_LOAD_COMPILED_PATH doesn't seem to work, at least file says that
the .go files this contains are built for a 64-bit architecture. I
worked around this by constructing the GUILE_LOAD_COMPILED_PATH from the
inputs I knew should be on it. Maybe it should always have been done
this way, any ideas?

There's also one problem probably within the Guix Build Coordinator
itself, after doing a few builds, it will just stop. I've only seen this
behaviour on the Hurd, but I'm unsure how to debug it, any suggestions?
My only idea is add more logging.

Thanks,

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: Getting the Guix Build Coordinator agent working on the Hurd

[Joshua Branson]

Christopher Baines <mail@cbaines.net> writes:

> Hey,
>
> This has been on my todo list ever since the childhurd service came in
> to existence. I spent some more time looking at it yesterday, and did
> manage to get the agent working in a childhurd VM, at least somewhat.

Firstly awesome!

> There's also one problem probably within the Guix Build Coordinator
> itself, after doing a few builds, it will just stop. I've only seen this
> behaviour on the Hurd, but I'm unsure how to debug it, any suggestions?
> My only idea is add more logging.

You could always email bug-hurd@gnu.org or hang out in the #hurd
channel.  Samuel is pretty quick to reply.  He's about as responsive as
Ludo!

--
Joshua Branson (joshuaBPMan in #guix)
Sent from Emacs and Gnus
  https://gnucode.me
  https://video.hardlimit.com/accounts/joshua_branson/video-channels
  https://propernaming.org
  "You can have whatever you want, as long as you help
enough other people get what they want." - Zig Ziglar

Re: Getting the Guix Build Coordinator agent working on the Hurd

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 2207 bytes --]


Christopher Baines <mail@cbaines.net> writes:

> This has been on my todo list ever since the childhurd service came in
> to existence. I spent some more time looking at it yesterday, and did
> manage to get the agent working in a childhurd VM, at least somewhat.
>
> Firstly, guile-lib needed tweaking so that it actually cross compiles,
> I've sent a patch [1].
>
> 1: https://lists.gnu.org/archive/html/guile-devel/2021-02/msg00004.html
>
> Adding the agent service also required this tweak [2].
>
> 2: https://issues.guix.gnu.org/46506
>
> Then I faced two problems with the guix-build-coordinator
> package. Firstly, wrap-program picks bash for Linux for the wrapper
> script, which isn't very useful. I hacked around this by setting the
> PATH such that it picked bash for the Hurd. In terms of properly fixing
> this, I guess that needs to somehow be able to find the right bash, I'm
> not sure how though?
>
> The second issue is that I'm not sure capturing the build time
> GUILE_LOAD_COMPILED_PATH doesn't seem to work, at least file says that
> the .go files this contains are built for a 64-bit architecture. I
> worked around this by constructing the GUILE_LOAD_COMPILED_PATH from the
> inputs I knew should be on it. Maybe it should always have been done
> this way, any ideas?

The above patches, hacks and questions are still relevant, but I've made
some progress on the issue below.

> There's also one problem probably within the Guix Build Coordinator
> itself, after doing a few builds, it will just stop. I've only seen this
> behaviour on the Hurd, but I'm unsure how to debug it, any suggestions?
> My only idea is add more logging.

I tried disabling a few of the optional features plus unnecessary locks
on the hurd [1], and with those changes I no longer see it just stop,
which is great, as then builds just keep happening :)

1: https://git.cbaines.net/guix/build-coordinator/commit/guix-build-coordinator?id=b3733bca21de607fd7a70319e66e3ff49996a974

For the guix.cbaines.net build farm, I've now got an additional machine
running two childhurd VM's, and I plan to scale this up to 7 or 8 in the
coming days to see how far it's possible to get building things for the
i586-gnu.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: Getting the Guix Build Coordinator agent working on the Hurd

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 1020 bytes --]


Joshua Branson <jbranso@dismail.de> writes:

> Christopher Baines <mail@cbaines.net> writes:
>
>> Hey,
>>
>> This has been on my todo list ever since the childhurd service came in
>> to existence. I spent some more time looking at it yesterday, and did
>> manage to get the agent working in a childhurd VM, at least somewhat.
>
> Firstly awesome!

Thanks!

>> There's also one problem probably within the Guix Build Coordinator
>> itself, after doing a few builds, it will just stop. I've only seen this
>> behaviour on the Hurd, but I'm unsure how to debug it, any suggestions?
>> My only idea is add more logging.
>
> You could always email bug-hurd@gnu.org or hang out in the #hurd
> channel.  Samuel is pretty quick to reply.  He's about as responsive as
> Ludo!

I disabled some features, and that seemed to help, so maybe I'll try to
come up with a small reproducer which hangs next [1].

1: https://git.cbaines.net/guix/build-coordinator/commit/guix-build-coordinator?id=b3733bca21de607fd7a70319e66e3ff49996a974

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: Getting the Guix Build Coordinator agent working on the Hurd

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 1483 bytes --]


Christopher Baines <mail@cbaines.net> writes:

> For the guix.cbaines.net build farm, I've now got an additional machine
> running two childhurd VM's, and I plan to scale this up to 7 or 8 in the
> coming days to see how far it's possible to get building things for the
> i586-gnu.

This has now happened, I've now got one machine running 8 childhurd VMs,
and another machine just running the one childhurd.

Builds for packages seem to be happening (see [1] and [2]), although I'm
suspicious some of the VMs are getting stuck sometimes, as I can't
always SSH in. There's also the GC not working issue, but both things
can be worked around by just restarting the VMs.

1: https://data.guix-patches.cbaines.net/repository/2/branch/master/latest-processed-revision/package-derivations?search_query=&system=i586-gnu&minimum_builds=&maximum_builds=&build_status=working&field=builds&after_name=&limit_results=&all_results=on
2: https://data.guix-patches.cbaines.net/repository/2/branch/master/latest-processed-revision/package-derivations?search_query=&system=i586-gnu&minimum_builds=&maximum_builds=&build_status=failing&field=builds&after_name=&limit_results=&all_results=on

It would be useful to be able to compare builds for packages across
different architectures, to find instances where a package builds, but
not on all architectures, as those could be useful to know about (either
to try and fix, or change to just support the working architectures in
the package definition).

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: Getting the Guix Build Coordinator agent working on the Hurd

[Ludovic Courtès]

Hi!

Christopher Baines <mail@cbaines.net> skribis:

> Then I faced two problems with the guix-build-coordinator
> package. Firstly, wrap-program picks bash for Linux for the wrapper
> script, which isn't very useful. I hacked around this by setting the
> PATH such that it picked bash for the Hurd. In terms of properly fixing
> this, I guess that needs to somehow be able to find the right bash, I'm
> not sure how though?

Looks like a bug in ‘wrap-program’ that we should fix in ‘core-updates’.
‘wrap-program’ uses (which "bash"), which is wrong in a
cross-compilation context.

We should at least add a #:bash parameter to ‘wrap-program’, but then
all callers will have to pass it.  I’m not sure how to let it do the
right thing by default.

> The second issue is that I'm not sure capturing the build time
> GUILE_LOAD_COMPILED_PATH doesn't seem to work, at least file says that
> the .go files this contains are built for a 64-bit architecture. I
> worked around this by constructing the GUILE_LOAD_COMPILED_PATH from the
> inputs I knew should be on it. Maybe it should always have been done
> this way, any ideas?

Instead of capturing the build-time ‘GUILE_LOAD_COMPILED_PATH’, which
doesn’t contain the target .go files, you should explicitly list the
inputs as is done in the ‘guix’ package for example.  That’ll ensure the
binary refers to the cross-compiled .go files.

> There's also one problem probably within the Guix Build Coordinator
> itself, after doing a few builds, it will just stop. I've only seen this
> behaviour on the Hurd, but I'm unsure how to debug it, any suggestions?
> My only idea is add more logging.

No idea, but I guess that could just be a crash.  Can you still log in
afterwards?

BTW, note that builds on GNU/Hurd are currently not isolated, and thus
it’s the wild west in terms of reproducibility:

  https://issues.guix.gnu.org/43857

There are open questions as to what to include in the build environment:

  https://guix.gnu.org/en/blog/2020/childhurds-and-substitutes/

Ludo’.

Re: Getting the Guix Build Coordinator agent working on the Hurd

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 2005 bytes --]


Ludovic Courtès <ludo@gnu.org> writes:

>> The second issue is that I'm not sure capturing the build time
>> GUILE_LOAD_COMPILED_PATH doesn't seem to work, at least file says that
>> the .go files this contains are built for a 64-bit architecture. I
>> worked around this by constructing the GUILE_LOAD_COMPILED_PATH from the
>> inputs I knew should be on it. Maybe it should always have been done
>> this way, any ideas?
>
> Instead of capturing the build-time ‘GUILE_LOAD_COMPILED_PATH’, which
> doesn’t contain the target .go files, you should explicitly list the
> inputs as is done in the ‘guix’ package for example.  That’ll ensure the
> binary refers to the cross-compiled .go files.

This has now happened [1], so the guix-build-coordinator package should
now cross compile in a useable way.

1: https://git.savannah.gnu.org/cgit/guix.git/commit/?id=eec127822a74c6a1a6100b07d94c8fb275d571bf

>> There's also one problem probably within the Guix Build Coordinator
>> itself, after doing a few builds, it will just stop. I've only seen this
>> behaviour on the Hurd, but I'm unsure how to debug it, any suggestions?
>> My only idea is add more logging.
>
> No idea, but I guess that could just be a crash.  Can you still log in
> afterwards?

Not through SSH at least.

Someone on IRC mentioned they'd had issues running Hurd VMs without
swap, so getting swap in childhurds might be something to try. I started
looking at this [2].

2: https://issues.guix.gnu.org/46726

> BTW, note that builds on GNU/Hurd are currently not isolated, and thus
> it’s the wild west in terms of reproducibility:
>
>   https://issues.guix.gnu.org/43857
>
> There are open questions as to what to include in the build environment:
>
>   https://guix.gnu.org/en/blog/2020/childhurds-and-substitutes/

Isolation would be nice of course, although I'm not sure how much this
will affect reproducibility, unless things are poking around in the
store directly.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: Getting the Guix Build Coordinator agent working on the Hurd

[Ludovic Courtès]

Hi,

Christopher Baines <mail@cbaines.net> skribis:

>> There are open questions as to what to include in the build environment:
>>
>>   https://guix.gnu.org/en/blog/2020/childhurds-and-substitutes/
>
> Isolation would be nice of course, although I'm not sure how much this
> will affect reproducibility, unless things are poking around in the
> store directly.

Lack of isolation is an immediate problem.  I was hopeful until we
stumbled upon an util-linux test failure; whether it fails depends on
what’s in the environment.

So I think isolation is not superfluous, which is annoying because it
means we pretty much have to address that before we can profit.

Ludo’.