Not sure how to add an offload server

Not sure how to add an offload server

[Simen Endsjø]

Hi, I have some problems creating my own offload server.

Adding a build-machine section in guix-configuration fills
/etc/guix/machines.scm.

But when I do `guix offload test` or status, it checks
/usr/local/etc/guix/machines.scm and not /etc/guix/machinecs.scm.

Adding the build machine in that line works, but then it doesn't look like it
checks /etc/guix/acl. The key is listed there, but `guix offload test` still
complains that the key isn't trusted.

(guix-service-type config => (guix-configuration
                                                    (inherit config)
                                                    (authorized-keys
                                                     (append (list (plain-file "simendsjo-ser7"
                                                                               "(public-key (ecc (curve Ed25519) (q #65EB2980FCA088C7D2C4200DAB417ABB96424ECC5EE65F3BD7F7B1F7D4FD093E#)))"))
                                                             %default-authorized-guix-keys))
                                                    (build-machines
                                                     (list '(build-machine
                                                            ;; simendsjo-ser7
                                                            (name "192.168.0.5")
                                                            (port 52571)
                                                            (user "simendsjo")
                                                            (systems (list "x86_64-linux"))
                                                            (host-key "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMjlkRdkV7AXzKpQyNY/0g7SKaECxC6JxcinzchDqQt5 root@(none)")
                                                            (private-key "/home/simendsjo/.ssh/id_rsa")
                                                            (speed 2)))))))
                                                            
$ guix offload test
guix offload: testing 1 build machines defined in '/usr/local/etc/guix/machines.scm'...
guix offload: Guix is usable on '192.168.0.5' (test returned "/gnu/store/883yjkl46dxw9mzykykmbs0yzwyxm17z-test")
guix offload: '192.168.0.5' is running GNU Guile 3.0.9
guix offload: sending 1 store item (0 MiB) to '192.168.0.5'...
exporting path `/gnu/store/pf0w80gakb3bj28xvcvyajcnwgxbnkvj-export-test'
guix offload: error: unauthorized public key: (public-key
 (ecc
  (curve Ed25519)
  (q #65EB2980FCA088C7D2C4200DAB417ABB96424ECC5EE65F3BD7F7B1F7D4FD093E#)
  )
 )

Re: Not sure how to add an offload server

[Tomas Volf]

[-- Attachment #1: Type: text/plain, Size: 507 bytes --]

Simen Endsjø <contact@simendsjo.me> writes:

> Hi, I have some problems creating my own offload server.
>
> Adding a build-machine section in guix-configuration fills
> /etc/guix/machines.scm.
>
> But when I do `guix offload test` or status, it checks
> /usr/local/etc/guix/machines.scm and not /etc/guix/machinecs.scm.

This seems weird.  How did you install your Guix?

T.

-- 
There are only two hard things in Computer Science:
cache invalidation, naming things and off-by-one errors.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 853 bytes --]

Re: Not sure how to add an offload server

[Simen Endsjø]

Both computers use Guix System. Nothing in my configuration tweaks anything related to this.

On Sun, Nov 3, 2024, at 15:06, Tomas Volf wrote:
> Simen Endsjø <contact@simendsjo.me> writes:
> 
> > Hi, I have some problems creating my own offload server.
> >
> > Adding a build-machine section in guix-configuration fills
> > /etc/guix/machines.scm.
> >
> > But when I do `guix offload test` or status, it checks
> > /usr/local/etc/guix/machines.scm and not /etc/guix/machinecs.scm.
> 
> This seems weird.  How did you install your Guix?
> 
> T.
> 
> -- 
> There are only two hard things in Computer Science:
> cache invalidation, naming things and off-by-one errors.
> 
> 
> *Attachments:*
>  • signature.asc

Re: Not sure how to add an offload server

[Simen Endsjø]

I don't understand how /etc/guix and /usr/local/etc/guix fits together.
/usr/local/etc/guix/acl does not contain the public key, but it exists and symlinks to a store item.

As guix offload looks at /usr/local/etc/guix/machines.scm, I manually added the machines.scm there, and `guix offload test /usr/local/etc/guix/machines.scm` fails with the missing public key.

$ ls -lah /usr/local/etc/guix
total 24K
drwxr-xr-x 2 root root 4.0K Nov  6 22:02 ./
drwxr-xr-x 3 root root 4.0K May  1  2023 ../
lrwxrwxrwx 1 root root   47 Nov  6 22:02 acl -> /gnu/store/y0xvx5900zxj4s7rwyfr3baazkkhzkbw-acl
-rw------- 1 root root  701 Nov  2 22:00 acl.bak
-rw-r--r-- 1 root root  342 Nov  2 22:48 machines.scm
-r--r--r-- 1 root root  118 Oct 23  2023 signing-key.pub
-r-------- 1 root root  192 Oct 23  2023 signing-key.sec

/etc/guix on the other hand has an acl which is not symlinked, but is using the machines.scm from my system.
This acl contains the public key.
But `guix offload test /etc/guix/machines.scm` also fails on the missing public key.

$ sudo ls /etc/guix -lah
total 24K
dr-x--x--x 2 root root 4.0K Nov  6 22:02 .
drwxr-xr-x 7 root root 4.0K Nov  6 22:02 ..
-rw------- 1 root root  498 Jul 24  2023 acl
-rw------- 1 root root  182 Jul 21  2023 acl.backup
lrwxrwxrwx 1 root root   56 Jun 26 21:02 channels.scm -> /gnu/store/8nyjma3zafvhv4a31561n6ckhr9d3f90-channels.scm
lrwxrwxrwx 1 root root   56 Nov  6 22:02 machines.scm -> /gnu/store/kj5lfjhxi81x7xp3nym3ssdyp2177waq-machines.scm
-r--r--r-- 1 root root  118 Oct 14  2022 signing-key.pub
-r-------- 1 root root  192 Oct 14  2022 signing-key.sec

I really have no idea what is happening here, but I fail to see what I'm doing wrong.

Could someone link me to a working offload setup?

On Sun, Nov 3, 2024, at 16:05, Simen Endsjø wrote:
> Both computers use Guix System. Nothing in my configuration tweaks anything related to this.
> 
> On Sun, Nov 3, 2024, at 15:06, Tomas Volf wrote:
>> Simen Endsjø <contact@simendsjo.me> writes:
>> 
>> > Hi, I have some problems creating my own offload server.
>> >
>> > Adding a build-machine section in guix-configuration fills
>> > /etc/guix/machines.scm.
>> >
>> > But when I do `guix offload test` or status, it checks
>> > /usr/local/etc/guix/machines.scm and not /etc/guix/machinecs.scm.
>> 
>> This seems weird.  How did you install your Guix?
>> 
>> T.
>> 
>> -- 
>> There are only two hard things in Computer Science:
>> cache invalidation, naming things and off-by-one errors.
>> 
>> 
>> *Attachments:*
>>  • signature.asc

Re: Not sure how to add an offload server

[Tomas Volf]

[-- Attachment #1: Type: text/plain, Size: 1496 bytes --]

Simen Endsjø <contact@simendsjo.me> writes:

> I don't understand how /etc/guix and /usr/local/etc/guix fits together.
> /usr/local/etc/guix/acl does not contain the public key, but it exists
> and symlinks to a store item.

They should not fit together, /usr/local does not exist at all on my
GuixSD system.

> Could someone link me to a working offload setup?

These two blocks are all I need.

--8<---------------cut here---------------start------------->8---
(define %build-machines
  (list #~(build-machine
           (name "some-server")
           (systems (list "x86_64-linux" "i686-linux" "aarch64-linux"))
           (host-key "some-host-key")
           (user "build-offload")
           (parallel-builds 4)
           (overload-threshold #f)
           (private-key "/some/key"))))
--8<---------------cut here---------------end--------------->8---

--8<---------------cut here---------------start------------->8---
              (simple-service 'offload guix-service-type
                              (guix-extension
                               (build-machines %build-machines)))
--8<---------------cut here---------------end--------------->8---

(Well, I also needed to authorized the keys using `guix archive', as the
manual instructs.  And create the `build-offload' user on the server
with allowed login by ssh key.)

T.

-- 
There are only two hard things in Computer Science:
cache invalidation, naming things and off-by-one errors.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 853 bytes --]

Re: Not sure how to add an offload server

[Simen Endsjø]

Strange! I found out the error was passed by the offload server, so it was
complaining that it didn't trust my laptop! I added my laptop to authorized
keys on the server, and then it works! `guix archive --authorize` complains that
it should be set in the system configuration for Guix SD though, so I did that.

But /usr/local is a mystery. I tried deleting it, but a reconfigure added back
/usr/local/guix/acl. And `guix offload` commands still uses (the now missing
file) /etc/local/guix/machines.scm by default.

A difference between our configurations is that I'm modifying build-machines in
the original guix-configuration using modify-services. But that really shouldn't
make any difference, should it?


On Thu, Nov 7, 2024, at 00:49, Tomas Volf wrote:
> Simen Endsjø <contact@simendsjo.me> writes:
> 
> > I don't understand how /etc/guix and /usr/local/etc/guix fits together.
> > /usr/local/etc/guix/acl does not contain the public key, but it exists
> > and symlinks to a store item.
> 
> They should not fit together, /usr/local does not exist at all on my
> GuixSD system.
> 
> > Could someone link me to a working offload setup?
> 
> These two blocks are all I need.
> 
> --8<---------------cut here---------------start------------->8---
> (define %build-machines
>   (list #~(build-machine
>            (name "some-server")
>            (systems (list "x86_64-linux" "i686-linux" "aarch64-linux"))
>            (host-key "some-host-key")
>            (user "build-offload")
>            (parallel-builds 4)
>            (overload-threshold #f)
>            (private-key "/some/key"))))
> --8<---------------cut here---------------end--------------->8---
> 
> --8<---------------cut here---------------start------------->8---
>               (simple-service 'offload guix-service-type
>                               (guix-extension
>                                (build-machines %build-machines)))
> --8<---------------cut here---------------end--------------->8---
> 
> (Well, I also needed to authorized the keys using `guix archive', as the
> manual instructs.  And create the `build-offload' user on the server
> with allowed login by ssh key.)
> 
> T.
> 
> -- 
> There are only two hard things in Computer Science:
> cache invalidation, naming things and off-by-one errors.
> 
> 
> *Attachments:*
>  • signature.asc

Re: Not sure how to add an offload server

[Hilton Chain]

Hi Simen,

On Thu, 07 Nov 2024 16:03:46 +0800,
Simen Endsjø wrote:
>
> Strange! I found out the error was passed by the offload server, so it was
> complaining that it didn't trust my laptop! I added my laptop to authorized
> keys on the server, and then it works! `guix archive --authorize` complains that
> it should be set in the system configuration for Guix SD though, so I did that.
>
> But /usr/local is a mystery. I tried deleting it, but a reconfigure added back
> /usr/local/guix/acl. And `guix offload` commands still uses (the now missing
> file) /etc/local/guix/machines.scm by default.
>
> A difference between our configurations is that I'm modifying build-machines in
> the original guix-configuration using modify-services. But that really shouldn't
> make any difference, should it?


What's the result of the following command?
--8<---------------cut here---------------start------------->8---
grep --context=3 \
	%sysconfdir \
	"$(dirname $(realpath /run/current-system/profile/bin/guix-daemon))/../share/guile/site/3.0/guix/config.scm"
--8<---------------cut here---------------end--------------->8---


For me, it's:
--8<---------------cut here---------------start------------->8---
(define %localstatedir
  "/var")

(define %sysconfdir
  "/etc")
--8<---------------cut here---------------end--------------->8---


Can you build Guix from git[1], reconfigure your system with it[2], then run
above command again to see if the result changes?

If it works, don't forget to move files in /usr/local to their new paths and
restart guix-daemon.  Then try to pull once with the new /run/current-system/profile/bin/guix.


Thanks

[1]: https://guix.gnu.org/manual/devel/en/html_node/Building-from-Git.html
[2]: https://guix.gnu.org/manual/devel/en/html_node/Running-Guix-Before-It-Is-Installed.html

Re: Not sure how to add an offload server

[Simen Endsjø]

My config doesn't show any trace of /usr/local:

    $ grep --context=3 \
            %sysconfdir \
            "$(dirname $(realpath /run/current-system/profile/bin/guix-daemon))/../share/guile/site/3.0/guix/config.scm"
    
                %storedir
                %localstatedir
                %sysconfdir
    
                %store-directory
                %state-directory
    --
    (define %localstatedir
      "/var")
    
    (define %sysconfdir
      "/etc")
    
    (define %store-directory
    --
    (define %config-directory
      ;; This must match `GUIX_CONFIGURATION_DIRECTORY' as defined in `nix/local.mk'.
      (or (getenv "GUIX_CONFIGURATION_DIRECTORY")
          (string-append %sysconfdir "/guix")))
    
    (define %system
      "x86_64-linux")

      
I'm unable to rebuild the system using this method as it looks like my channels
is not loaded, giving me
        no code for module (sijo packages foobar)


On Thu, Nov 7, 2024, at 10:06, Hilton Chain wrote:
> Hi Simen,
> 
> On Thu, 07 Nov 2024 16:03:46 +0800,
> Simen Endsjø wrote:
> >
> > Strange! I found out the error was passed by the offload server, so it was
> > complaining that it didn't trust my laptop! I added my laptop to authorized
> > keys on the server, and then it works! `guix archive --authorize` complains that
> > it should be set in the system configuration for Guix SD though, so I did that.
> >
> > But /usr/local is a mystery. I tried deleting it, but a reconfigure added back
> > /usr/local/guix/acl. And `guix offload` commands still uses (the now missing
> > file) /etc/local/guix/machines.scm by default.
> >
> > A difference between our configurations is that I'm modifying build-machines in
> > the original guix-configuration using modify-services. But that really shouldn't
> > make any difference, should it?
> 
> 
> What's the result of the following command?
> --8<---------------cut here---------------start------------->8---
> grep --context=3 \
> %sysconfdir \
> "$(dirname $(realpath /run/current-system/profile/bin/guix-daemon))/../share/guile/site/3.0/guix/config.scm"
> --8<---------------cut here---------------end--------------->8---
> 
> 
> For me, it's:
> --8<---------------cut here---------------start------------->8---
> (define %localstatedir
>   "/var")
> 
> (define %sysconfdir
>   "/etc")
> --8<---------------cut here---------------end--------------->8---
> 
> 
> Can you build Guix from git[1], reconfigure your system with it[2], then run
> above command again to see if the result changes?
> 
> If it works, don't forget to move files in /usr/local to their new paths and
> restart guix-daemon.  Then try to pull once with the new /run/current-system/profile/bin/guix.
> 
> 
> Thanks
> 
> [1]: https://guix.gnu.org/manual/devel/en/html_node/Building-from-Git.html
> [2]: https://guix.gnu.org/manual/devel/en/html_node/Running-Guix-Before-It-Is-Installed.html
> 

Re: Not sure how to add an offload server

[Felix Lechner via]

Hi Simen,

On Thu, Nov 07 2024, Simen Endsjø wrote:

> I'm unable to rebuild the system using this method as it looks like my
> channels is not loaded

Sorry about your troubles.  Guix will accept -L to include local
folders, which can hold a cloned Git repository.  It's not convenient,
but will work at least once.

Kind regards
Felix

Re: Not sure how to add an offload server

[Simen Endsjø]

Not sure what you mean. Tried adding the channel with -L, but I get an error about a record ABI mismatch. How is this possible? Is somehow two different guix channels used...?

$ ./pre-inst-env guix system \
  -L ~/the-locally-cloned-required-channel \
  reconfigure the-system.scm

guix system: error: failed to load 'the-system.scm':
guix/records.scm:65:4: In procedure record-abi-mismatch-error:
In procedure abi-check: #<record-type <guix-configuration>>: record ABI mismatch; recompilation needed


On Thu, Nov 7, 2024, at 15:35, Felix Lechner wrote:
> Hi Simen,
> 
> On Thu, Nov 07 2024, Simen Endsjø wrote:
> 
> > I'm unable to rebuild the system using this method as it looks like my
> > channels is not loaded
> 
> Sorry about your troubles.  Guix will accept -L to include local
> folders, which can hold a cloned Git repository.  It's not convenient,
> but will work at least once.
> 
> Kind regards
> Felix
> 

Re: Not sure how to add an offload server

[Simen Endsjø]

Ok, so now it suddenly works. I cannot see any recent commits in guix which should indicate a change related to this.
What kind of bad state could I have had on my system which has been fixed by experimenting with the local guix and pre-inst-env? And is it permanently fixed now? I find this both strange and worrying.

$ guix offload test
guix offload: testing 1 build machines defined in '/etc/guix/machines.scm'...

On Thu, Nov 7, 2024, at 22:17, Simen Endsjø wrote:
> Not sure what you mean. Tried adding the channel with -L, but I get an error about a record ABI mismatch. How is this possible? Is somehow two different guix channels used...?
> 
> $ ./pre-inst-env guix system \
>   -L ~/the-locally-cloned-required-channel \
>   reconfigure the-system.scm
> 
> guix system: error: failed to load 'the-system.scm':
> guix/records.scm:65:4: In procedure record-abi-mismatch-error:
> In procedure abi-check: #<record-type <guix-configuration>>: record ABI mismatch; recompilation needed
> 
> 
> On Thu, Nov 7, 2024, at 15:35, Felix Lechner wrote:
>> Hi Simen,
>> 
>> On Thu, Nov 07 2024, Simen Endsjø wrote:
>> 
>> > I'm unable to rebuild the system using this method as it looks like my
>> > channels is not loaded
>> 
>> Sorry about your troubles.  Guix will accept -L to include local
>> folders, which can hold a cloned Git repository.  It's not convenient,
>> but will work at least once.
>> 
>> Kind regards
>> Felix
>> 
> 

Re: Not sure how to add an offload server

[Simen Endsjø]

I also see the following commented out lines in the offload script:
      ;; Avoid ABI incompatibility with the <build-machine> record.
      ;; (set! %fresh-auto-compile #t)

And this from load* in ui.scm:
      ;; XXX: Force a recompilation to avoid ABI issues.
      (set! %fresh-auto-compile #t)

So my ABI mismatch issue might be triggering a similar issue and we need a %fresh-auto-compile somewhere.

On Thu, Nov 7, 2024, at 22:51, Simen Endsjø wrote:
> Ok, so now it suddenly works. I cannot see any recent commits in guix which should indicate a change related to this.
> What kind of bad state could I have had on my system which has been fixed by experimenting with the local guix and pre-inst-env? And is it permanently fixed now? I find this both strange and worrying.
> 
> $ guix offload test
> guix offload: testing 1 build machines defined in '/etc/guix/machines.scm'...
> 
> On Thu, Nov 7, 2024, at 22:17, Simen Endsjø wrote:
>> Not sure what you mean. Tried adding the channel with -L, but I get an error about a record ABI mismatch. How is this possible? Is somehow two different guix channels used...?
>> 
>> $ ./pre-inst-env guix system \
>>   -L ~/the-locally-cloned-required-channel \
>>   reconfigure the-system.scm
>> 
>> guix system: error: failed to load 'the-system.scm':
>> guix/records.scm:65:4: In procedure record-abi-mismatch-error:
>> In procedure abi-check: #<record-type <guix-configuration>>: record ABI mismatch; recompilation needed
>> 
>> 
>> On Thu, Nov 7, 2024, at 15:35, Felix Lechner wrote:
>>> Hi Simen,
>>> 
>>> On Thu, Nov 07 2024, Simen Endsjø wrote:
>>> 
>>> > I'm unable to rebuild the system using this method as it looks like my
>>> > channels is not loaded
>>> 
>>> Sorry about your troubles.  Guix will accept -L to include local
>>> folders, which can hold a cloned Git repository.  It's not convenient,
>>> but will work at least once.
>>> 
>>> Kind regards
>>> Felix
>>> 
>> 
>