bug#40125: Problem with guix offload: Remote channel closed

bug#40125: Problem with guix offload: Remote channel closed

[Mikael Djurfeldt]

[-- Attachment #1: Type: text/plain, Size: 1506 bytes --]

Hi,

I just installed guix on top of Debian Buster using the installation script
at guix.gnu.org. I did this on two machines, hat and wand, with the hope
that I could offload compilation to wand.

This is what I get:

root@hat:~# guix offload test
guile: warning: failed to install locale
hint: Consider installing the `glibc-utf8-locales' or `glibc-locales'
package and
defining `GUIX_LOCPATH', along these lines:

     guix package -i glibc-utf8-locales
     export GUIX_LOCPATH="$HOME/.guix-profile/lib/locale"

See the "Application Setup" section in the manual, for more info.

guix offload: testing 1 build machines defined in
'/etc/guix/machines.scm'...
guix offload: Guix is usable on 'wand.pdc.kth.se' (test returned
"/gnu/store/883yjkl46dxw9mzykykmbs0yzwyxm17z-test")
guix offload: 'wand.pdc.kth.se' is running GNU Guile 3.0.1
sending 1 store item (0 MiB) to 'wand.pdc.kth.se'...
exporting path `/gnu/store/sd0wqvaffi1cbpvf0dq37mab34rmlnav-export-test'
;;; [2020/03/17 12:45:48.671038, 0] write_to_channel_port: [GSSH ERROR]
Remote channel is closed: #<input-output: channel (open) 7f2c1c9831e0>
Backtrace:
           1 (primitive-load "/root/.config/guix/current/bin/guix")
In guix/ui.scm:
  1833:12  0 (run-guix-command _ . _)

guix/ui.scm:1833:12: In procedure run-guix-command:
Throw to key `guile-ssh-error' with args `("write_to_channel_port" "Remote
channel is closed" #<input-output: channel (open) 7f2c1c9831e0> #f)'.

Any hints about what could be wrong?

Best regards,
Mikael Djurfeldt

[-- Attachment #2: Type: text/html, Size: 2152 bytes --]

bug#40125: Problem with guix offload: Remote channel closed

[Mikael Djurfeldt]

[-- Attachment #1: Type: text/plain, Size: 994 bytes --]

This is a more up-to-date log where a problem in my environment (both on
local and build host) was fixed:

mdj@hat:~$ guix offload test
guix offload: testing 1 build machines defined in
'/etc/guix/machines.scm'...
guix offload: Guix is usable on 'wand.pdc.kth.se' (test returned
"/gnu/store/883yjkl46dxw9mzykykmbs0yzwyxm17z-test")
guix offload: 'wand.pdc.kth.se' is running GNU Guile 3.0.1
sending 1 store item (0 MiB) to 'wand.pdc.kth.se'...
exporting path `/gnu/store/gl7dps3yx0vrxz16sj7q3w7gs3vdbxj4-export-test'
;;; [2020/03/18 20:53:04.930901, 0] write_to_channel_port: [GSSH ERROR]
Remote channel is closed: #<input-output: channel (open) 7f5ed680e780>
Backtrace:
           1 (primitive-load "/home/mdj/.config/guix/current/bin/guix")
In guix/ui.scm:
  1833:12  0 (run-guix-command _ . _)

guix/ui.scm:1833:12: In procedure run-guix-command:
Throw to key `guile-ssh-error' with args `("write_to_channel_port" "Remote
channel is closed" #<input-output: channel (open) 7f5ed680e780> #f)'.

[-- Attachment #2: Type: text/html, Size: 1320 bytes --]

bug#40125: Problem with guix offload: Remote channel closed

[Ludovic Courtès]

Hi Mikael!

Good to see you here.  :-)

Mikael Djurfeldt <mikael@djurfeldt.com> skribis:

> mdj@hat:~$ guix offload test
> guix offload: testing 1 build machines defined in
> '/etc/guix/machines.scm'...
> guix offload: Guix is usable on 'wand.pdc.kth.se' (test returned
> "/gnu/store/883yjkl46dxw9mzykykmbs0yzwyxm17z-test")
> guix offload: 'wand.pdc.kth.se' is running GNU Guile 3.0.1
> sending 1 store item (0 MiB) to 'wand.pdc.kth.se'...
> exporting path `/gnu/store/gl7dps3yx0vrxz16sj7q3w7gs3vdbxj4-export-test'
> ;;; [2020/03/18 20:53:04.930901, 0] write_to_channel_port: [GSSH ERROR]
> Remote channel is closed: #<input-output: channel (open) 7f5ed680e780>
> Backtrace:
>            1 (primitive-load "/home/mdj/.config/guix/current/bin/guix")
> In guix/ui.scm:
>   1833:12  0 (run-guix-command _ . _)
>
> guix/ui.scm:1833:12: In procedure run-guix-command:
> Throw to key `guile-ssh-error' with args `("write_to_channel_port" "Remote
> channel is closed" #<input-output: channel (open) 7f5ed680e780> #f)'.

Did you generate a signing key on wand.pdc.kth.se?  Specifically with:

  sudo guix archive --generate-key

(See
<https://guix.gnu.org/manual/devel/en/html_node/Invoking-guix-archive.html>.)

Error reporting in guix offload is currently suboptimal as you have
seen.  There’s a couple of bug reports open on that topic, so hopefully
we’ll get there soon!

Ludo’.

bug#40125: Problem with guix offload: Remote channel closed

[Mikael Djurfeldt]

[-- Attachment #1: Type: text/plain, Size: 1922 bytes --]

Hi Ludovic! :-)

I did generate the signing key on wand. (But is this sensitive to what user
I used? O might have done it as root rather than as the ssh user used to
login on wand.)

I will look at the other buh reports. Meanwhile, do you have any hint on
where to look/insert logging code in order to see what is going on?

Best regards,
Mikael

Den lör 21 mars 2020 16:50Ludovic Courtès <ludo@gnu.org> skrev:

> Hi Mikael!
>
> Good to see you here.  :-)
>
> Mikael Djurfeldt <mikael@djurfeldt.com> skribis:
>
> > mdj@hat:~$ guix offload test
> > guix offload: testing 1 build machines defined in
> > '/etc/guix/machines.scm'...
> > guix offload: Guix is usable on 'wand.pdc.kth.se' (test returned
> > "/gnu/store/883yjkl46dxw9mzykykmbs0yzwyxm17z-test")
> > guix offload: 'wand.pdc.kth.se' is running GNU Guile 3.0.1
> > sending 1 store item (0 MiB) to 'wand.pdc.kth.se'...
> > exporting path `/gnu/store/gl7dps3yx0vrxz16sj7q3w7gs3vdbxj4-export-test'
> > ;;; [2020/03/18 20:53:04.930901, 0] write_to_channel_port: [GSSH ERROR]
> > Remote channel is closed: #<input-output: channel (open) 7f5ed680e780>
> > Backtrace:
> >            1 (primitive-load "/home/mdj/.config/guix/current/bin/guix")
> > In guix/ui.scm:
> >   1833:12  0 (run-guix-command _ . _)
> >
> > guix/ui.scm:1833:12: In procedure run-guix-command:
> > Throw to key `guile-ssh-error' with args `("write_to_channel_port"
> "Remote
> > channel is closed" #<input-output: channel (open) 7f5ed680e780> #f)'.
>
> Did you generate a signing key on wand.pdc.kth.se?  Specifically with:
>
>   sudo guix archive --generate-key
>
> (See
> <https://guix.gnu.org/manual/devel/en/html_node/Invoking-guix-archive.html
> >.)
>
> Error reporting in guix offload is currently suboptimal as you have
> seen.  There’s a couple of bug reports open on that topic, so hopefully
> we’ll get there soon!
>
> Ludo’.
>

[-- Attachment #2: Type: text/html, Size: 3099 bytes --]

bug#40125: Problem with guix offload: Remote channel closed

[Mikael Djurfeldt]

[-- Attachment #1: Type: text/plain, Size: 379 bytes --]

On Sat, Mar 21, 2020 at 5:05 PM Mikael Djurfeldt <mikael@djurfeldt.com>
wrote:

> Hi Ludovic! :-)
>
> I did generate the signing key on wand. (But is this sensitive to what
> user I used? O might have done it as root rather than as the ssh user used
> to login on wand.)
>

Sorry, I read your email on a cell phone and missed the "sudo" in front of
the command. Yes, I did that.

[-- Attachment #2: Type: text/html, Size: 732 bytes --]

bug#40125: Problem with guix offload: Remote channel closed

[Ludovic Courtès]

Hi Mikael,

Mikael Djurfeldt <mikael@djurfeldt.com> skribis:

> I will look at the other buh reports. Meanwhile, do you have any hint on
> where to look/insert logging code in order to see what is going on?

On ‘wand’, can you run something like:

  sudo strace -s 500 -p $(pidof sshd) -f -o sshd.log

After that, run ‘guix offload test’ from the other host, and check what
‘sshd.log’ contains.  If would in particular look for all the ‘execve’
syscalls as well as what happens right before ‘exit_group’ syscalls.
There are probably error messages lurking there.  :-)

You can share excerpts of the log but note that it will contain
sensitive crypto material, so beware.

Thanks in advance,
Ludo’.

bug#40125: Problem with guix offload: Remote channel closed

[Mikael Djurfeldt]

[-- Attachment #1: Type: text/plain, Size: 2703 bytes --]

I did this.

What is a bit strange is that the error message seemingly randomly alters
between what I already reported above and the following:

mdj@hat:~/prusa$ guix offload test
guix offload: testing 1 build machines defined in
'/etc/guix/machines.scm'...
guix offload: Guix is usable on 'wand.pdc.kth.se' (test returned
"/gnu/store/883yjkl46dxw9mzykykmbs0yzwyxm17z-test")
guix offload: 'wand.pdc.kth.se' is running GNU Guile 3.0.1
sending 1 store item (0 MiB) to 'wand.pdc.kth.se'...
exporting path `/gnu/store/kw12ny565xl5p05ag6ysgshc9mhh4rz2-export-test'
guix offload: error: unknown error while sending files over SSH

This error message is more common. In this case, I couldn't see anything
obviously strange except the following backtrace which contains strange
characters:

31963 write(2, "Backtrace:\n          13 (apply-smob/1 #<catch-closure
56464c2232a0>)\nIn ice-9/boot-9.scm:\n    718:2 12 (call-with-prompt
(\"prompt\") #<procedure 56464c2407e0 \342\200\246> \342\200\246)\nIn
ice-9/eval.scm:\n    619:8 11 (_ #(#(#<directory (guile-user)
56464c2c4750>)))\nIn ice-9/command-line.scm:\n   181:18 10 (_ #<input:
string 56464c2d2bd0>)\nIn unknown file:\n           9 (eval (begin
(use-modules (guix) (srfi srfi-34) # #) \342\200\246) #)\nIn
ice-9/eval.scm:\n   721:20  8 (primitive-eval (begin (use-modules (guix)
(srfi \342\200\246) \342"..., 1150 <unfinished ...>

I have also noticed that the above (the error message) happens inside the
function send-files. It is the last error clause at the end of the function
which reports the error message.

I'd like to be able to, e.g., modify the function send-files to give some
debugging output. How can I easily do that? Should I checkout guix with git
and try to run it in the source tree? Or do you have other suggestions?

Best regards,
Mikael

On Sat, Mar 21, 2020 at 11:16 PM Ludovic Courtès <ludo@gnu.org> wrote:

> Hi Mikael,
>
> Mikael Djurfeldt <mikael@djurfeldt.com> skribis:
>
> > I will look at the other buh reports. Meanwhile, do you have any hint on
> > where to look/insert logging code in order to see what is going on?
>
> On ‘wand’, can you run something like:
>
>   sudo strace -s 500 -p $(pidof sshd) -f -o sshd.log
>
> After that, run ‘guix offload test’ from the other host, and check what
> ‘sshd.log’ contains.  If would in particular look for all the ‘execve’
> syscalls as well as what happens right before ‘exit_group’ syscalls.
> There are probably error messages lurking there.  :-)
>
> You can share excerpts of the log but note that it will contain
> sensitive crypto material, so beware.
>
> Thanks in advance,
> Ludo’.
>

[-- Attachment #2: Type: text/html, Size: 3519 bytes --]

bug#40125: Problem with guix offload: Remote channel closed

[Ludovic Courtès]

Hello!

Mikael Djurfeldt <mikael@djurfeldt.com> skribis:

> This error message is more common. In this case, I couldn't see anything
> obviously strange except the following backtrace which contains strange
> characters:
>
> 31963 write(2, "Backtrace:\n          13 (apply-smob/1 #<catch-closure
> 56464c2232a0>)\nIn ice-9/boot-9.scm:\n    718:2 12 (call-with-prompt
> (\"prompt\") #<procedure 56464c2407e0 \342\200\246> \342\200\246)\nIn
> ice-9/eval.scm:\n    619:8 11 (_ #(#(#<directory (guile-user)
> 56464c2c4750>)))\nIn ice-9/command-line.scm:\n   181:18 10 (_ #<input:
> string 56464c2d2bd0>)\nIn unknown file:\n           9 (eval (begin
> (use-modules (guix) (srfi srfi-34) # #) \342\200\246) #)\nIn
> ice-9/eval.scm:\n   721:20  8 (primitive-eval (begin (use-modules (guix)
> (srfi \342\200\246) \342"..., 1150 <unfinished ...>

We’re missing a tiny bit.  :-)  Could you run strace with ‘-s 3000’ or
similar so that it doesn’t truncate it?

(The “strange characters” are Unicode HORIZONTAL ELLIPSIS.)

> I have also noticed that the above (the error message) happens inside the
> function send-files. It is the last error clause at the end of the function
> which reports the error message.
>
> I'd like to be able to, e.g., modify the function send-files to give some
> debugging output. How can I easily do that? Should I checkout guix with git
> and try to run it in the source tree? Or do you have other suggestions?

Note that the problems happens on the remote machine, which is what
makes it harder to debug.

If you want to debug (guix ssh), you’ll then have to run the daemon from
your Git checkout to exercise that code:

  sudo -E ./pre-inst-env guix-daemon --build-users-group=guixbuild

HTH!

Ludo’.

bug#40125: Problem with guix offload: Remote channel closed

[Mikael Djurfeldt]

[-- Attachment #1: Type: text/plain, Size: 1751 bytes --]

On Sun, Mar 22, 2020 at 9:46 PM Ludovic Courtès <ludo@gnu.org> wrote:

> Mikael Djurfeldt <mikael@djurfeldt.com> skribis:
>
> We’re missing a tiny bit.  :-)  Could you run strace with ‘-s 3000’ or
> similar so that it doesn’t truncate it?
>
> (The “strange characters” are Unicode HORIZONTAL ELLIPSIS.)
>

Ah... :)

Here's a larger chunk of backtrace:

19530 read(20, "Backtrace:\n          13 (apply-smob/1 #<catch-closure
563a9e2f92a0>)\nIn ice-9/boot-9.scm:\n    718:2 12 (call-with-prompt
(\"prompt\") #<procedure 563a9e3167e0 \342\200\246> \342\200\246)\nIn
ice-9/eval.scm:\n    619:8 11 (_ #(#(#<directory (guile-user)
563a9e39a750>)))\nIn ice-9/command-line.scm:\n   181:18 10 (_ #<input:
string 563a9e3a8bd0>)\nIn unknown file:\n           9 (eval (begin
(use-modules (guix) (srfi srfi-34) # #) \342\200\246) #)\nIn
ice-9/eval.scm:\n   721:20  8 (primitive-eval (begin (use-modules (guix)
(srfi \342\200\246) \342\200\246) \342\200\246))\nIn ice-9/psyntax.scm:\n
 1241:36  7 (expand-top-sequence ((begin (use-modules (guix) \342\200\246)
\342\200\246)) \342\200\246)\n  1194:19  6 (parse _ ((\"placeholder\"
placeholder)) ((top) #(# # \342\200\246)) \342\200\246)\n  1233:19  5
(parse _ ((\"placeholder\" placeholder)) ((top) #(# # \342\200\246))
\342\200\246)\n   285:10  4 (parse _ ((\"placeholder\" placeholder)) (()) _
c&e (eval) \342\200\246)\nIn ice-9/boot-9.scm:\n  3389:20  3
(process-use-modules _)\n   222:17  2 (map1 (((guix)) ((srfi srfi-34))
((rnrs io ports)) (#)))\n  3390:31  1 (_ ((guix)))\n   2809:6  0
(resolve-interface (guix) #:select _ #:hide _ #:prefix _
\342\200\246)\n\nice-9/boot-9.scm:2809:6: In procedure
resolve-interface:\nno code for module (guix)\n", 16384) = 1150

[-- Attachment #2: Type: text/html, Size: 2243 bytes --]

bug#40125: Problem with guix offload: Remote channel closed

[Mikael Djurfeldt]

[-- Attachment #1: Type: text/plain, Size: 2107 bytes --]

On Sun, Mar 22, 2020 at 10:40 PM Mikael Djurfeldt <mikael@djurfeldt.com>
wrote:

> On Sun, Mar 22, 2020 at 9:46 PM Ludovic Courtès <ludo@gnu.org> wrote:
>
>> Mikael Djurfeldt <mikael@djurfeldt.com> skribis:
>>
>> We’re missing a tiny bit.  :-)  Could you run strace with ‘-s 3000’ or
>> similar so that it doesn’t truncate it?
>>
>> (The “strange characters” are Unicode HORIZONTAL ELLIPSIS.)
>>
>
> Ah... :)
>
> Here's a larger chunk of backtrace:
>
> 19530 read(20, "Backtrace:\n          13 (apply-smob/1 #<catch-closure
> 563a9e2f92a0>)\nIn ice-9/boot-9.scm:\n    718:2 12 (call-with-prompt
> (\"prompt\") #<procedure 563a9e3167e0 \342\200\246> \342\200\246)\nIn
> ice-9/eval.scm:\n    619:8 11 (_ #(#(#<directory (guile-user)
> 563a9e39a750>)))\nIn ice-9/command-line.scm:\n   181:18 10 (_ #<input:
> string 563a9e3a8bd0>)\nIn unknown file:\n           9 (eval (begin
> (use-modules (guix) (srfi srfi-34) # #) \342\200\246) #)\nIn
> ice-9/eval.scm:\n   721:20  8 (primitive-eval (begin (use-modules (guix)
> (srfi \342\200\246) \342\200\246) \342\200\246))\nIn ice-9/psyntax.scm:\n
>  1241:36  7 (expand-top-sequence ((begin (use-modules (guix) \342\200\246)
> \342\200\246)) \342\200\246)\n  1194:19  6 (parse _ ((\"placeholder\"
> placeholder)) ((top) #(# # \342\200\246)) \342\200\246)\n  1233:19  5
> (parse _ ((\"placeholder\" placeholder)) ((top) #(# # \342\200\246))
> \342\200\246)\n   285:10  4 (parse _ ((\"placeholder\" placeholder)) (()) _
> c&e (eval) \342\200\246)\nIn ice-9/boot-9.scm:\n  3389:20  3
> (process-use-modules _)\n   222:17  2 (map1 (((guix)) ((srfi srfi-34))
> ((rnrs io ports)) (#)))\n  3390:31  1 (_ ((guix)))\n   2809:6  0
> (resolve-interface (guix) #:select _ #:hide _ #:prefix _
> \342\200\246)\n\nice-9/boot-9.scm:2809:6: In procedure
> resolve-interface:\nno code for module (guix)\n", 16384) = 1150
>

I think I understand part of the problem. It's not guix guile which is
invoked on wand but the guile which I have installed under /usr/local. But
why is this? guix as well as guile-ssh is installed at wand.

[-- Attachment #2: Type: text/html, Size: 2840 bytes --]

bug#40125: Problem with guix offload: Remote channel closed

[Mikael Djurfeldt]

[-- Attachment #1: Type: text/plain, Size: 2546 bytes --]

On Sun, Mar 22, 2020 at 10:56 PM Mikael Djurfeldt <mikael@djurfeldt.com>
wrote:

> On Sun, Mar 22, 2020 at 10:40 PM Mikael Djurfeldt <mikael@djurfeldt.com>
> wrote:
>
>> On Sun, Mar 22, 2020 at 9:46 PM Ludovic Courtès <ludo@gnu.org> wrote:
>>
>>> Mikael Djurfeldt <mikael@djurfeldt.com> skribis:
>>>
>>> We’re missing a tiny bit.  :-)  Could you run strace with ‘-s 3000’ or
>>> similar so that it doesn’t truncate it?
>>>
>>> (The “strange characters” are Unicode HORIZONTAL ELLIPSIS.)
>>>
>>
>> Ah... :)
>>
>> Here's a larger chunk of backtrace:
>>
>> 19530 read(20, "Backtrace:\n          13 (apply-smob/1 #<catch-closure
>> 563a9e2f92a0>)\nIn ice-9/boot-9.scm:\n    718:2 12 (call-with-prompt
>> (\"prompt\") #<procedure 563a9e3167e0 \342\200\246> \342\200\246)\nIn
>> ice-9/eval.scm:\n    619:8 11 (_ #(#(#<directory (guile-user)
>> 563a9e39a750>)))\nIn ice-9/command-line.scm:\n   181:18 10 (_ #<input:
>> string 563a9e3a8bd0>)\nIn unknown file:\n           9 (eval (begin
>> (use-modules (guix) (srfi srfi-34) # #) \342\200\246) #)\nIn
>> ice-9/eval.scm:\n   721:20  8 (primitive-eval (begin (use-modules (guix)
>> (srfi \342\200\246) \342\200\246) \342\200\246))\nIn ice-9/psyntax.scm:\n
>>  1241:36  7 (expand-top-sequence ((begin (use-modules (guix) \342\200\246)
>> \342\200\246)) \342\200\246)\n  1194:19  6 (parse _ ((\"placeholder\"
>> placeholder)) ((top) #(# # \342\200\246)) \342\200\246)\n  1233:19  5
>> (parse _ ((\"placeholder\" placeholder)) ((top) #(# # \342\200\246))
>> \342\200\246)\n   285:10  4 (parse _ ((\"placeholder\" placeholder)) (()) _
>> c&e (eval) \342\200\246)\nIn ice-9/boot-9.scm:\n  3389:20  3
>> (process-use-modules _)\n   222:17  2 (map1 (((guix)) ((srfi srfi-34))
>> ((rnrs io ports)) (#)))\n  3390:31  1 (_ ((guix)))\n   2809:6  0
>> (resolve-interface (guix) #:select _ #:hide _ #:prefix _
>> \342\200\246)\n\nice-9/boot-9.scm:2809:6: In procedure
>> resolve-interface:\nno code for module (guix)\n", 16384) = 1150
>>
>
> I think I understand part of the problem. It's not guix guile which is
> invoked on wand but the guile which I have installed under /usr/local. But
> why is this? guix as well as guile-ssh is installed at wand.
>

Which guile is it supposed to use? Which version and which "fusion of
modules"? And how are these supposed to be found on the build machine.
Looking in the strace, it seems like a bash shell with "guile -c" is
started without any path information.

(Not sure that I even ask the right questions.)

[-- Attachment #2: Type: text/html, Size: 3564 bytes --]

bug#40125: Problem with guix offload: Remote channel closed

[Mikael Djurfeldt]

[-- Attachment #1: Type: text/plain, Size: 2978 bytes --]

On Sun, Mar 22, 2020 at 11:13 PM Mikael Djurfeldt <mikael@djurfeldt.com>
wrote:

> On Sun, Mar 22, 2020 at 10:56 PM Mikael Djurfeldt <mikael@djurfeldt.com>
> wrote:
>
>> On Sun, Mar 22, 2020 at 10:40 PM Mikael Djurfeldt <mikael@djurfeldt.com>
>> wrote:
>>
>>> On Sun, Mar 22, 2020 at 9:46 PM Ludovic Courtès <ludo@gnu.org> wrote:
>>>
>>>> Mikael Djurfeldt <mikael@djurfeldt.com> skribis:
>>>>
>>>> We’re missing a tiny bit.  :-)  Could you run strace with ‘-s 3000’ or
>>>> similar so that it doesn’t truncate it?
>>>>
>>>> (The “strange characters” are Unicode HORIZONTAL ELLIPSIS.)
>>>>
>>>
>>> Ah... :)
>>>
>>> Here's a larger chunk of backtrace:
>>>
>>> 19530 read(20, "Backtrace:\n          13 (apply-smob/1 #<catch-closure
>>> 563a9e2f92a0>)\nIn ice-9/boot-9.scm:\n    718:2 12 (call-with-prompt
>>> (\"prompt\") #<procedure 563a9e3167e0 \342\200\246> \342\200\246)\nIn
>>> ice-9/eval.scm:\n    619:8 11 (_ #(#(#<directory (guile-user)
>>> 563a9e39a750>)))\nIn ice-9/command-line.scm:\n   181:18 10 (_ #<input:
>>> string 563a9e3a8bd0>)\nIn unknown file:\n           9 (eval (begin
>>> (use-modules (guix) (srfi srfi-34) # #) \342\200\246) #)\nIn
>>> ice-9/eval.scm:\n   721:20  8 (primitive-eval (begin (use-modules (guix)
>>> (srfi \342\200\246) \342\200\246) \342\200\246))\nIn ice-9/psyntax.scm:\n
>>>  1241:36  7 (expand-top-sequence ((begin (use-modules (guix) \342\200\246)
>>> \342\200\246)) \342\200\246)\n  1194:19  6 (parse _ ((\"placeholder\"
>>> placeholder)) ((top) #(# # \342\200\246)) \342\200\246)\n  1233:19  5
>>> (parse _ ((\"placeholder\" placeholder)) ((top) #(# # \342\200\246))
>>> \342\200\246)\n   285:10  4 (parse _ ((\"placeholder\" placeholder)) (()) _
>>> c&e (eval) \342\200\246)\nIn ice-9/boot-9.scm:\n  3389:20  3
>>> (process-use-modules _)\n   222:17  2 (map1 (((guix)) ((srfi srfi-34))
>>> ((rnrs io ports)) (#)))\n  3390:31  1 (_ ((guix)))\n   2809:6  0
>>> (resolve-interface (guix) #:select _ #:hide _ #:prefix _
>>> \342\200\246)\n\nice-9/boot-9.scm:2809:6: In procedure
>>> resolve-interface:\nno code for module (guix)\n", 16384) = 1150
>>>
>>
>> I think I understand part of the problem. It's not guix guile which is
>> invoked on wand but the guile which I have installed under /usr/local. But
>> why is this? guix as well as guile-ssh is installed at wand.
>>
>
> Which guile is it supposed to use? Which version and which "fusion of
> modules"? And how are these supposed to be found on the build machine.
> Looking in the strace, it seems like a bash shell with "guile -c" is
> started without any path information.
>
> (Not sure that I even ask the right questions.)
>

The PATH of the build machine user is:

/home/guix/.guix-profile/bin:/home/guix/.config/guix/current/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/usr/lib/jvm/java-8-oracle/bin:/usr/lib/jvm/java-8-oracle/db/bin:/usr/lib/jvm/java-8-oracle/jre/bin:/opt/thinlinc/bin

[-- Attachment #2: Type: text/html, Size: 4248 bytes --]

bug#40125: Problem with guix offload: Remote channel closed

[Mikael Djurfeldt]

[-- Attachment #1: Type: text/plain, Size: 803 bytes --]

On Sun, Mar 22, 2020 at 11:14 PM Mikael Djurfeldt <mikael@djurfeldt.com>
wrote:

> The PATH of the build machine user is:
>
> /home/guix/.guix-profile/bin:/home/guix/.config/guix/current/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/usr/lib/jvm/java-8-oracle/bin:/usr/lib/jvm/java-8-oracle/db/bin:/usr/lib/jvm/java-8-oracle/jre/bin:/opt/thinlinc/bin
>
>

The build host is not a pure guix installation but also has guix on top of
Debian Buster. It seems like the file transfer command is using an ssh call
which doesn't setup the proper guix environment (as, e.g., the
/usr/local/bin/guix script does). For example, PATH will have
/usr/local/bin before the guix environment such that the wrong guile is
selected.

Not sure how the proper setup should look like on the build host side...

[-- Attachment #2: Type: text/html, Size: 1196 bytes --]

bug#40125: Problem with guix offload: Remote channel closed

[Marius Bakke]

[-- Attachment #1: Type: text/plain, Size: 541 bytes --]

Mikael Djurfeldt <mikael@djurfeldt.com> writes:

> The PATH of the build machine user is:
>
> /home/guix/.guix-profile/bin:/home/guix/.config/guix/current/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/usr/lib/jvm/java-8-oracle/bin:/usr/lib/jvm/java-8-oracle/db/bin:/usr/lib/jvm/java-8-oracle/jre/bin:/opt/thinlinc/bin

Do you get the same result if you run:

  ssh wand 'echo $PATH'

?

It might be .bashrc is configured not to source the relevant scripts
when being invoked non-interactively, and/or through an SSH session.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

bug#40125: Problem with guix offload: Remote channel closed

[Mikael Djurfeldt]

[-- Attachment #1: Type: text/plain, Size: 766 bytes --]

On Sun, Mar 22, 2020 at 11:30 PM Marius Bakke <mbakke@fastmail.com> wrote:

> Mikael Djurfeldt <mikael@djurfeldt.com> writes:
>
> > The PATH of the build machine user is:
> >
> >
> /home/guix/.guix-profile/bin:/home/guix/.config/guix/current/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/usr/lib/jvm/java-8-oracle/bin:/usr/lib/jvm/java-8-oracle/db/bin:/usr/lib/jvm/java-8-oracle/jre/bin:/opt/thinlinc/bin
>
> Do you get the same result if you run:
>
>   ssh wand 'echo $PATH'
>
> ?
>
> It might be .bashrc is configured not to source the relevant scripts
> when being invoked non-interactively, and/or through an SSH session.
>

You're right. It doesn't source the scripts. The PATH only becomes
/usr/local/bin:/usr/bin:/bin:/usr/games in this case.

[-- Attachment #2: Type: text/html, Size: 1196 bytes --]

bug#40125: Problem with guix offload: Remote channel closed

[Marius Bakke]

[-- Attachment #1: Type: text/plain, Size: 956 bytes --]

Mikael Djurfeldt <mikael@djurfeldt.com> writes:

>> It might be .bashrc is configured not to source the relevant scripts
>> when being invoked non-interactively, and/or through an SSH session.
>>
>
> You're right. It doesn't source the scripts. The PATH only becomes
> /usr/local/bin:/usr/bin:/bin:/usr/games in this case.

On Guix System, the default .bashrc does this:

if [[ $- != *i* ]]
then
    # We are being invoked from a non-interactive shell.  If this
    # is an SSH session (as in "ssh host command"), source
    # /etc/profile so we get PATH and other essential variables.
    [[ -n "$SSH_CLIENT" ]] && source /etc/profile

    # Don't do anything else.
    return
fi

IIRC Debian does something similar to detect if being invoked
non-interactively, but then just returns instead of sourcing anything.

Adding a line that sources ~/.guix-profile/etc/profile before the check
for an interactive shell might be enough in that case.

HTH,
Marius

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

bug#40125: Problem with guix offload: Remote channel closed

[Mikael Djurfeldt]

[-- Attachment #1: Type: text/plain, Size: 2077 bytes --]

On Sun, Mar 22, 2020 at 11:59 PM Marius Bakke <mbakke@fastmail.com> wrote:

> Mikael Djurfeldt <mikael@djurfeldt.com> writes:
>
> >> It might be .bashrc is configured not to source the relevant scripts
> >> when being invoked non-interactively, and/or through an SSH session.
> >>
> >
> > You're right. It doesn't source the scripts. The PATH only becomes
> > /usr/local/bin:/usr/bin:/bin:/usr/games in this case.
>
> On Guix System, the default .bashrc does this:
>
> if [[ $- != *i* ]]
> then
>     # We are being invoked from a non-interactive shell.  If this
>     # is an SSH session (as in "ssh host command"), source
>     # /etc/profile so we get PATH and other essential variables.
>     [[ -n "$SSH_CLIENT" ]] && source /etc/profile
>
>     # Don't do anything else.
>     return
> fi
>
> IIRC Debian does something similar to detect if being invoked
> non-interactively, but then just returns instead of sourcing anything.
>
> Adding a line that sources ~/.guix-profile/etc/profile before the check
> for an interactive shell might be enough in that case.
>

I examined this a bit. You're right about .bashrc. But sourcing that
profile is not sufficient.

Just as a test, I enabled user specified environments in sshd_config (such
that ssh reads .ssh/environment) and added the following there:

PATH=/gnu/store/0awhym5h0m890n0wq87y0dxznh14rk88-guile-next-3.0.1/bin:/home/guix/.guix-profile/bin:/usr/local/bin:/usr/bin:/bin:/usr/games
GUILE_LOAD_COMPILED_PATH=/gnu/store/nkh7c4ygaivfxdq3zhccl4a5qxrn6s88-guix-module-union/lib/guile/3.0/site-ccache:/gnu/store/0awhym5h0m890n0wq87y0dxznh14rk88-guile-next-3.0.1/lib/guile/3.0/ccache
GUILE_LOAD_PATH=/gnu/store/nkh7c4ygaivfxdq3zhccl4a5qxrn6s88-guix-module-union/share/guile/site/3.0:/gnu/store/0awhym5h0m890n0wq87y0dxznh14rk88-guile-next-3.0.1/share/guile/3.0
GUIX_PROFILE=/home/guix/.guix-profile
GUIX_LOCPATH=/home/guix/.guix-profile/lib/locale

This made the offload test work.

Crucial here is the PATH to guile 3.0.1 as well as the GUILE_LOAD_* paths.

But how are these *supposed* to be setup on the build host???

[-- Attachment #2: Type: text/html, Size: 2726 bytes --]

bug#40125: Problem with guix offload: Remote channel closed

[Mikael Djurfeldt]

[-- Attachment #1: Type: text/plain, Size: 1251 bytes --]

On Mon, Mar 23, 2020 at 12:12 AM Mikael Djurfeldt <mikael@djurfeldt.com>
wrote:

>
> Just as a test, I enabled user specified environments in sshd_config (such
> that ssh reads .ssh/environment) and added the following there:
>
>
> PATH=/gnu/store/0awhym5h0m890n0wq87y0dxznh14rk88-guile-next-3.0.1/bin:/home/guix/.guix-profile/bin:/usr/local/bin:/usr/bin:/bin:/usr/games
>
> GUILE_LOAD_COMPILED_PATH=/gnu/store/nkh7c4ygaivfxdq3zhccl4a5qxrn6s88-guix-module-union/lib/guile/3.0/site-ccache:/gnu/store/0awhym5h0m890n0wq87y0dxznh14rk88-guile-next-3.0.1/lib/guile/3.0/ccache
>
> GUILE_LOAD_PATH=/gnu/store/nkh7c4ygaivfxdq3zhccl4a5qxrn6s88-guix-module-union/share/guile/site/3.0:/gnu/store/0awhym5h0m890n0wq87y0dxznh14rk88-guile-next-3.0.1/share/guile/3.0
> GUIX_PROFILE=/home/guix/.guix-profile
> GUIX_LOCPATH=/home/guix/.guix-profile/lib/locale
>
> This made the offload test work.
>
> Crucial here is the PATH to guile 3.0.1 as well as the GUILE_LOAD_* paths.
>
> But how are these *supposed* to be setup on the build host???
>

Just to be clear:

You're right that this can be done without .ssh/environment and just using
a modified .bashrc.

But how am I supposed to retrieve all those PATHS in a way that doesn't
have to be modified as I update guix?

[-- Attachment #2: Type: text/html, Size: 1747 bytes --]

bug#40125: Problem with guix offload: Remote channel closed

[Marius Bakke]

[-- Attachment #1: Type: text/plain, Size: 1785 bytes --]

Mikael Djurfeldt <mikael@djurfeldt.com> writes:

> Just as a test, I enabled user specified environments in sshd_config (such
> that ssh reads .ssh/environment) and added the following there:
>
> PATH=/gnu/store/0awhym5h0m890n0wq87y0dxznh14rk88-guile-next-3.0.1/bin:/home/guix/.guix-profile/bin:/usr/local/bin:/usr/bin:/bin:/usr/games
> GUILE_LOAD_COMPILED_PATH=/gnu/store/nkh7c4ygaivfxdq3zhccl4a5qxrn6s88-guix-module-union/lib/guile/3.0/site-ccache:/gnu/store/0awhym5h0m890n0wq87y0dxznh14rk88-guile-next-3.0.1/lib/guile/3.0/ccache
> GUILE_LOAD_PATH=/gnu/store/nkh7c4ygaivfxdq3zhccl4a5qxrn6s88-guix-module-union/share/guile/site/3.0:/gnu/store/0awhym5h0m890n0wq87y0dxznh14rk88-guile-next-3.0.1/share/guile/3.0
> GUIX_PROFILE=/home/guix/.guix-profile
> GUIX_LOCPATH=/home/guix/.guix-profile/lib/locale
>
> This made the offload test work.
>
> Crucial here is the PATH to guile 3.0.1 as well as the GUILE_LOAD_* paths.
>
> But how are these *supposed* to be setup on the build host???

Installing 'guile' and 'guix' (or their Guile 3.0 equivalents) into the
user profile is the simplest solution.  Then you can just source
~/.guix-profile/etc/profile early in .bashrc, because it will contain
all the relevant variables.

Otherwise manipulating PATH and GUILE_LOAD_PATH directly seems like a
fine solution to me, but you should refer to
~/.config/guix/current/share/guile and friends instead of
/gnu/store/...guix-module-union/share/guile because the latter is prone
to being garbage collected.

Does that make sense?  It might be useful to create a dedicated user
account for this to avoid clobbering your regular dotfiles/profiles.

I don't think there is an established practice here, so your feedback on
this is very valuable!  Offloading to a Guix System will just work, of
course.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

bug#40125: Problem with guix offload: Remote channel closed

[Ludovic Courtès]

Hi Mikael,

Mikael Djurfeldt <mikael@djurfeldt.com> skribis:

> On Sun, Mar 22, 2020 at 9:46 PM Ludovic Courtès <ludo@gnu.org> wrote:
>
>> Mikael Djurfeldt <mikael@djurfeldt.com> skribis:
>>
>> We’re missing a tiny bit.  :-)  Could you run strace with ‘-s 3000’ or
>> similar so that it doesn’t truncate it?
>>
>> (The “strange characters” are Unicode HORIZONTAL ELLIPSIS.)
>>
>
> Ah... :)
>
> Here's a larger chunk of backtrace:
>
> 19530 read(20, "Backtrace:\n          13 (apply-smob/1 #<catch-closure
> 563a9e2f92a0>)\nIn ice-9/boot-9.scm:\n    718:2 12 (call-with-prompt
> (\"prompt\") #<procedure 563a9e3167e0 \342\200\246> \342\200\246)\nIn
> ice-9/eval.scm:\n    619:8 11 (_ #(#(#<directory (guile-user)
> 563a9e39a750>)))\nIn ice-9/command-line.scm:\n   181:18 10 (_ #<input:
> string 563a9e3a8bd0>)\nIn unknown file:\n           9 (eval (begin
> (use-modules (guix) (srfi srfi-34) # #) \342\200\246) #)\nIn
> ice-9/eval.scm:\n   721:20  8 (primitive-eval (begin (use-modules (guix)
> (srfi \342\200\246) \342\200\246) \342\200\246))\nIn ice-9/psyntax.scm:\n
>  1241:36  7 (expand-top-sequence ((begin (use-modules (guix) \342\200\246)
> \342\200\246)) \342\200\246)\n  1194:19  6 (parse _ ((\"placeholder\"
> placeholder)) ((top) #(# # \342\200\246)) \342\200\246)\n  1233:19  5
> (parse _ ((\"placeholder\" placeholder)) ((top) #(# # \342\200\246))
> \342\200\246)\n   285:10  4 (parse _ ((\"placeholder\" placeholder)) (()) _
> c&e (eval) \342\200\246)\nIn ice-9/boot-9.scm:\n  3389:20  3
> (process-use-modules _)\n   222:17  2 (map1 (((guix)) ((srfi srfi-34))
> ((rnrs io ports)) (#)))\n  3390:31  1 (_ ((guix)))\n   2809:6  0
> (resolve-interface (guix) #:select _ #:hide _ #:prefix _
> \342\200\246)\n\nice-9/boot-9.scm:2809:6: In procedure
> resolve-interface:\nno code for module (guix)\n", 16384) = 1150

Thanks!  Commit 8f53d73493a2949e2db28cd7d689a690b2d9479a improves error
reporting in this case.

This can be tested by running “guix copy --to=HOST coreutils” for a host
where Guix modules are missing.

Thanks,
Ludo’.

bug#40125: Problem with guix offload: Remote channel closed

[Maxim Cournoyer]

Hello,

Ludovic Courtès <ludo@gnu.org> writes:

> Hi Mikael,
>
> Mikael Djurfeldt <mikael@djurfeldt.com> skribis:
>
>> On Sun, Mar 22, 2020 at 9:46 PM Ludovic Courtès <ludo@gnu.org> wrote:
>>
>>> Mikael Djurfeldt <mikael@djurfeldt.com> skribis:
>>>
>>> We’re missing a tiny bit.  :-)  Could you run strace with ‘-s 3000’ or
>>> similar so that it doesn’t truncate it?
>>>
>>> (The “strange characters” are Unicode HORIZONTAL ELLIPSIS.)
>>>
>>
>> Ah... :)
>>
>> Here's a larger chunk of backtrace:
>>
>> 19530 read(20, "Backtrace:\n          13 (apply-smob/1 #<catch-closure
>> 563a9e2f92a0>)\nIn ice-9/boot-9.scm:\n    718:2 12 (call-with-prompt
>> (\"prompt\") #<procedure 563a9e3167e0 \342\200\246> \342\200\246)\nIn
>> ice-9/eval.scm:\n    619:8 11 (_ #(#(#<directory (guile-user)
>> 563a9e39a750>)))\nIn ice-9/command-line.scm:\n   181:18 10 (_ #<input:
>> string 563a9e3a8bd0>)\nIn unknown file:\n           9 (eval (begin
>> (use-modules (guix) (srfi srfi-34) # #) \342\200\246) #)\nIn
>> ice-9/eval.scm:\n   721:20  8 (primitive-eval (begin (use-modules (guix)
>> (srfi \342\200\246) \342\200\246) \342\200\246))\nIn ice-9/psyntax.scm:\n
>>  1241:36  7 (expand-top-sequence ((begin (use-modules (guix) \342\200\246)
>> \342\200\246)) \342\200\246)\n  1194:19  6 (parse _ ((\"placeholder\"
>> placeholder)) ((top) #(# # \342\200\246)) \342\200\246)\n  1233:19  5
>> (parse _ ((\"placeholder\" placeholder)) ((top) #(# # \342\200\246))
>> \342\200\246)\n   285:10  4 (parse _ ((\"placeholder\" placeholder)) (()) _
>> c&e (eval) \342\200\246)\nIn ice-9/boot-9.scm:\n  3389:20  3
>> (process-use-modules _)\n   222:17  2 (map1 (((guix)) ((srfi srfi-34))
>> ((rnrs io ports)) (#)))\n  3390:31  1 (_ ((guix)))\n   2809:6  0
>> (resolve-interface (guix) #:select _ #:hide _ #:prefix _
>> \342\200\246)\n\nice-9/boot-9.scm:2809:6: In procedure
>> resolve-interface:\nno code for module (guix)\n", 16384) = 1150
>
> Thanks!  Commit 8f53d73493a2949e2db28cd7d689a690b2d9479a improves error
> reporting in this case.
>
> This can be tested by running “guix copy --to=HOST coreutils” for a host
> where Guix modules are missing.

Great, closing!

-- 
Thanks,
Maxim