Nginx service fails

Nginx service fails

[Andreas Enge]

Hello,

writing to guix-devel first, since I am not sure whether I should file
a bug report. When reconfiguring my virtual machine today after I had
added a virtual host to nginx, it failed with the following error message:

...
creating nginx log directory '/var/log/nginx'
creating nginx run directory '/var/run/nginx'
creating nginx temp directories '/var/run/nginx/{client_body,proxy,fastcgi,uwsgi,scgi}_temp'
nginx: [alert] could not open error log file: open() "/gnu/store/pp71iff1qxwhh82vm34g18h9kmn0xrg5-nginx-1.13.9/logs/error.log" failed (2: No such file or directory)
nginx: the configuration file /gnu/store/5ixkryw6jl32cm6d1g9jb8dm9rbz8csc-nginx.conf syntax is ok
nginx: configuration file /gnu/store/5ixkryw6jl32cm6d1g9jb8dm9rbz8csc-nginx.conf test is successful
`/gnu/store/zchh8s3r1bbmia3zfxsyhsz3c4b9fmps-openssh-authorized-keys/root' -> `/etc/ssh/authorized_keys.d/root'
`/gnu/store/zchh8s3r1bbmia3zfxsyhsz3c4b9fmps-openssh-authorized-keys/andreas' -> `/etc/ssh/authorized_keys.d/andreas'
guix system: loading new services: user-homes term-auto nginx...
shepherd: Evaluating user expression (register-services (primitive-load "/gnu/st?") ?).
guix system: error: exception caught while executing 'eval' on service 'root':
find-long-options: unbound variable

And then the nginx service had disappeared. Even "guix system roll-back"
did not bring it back.

Since then I am trying to recompile a more recent git checkout, so I can
probably not reproduce the problem. But does it ring a bell?

Andreas

Re: Nginx service fails

[Andreas Enge]

Well, I am turning this into a bug report, since it still occurs with
the latest git commit ac1a9ce8b07f3b80900ee08436ff6e683e8dc195 .

This is the result of "./pre-inst-env guix system reconfigure ...",
where "..." is my configuration file:

...
creating nginx log directory '/var/log/nginx'
creating nginx run directory '/var/run/nginx'
creating nginx temp directories '/var/run/nginx/{client_body,proxy,fastcgi,uwsgi,scgi}_temp'
nginx: [alert] could not open error log file: open() "/gnu/store/pp71iff1qxwhh82vm34g18h9kmn0xrg5-nginx-1.13.9/logs/error.log" failed (2: No such file or directory)
nginx: the configuration file /gnu/store/5ixkryw6jl32cm6d1g9jb8dm9rbz8csc-nginx.conf syntax is ok
nginx: configuration file /gnu/store/5ixkryw6jl32cm6d1g9jb8dm9rbz8csc-nginx.conf test is successful
`/gnu/store/zchh8s3r1bbmia3zfxsyhsz3c4b9fmps-openssh-authorized-keys/root' -> `/etc/ssh/authorized_keys.d/root'
`/gnu/store/zchh8s3r1bbmia3zfxsyhsz3c4b9fmps-openssh-authorized-keys/andreas' -> `/etc/ssh/authorized_keys.d/andreas'
guix system: loading new services: user-homes term-auto nginx...
shepherd: Evaluating user expression (register-services (primitive-load "/gnu/st?") ?).
guix system: error: exception caught while executing 'eval' on service 'root':
find-long-options: unbound variable
Installing for i386-pc platform.
/gnu/store/1dnbfda2p1bxwyl0rcm96ka9pmi0wb88-grub-2.02/sbin/grub-install: warning: disk does not exist, so falling back to partition device /dev/xvda2.
/gnu/store/1dnbfda2p1bxwyl0rcm96ka9pmi0wb88-grub-2.02/sbin/grub-install: warning: disk does not exist, so falling back to partition device /dev/xvda2.
/gnu/store/1dnbfda2p1bxwyl0rcm96ka9pmi0wb88-grub-2.02/sbin/grub-install: warning: disk does not exist, so falling back to partition device /dev/xvda2.
/gnu/store/1dnbfda2p1bxwyl0rcm96ka9pmi0wb88-grub-2.02/sbin/grub-install: error: cannot find a GRUB drive for /dev/sda.  Check your device.map.
guix system: error: failed to install bootloader /gnu/store/9iv63jm07klxvrr4fpwv6q5vpnca13ja-bootloader-installer

The final error is "normal", since I am installing in a Xen virtual machine,
where /dev/sda does not exist; it did not matter before.
The real error occurs above, the "uncaught exception".

But:
# herd status nginx
herd: service 'nginx' could not be found

Then I do a
# ./pre-inst-env guix system roll-back
# herd status nginx
herd: service 'nginx' could not be found

Otherwise said, I can go back, but my previously running web server has
definitely gone! How do I get it back?

Andreas

Re: Nginx service fails

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 663 bytes --]

Hi Andreas,

> shepherd: Evaluating user expression (register-services (primitive-load "/gnu/st?") ?).
> guix system: error: exception caught while executing 'eval' on service 'root':
> find-long-options: unbound variable

find-long-options is supposed to be in gnu/build/linux-boot.scm and
gnu/services/base.scm is using it.

It's in there for some weeks now...

(define agetty-shepherd-service
  (match-lambda
...
         (modules '((ice-9 match) (gnu build linux-boot)))
...
         (start #~(let ((tty #$(default-serial-port)))
...

You can find out where guix is by:

readlink $(which guix)

and then examine the scm files there...

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Nginx service fails

[Andreas Enge]

Hello Danny,

thanks for your quick reply!

On Mon, Mar 05, 2018 at 12:07:44AM +0100, Danny Milosavljevic wrote:
> find-long-options is supposed to be in gnu/build/linux-boot.scm and
> gnu/services/base.scm is using it.
> 
> It's in there for some weeks now...
> 
> You can find out where guix is by:
> readlink $(which guix)
> and then examine the scm files there...

It should not matter, since I am using "./pre-inst-env" in a current
git checkout. But "guix --version" is at 0.14.0-9.bdf0c64, which also
has find-long-options. I did a successful reconfigure on February 28.

Trying to use the old guix (without ./pre-inst-env), I am also running
into trouble:
$ guix system reconfigure AE/clementi.scm
guix system: error: failed to load 'AE/clementi.scm':
AE/clementi.scm:5:0: In procedure allocate-struct: Wrong type argument in position 2: 25

The configuration file looks like this:
(use-modules (gnu))
(use-service-modules networking ssh web)
(use-package-modules screen ssh vim)

(operating-system
  (host-name "clementi")

So line 5 is "operating-system". I think there is a problem with caching
somewhere:
$ rm -rf ~/.cache
$ guix system reconfigure AE/clementi.scm
Now things happen. (But they hang when contacting the substitute server.)

It has also bitten me in the other direction:
When I run "guix system reconfigure...", press "Ctrl-C" immediately, and then
run "./pre-inst-env guix system reconfigure...", I get almost the same message:
guix system: error: failed to load 'AE/clementi.scm':
/home/andreas/guix/AE/clementi.scm:5:0: In procedure allocate-struct: Wrong type argument in position 2: 24

Is this normal? How are users supposed to know they should delete .cache?
And when is this actually necessary?

Andreas

Re: Nginx service fails

[Andreas Enge]

PS: I rebooted the virtual machine, and now it looks as if even the
openssh service did not start:
... port 22: Connection refused
:-(

bug#30706: Nginx service fails

[Ricardo Wurmus]

Andreas Enge <andreas@enge.fr> writes:

> Well, I am turning this into a bug report, since it still occurs with
> the latest git commit ac1a9ce8b07f3b80900ee08436ff6e683e8dc195 .
>
> This is the result of "./pre-inst-env guix system reconfigure ...",
> where "..." is my configuration file:
>
> ...
> creating nginx log directory '/var/log/nginx'
> creating nginx run directory '/var/run/nginx'
> creating nginx temp directories '/var/run/nginx/{client_body,proxy,fastcgi,uwsgi,scgi}_temp'
> nginx: [alert] could not open error log file: open() "/gnu/store/pp71iff1qxwhh82vm34g18h9kmn0xrg5-nginx-1.13.9/logs/error.log" failed (2: No such file or directory)
> nginx: the configuration file /gnu/store/5ixkryw6jl32cm6d1g9jb8dm9rbz8csc-nginx.conf syntax is ok
> nginx: configuration file /gnu/store/5ixkryw6jl32cm6d1g9jb8dm9rbz8csc-nginx.conf test is successful
> `/gnu/store/zchh8s3r1bbmia3zfxsyhsz3c4b9fmps-openssh-authorized-keys/root' -> `/etc/ssh/authorized_keys.d/root'
> `/gnu/store/zchh8s3r1bbmia3zfxsyhsz3c4b9fmps-openssh-authorized-keys/andreas' -> `/etc/ssh/authorized_keys.d/andreas'
> guix system: loading new services: user-homes term-auto nginx...
> shepherd: Evaluating user expression (register-services (primitive-load "/gnu/st?") ?).
> guix system: error: exception caught while executing 'eval' on service 'root':
> find-long-options: unbound variable

I had the same error when updating my i686 netbook after a long while.
After a reboot everything seemed to be fine, though.

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

bug#30706: Nginx service fails

[Andreas Enge]

On Mon, Mar 05, 2018 at 08:23:18AM +0100, Ricardo Wurmus wrote:
> I had the same error when updating my i686 netbook after a long while.
> After a reboot everything seemed to be fine, though.

Ah, thanks for the information! A reboot made things worse in my case -
I rebooted the virtual machine, and now I cannot ssh into it any more.
So it looks like I will have to set it up from scratch again...

In my case, the problem occurred between February 28 and March 4.

Andreas

Re: bug#30706: Nginx service fails

[Andreas Enge]

On Mon, Mar 05, 2018 at 08:23:18AM +0100, Ricardo Wurmus wrote:
> I had the same error when updating my i686 netbook after a long while.
> After a reboot everything seemed to be fine, though.

Ah, thanks for the information! A reboot made things worse in my case -
I rebooted the virtual machine, and now I cannot ssh into it any more.
So it looks like I will have to set it up from scratch again...

In my case, the problem occurred between February 28 and March 4.

Andreas

Re: Nginx service fails

[Andreas Enge]

Hello,

On Mon, Mar 05, 2018 at 12:43:35AM +0100, Andreas Enge wrote:
> $ guix system reconfigure AE/clementi.scm
> guix system: error: failed to load 'AE/clementi.scm':
> AE/clementi.scm:5:0: In procedure allocate-struct: Wrong type argument in position 2: 25

I think I know the answer: .cache contains the compiled file for the operating
system declaration (clementi.go), and with the recent changes to how these are
parsed, it is incompatible.

> Is this normal? How are users supposed to know they should delete .cache?

Hopefully, this does not occur with "guix pull", and then it might be
considered normal... Is .cache actually necessary, since we generally
recompile from make files? Or is it needed since the make files do not
encode the precise dependency graph of the modules, as they usually do
for C code? If it is not necessary, could it be disabled by the project?

Andreas

Re: Nginx service fails

[Ludovic Courtès]

Hi,

Andreas Enge <andreas@enge.fr> skribis:

> On Mon, Mar 05, 2018 at 12:43:35AM +0100, Andreas Enge wrote:
>> $ guix system reconfigure AE/clementi.scm
>> guix system: error: failed to load 'AE/clementi.scm':
>> AE/clementi.scm:5:0: In procedure allocate-struct: Wrong type argument in position 2: 25
>
> I think I know the answer: .cache contains the compiled file for the operating
> system declaration (clementi.go), and with the recent changes to how these are
> parsed, it is incompatible.

Indeed.  See <https://bugs.gnu.org/29881>.  I’ve added a workaround that
addresses the most common use case.

Ludo’.

bug#30706: Nginx service fails

[Ludovic Courtès]

Andreas Enge <andreas@enge.fr> skribis:

> guix system: loading new services: user-homes term-auto nginx...
> shepherd: Evaluating user expression (register-services (primitive-load "/gnu/st?") ?).
> guix system: error: exception caught while executing 'eval' on service 'root':
> find-long-options: unbound variable

The problem we have here is that the agetty service expects
‘find-long-options’ from linux-boot.scm, and it expects it at the top
level.

So what happens above is that we evaluate in PID 1 code like:

  (make <service>
    ;; …
    #:start (let ((tty … (find-long-options …) …))
               …))

If you run this on an “old” GuixSD, ‘find-long-options’ is undefined.

Thus the whole (register-services …) expression fails to evaluate, and
we end up with some of the services missing.

Conclusions:

  1. ‘guix system reconfigure’ should probably register services one by
     one so that if one of the service expressions is erroneous, we
     don’t bork everything.  See ‘upgrade-shepherd-services’.

  2. IWBN to delay execution of this whole default-tty thing to the
     #:start method.  Ideas, Danny?

In general we should do as little as possible at the top level in the
Shepherd config file.

Ludo’.

Re: bug#30706: Nginx service fails

[Ludovic Courtès]

Andreas Enge <andreas@enge.fr> skribis:

> guix system: loading new services: user-homes term-auto nginx...
> shepherd: Evaluating user expression (register-services (primitive-load "/gnu/st?") ?).
> guix system: error: exception caught while executing 'eval' on service 'root':
> find-long-options: unbound variable

The problem we have here is that the agetty service expects
‘find-long-options’ from linux-boot.scm, and it expects it at the top
level.

So what happens above is that we evaluate in PID 1 code like:

  (make <service>
    ;; …
    #:start (let ((tty … (find-long-options …) …))
               …))

If you run this on an “old” GuixSD, ‘find-long-options’ is undefined.

Thus the whole (register-services …) expression fails to evaluate, and
we end up with some of the services missing.

Conclusions:

  1. ‘guix system reconfigure’ should probably register services one by
     one so that if one of the service expressions is erroneous, we
     don’t bork everything.  See ‘upgrade-shepherd-services’.

  2. IWBN to delay execution of this whole default-tty thing to the
     #:start method.  Ideas, Danny?

In general we should do as little as possible at the top level in the
Shepherd config file.

Ludo’.

bug#30706: Nginx service fails

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 1320 bytes --]

Hi Ludo,

> If you run this on an “old” GuixSD, ‘find-long-options’ is undefined.

How can it be that (gnu services base) with find-long-options call is present
but the (gnu build linux-boot)'s find-long options isn't present?

Aren't they either both added by "guix system reconfigure" (or both removed)?

Also when selecting an old generation in the Grub boot menu, isn't both
(gnu build linux-boot) and (gnu services base) at the same generation when
starting up the service (as opposed to stopping the old service)?

>   1. ‘guix system reconfigure’ should probably register services one by
>      one so that if one of the service expressions is erroneous, we
>      don’t bork everything.  See ‘upgrade-shepherd-services’.

Yes please.

>   2. IWBN to delay execution of this whole default-tty thing to the
>      #:start method.  Ideas, Danny?

The idea was that if you specify a serial console at boot that you can
actually log in at that console.

So it's trying to find out whether, at the time of service start,
there is a serial console specified (in the Linux command line), and if
so, start an agetty.  Otherwise do not start that agetty.

We could also do that without a guix service - but I thought it would be
nice to have a guix service for it as well.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

bug#30706: Nginx service fails

[Ludovic Courtès]

Heya,

Danny Milosavljevic <dannym@scratchpost.org> skribis:

>> If you run this on an “old” GuixSD, ‘find-long-options’ is undefined.
>
> How can it be that (gnu services base) with find-long-options call is present
> but the (gnu build linux-boot)'s find-long options isn't present?

The service-upgrade code loads new service definitions in PID 1.
However, it does not force a reload of already-loaded modules.

What happens here is that (gnu build linux-boot), the one without
‘find-long-options’, is already available in PID 1.  Thus, when end up
using that one, which lacks ‘find-long-options’.

We could call ‘reload-module’, but that’s probably not a great idea as
it could cause breakage in previously-loaded code in PID 1.  So I think
the current approach is the safest, and breakage of this sort should be
quite rare; we should pay attention to such issues, though, and try hard
to avoid them.

(Note that there’s no problem once you reboot, of course.)

>>   1. ‘guix system reconfigure’ should probably register services one by
>>      one so that if one of the service expressions is erroneous, we
>>      don’t bork everything.  See ‘upgrade-shepherd-services’.
>
> Yes please.
>
>>   2. IWBN to delay execution of this whole default-tty thing to the
>>      #:start method.  Ideas, Danny?
>
> The idea was that if you specify a serial console at boot that you can
> actually log in at that console.
>
> So it's trying to find out whether, at the time of service start,
> there is a serial console specified (in the Linux command line), and if
> so, start an agetty.  Otherwise do not start that agetty.
>
> We could also do that without a guix service - but I thought it would be
> nice to have a guix service for it as well.

I agree.  I think what you did in
c32e3ddedd103318ca3f0a4bf0c91c91e2517806 is good.  The effect here is
just that agetty would fail to start upon reconfigure, but that’s an
acceptable limitation IMO.

Thanks,
Ludo’.

Re: bug#30706: Nginx service fails

[Ludovic Courtès]

Heya,

Danny Milosavljevic <dannym@scratchpost.org> skribis:

>> If you run this on an “old” GuixSD, ‘find-long-options’ is undefined.
>
> How can it be that (gnu services base) with find-long-options call is present
> but the (gnu build linux-boot)'s find-long options isn't present?

The service-upgrade code loads new service definitions in PID 1.
However, it does not force a reload of already-loaded modules.

What happens here is that (gnu build linux-boot), the one without
‘find-long-options’, is already available in PID 1.  Thus, when end up
using that one, which lacks ‘find-long-options’.

We could call ‘reload-module’, but that’s probably not a great idea as
it could cause breakage in previously-loaded code in PID 1.  So I think
the current approach is the safest, and breakage of this sort should be
quite rare; we should pay attention to such issues, though, and try hard
to avoid them.

(Note that there’s no problem once you reboot, of course.)

>>   1. ‘guix system reconfigure’ should probably register services one by
>>      one so that if one of the service expressions is erroneous, we
>>      don’t bork everything.  See ‘upgrade-shepherd-services’.
>
> Yes please.
>
>>   2. IWBN to delay execution of this whole default-tty thing to the
>>      #:start method.  Ideas, Danny?
>
> The idea was that if you specify a serial console at boot that you can
> actually log in at that console.
>
> So it's trying to find out whether, at the time of service start,
> there is a serial console specified (in the Linux command line), and if
> so, start an agetty.  Otherwise do not start that agetty.
>
> We could also do that without a guix service - but I thought it would be
> nice to have a guix service for it as well.

I agree.  I think what you did in
c32e3ddedd103318ca3f0a4bf0c91c91e2517806 is good.  The effect here is
just that agetty would fail to start upon reconfigure, but that’s an
acceptable limitation IMO.

Thanks,
Ludo’.