bug#36389: nginx/certbot interaction doesn't work as documented

bug#36389: nginx/certbot interaction doesn't work as documented

[Robert Vollmert]

I’ve tried setting up nginx with certbot on guix. Two immediate issues:

- certbot extends the nginx service to serve challenge files. It appears
  that this nginx service extension conflicts (silently) with an independently
  configured nginx service. I.e., I had nginx previously configured, and
  after adding certbot, my previous nginx kept running with the previous
  configuration (even after herd restart nginx), while there was an additional
  nginx config in the gnu store with the certbot-specific fragments. certbot
  activation called nginx to test that fragment, but apparently never started
  nginx (successfully?). There were no errors.

  After removing the stand-alone nginx service and restarting nginx, it started
  with the certbot configuration.

- After this, /var/lib/certbot/renew worked successfully to register a
  certificate, but then failed when calling the nginx deploy hook that I’d
  copied from the guix certbot documentation, because /var/run/nginx/pid
  doesn’t exist. That might be a bug in the nginx package, not sure. I can’t
  find an nginx pid file anywhere, and no other errors related to it either,
  even though the config file includes
    pid /var/run/nginx/pid;

bug#36389: nginx/certbot interaction doesn't work as documented

[Alex Sassmannshausen]

Hi Robert,

Robert Vollmert <rob@vllmrt.net> writes:

> I’ve tried setting up nginx with certbot on guix. Two immediate issues:
>
> - certbot extends the nginx service to serve challenge files. It appears
>   that this nginx service extension conflicts (silently) with an independently
>   configured nginx service. I.e., I had nginx previously configured, and
>   after adding certbot, my previous nginx kept running with the previous
>   configuration (even after herd restart nginx), while there was an additional
>   nginx config in the gnu store with the certbot-specific fragments. certbot
>   activation called nginx to test that fragment, but apparently never started
>   nginx (successfully?). There were no errors.
>
>   After removing the stand-alone nginx service and restarting nginx, it started
>   with the certbot configuration.

This sounds odd, and I don't recall having this issue on my servers with
nginx SSL server configuration extended with certbot service.

>
> - After this, /var/lib/certbot/renew worked successfully to register a
>   certificate, but then failed when calling the nginx deploy hook that I’d
>   copied from the guix certbot documentation, because /var/run/nginx/pid
>   doesn’t exist. That might be a bug in the nginx package, not sure. I can’t
>   find an nginx pid file anywhere, and no other errors related to it either,
>   even though the config file includes
>  pid /var/run/nginx/pid;

The pid exists on my servers running an SSL nginx server config
configuration extended with certbot.

I've found the certbot & nginx services, overall, work very well
together.  But there are a couple of gotchas in my experience:

- The certbot service includes a redirect from port 80 to 443 for all
  except .well-known location.  By itself this may cause no problems for
  you.

- If deploying on a server that hitherto has no SSL certificate you have
  a chicken and egg problem: you will want your site to be configured to
  use the letsencrypt cert directories, to serve ssl (the redirect means
  any non-ssl deployments won't work anyway), but those directories
  don't yet exist as you haven't generated certs with certbot yet.

Here's a journey that should work:
- run system configuration with just the certbot service
- use certbot to generate your initial certificates
- reconfigure with additional nginx server configuration, pointing to
  the SSL certificates created by certbot

If the above is not helpful, perhaps you could share the nginx
configuration generated when you have both certbot & your custom server
running?

Can't promise anything, but we might be able to spot what's happening.

Best wishes,

Alex

bug#36389: odd

[Robert Vollmert]

I agree that it sounds odd, and some of my original diagnostic
must be skewed. After several configuration changes and
system reconfigurations and nginx restarts, I do appear to
have a sensible state currently, and I can’t reliably
reproduce the problems I had before. I’m also pretty sure I
didn’t imagine it all, though.


Here’s something else I ran into while getting there:

At some point, nginx was running, even after calling

# herd stop nginx

and herd did list it as stopped. That nginx instance that got
away from shepherd might have been involved in the earlier
trouble. (Is it ok for shepherd to lose track of a child like
that?)

Another thing was that I got a failed nginx configuration test
that didn’t make sense. Notably, it complained that

(a) the user directive `user nginx nginx;` is ineffective when
when not running as root and
(b) it didn’t have permission to access the letsencrypt keys.

Both of these indicate that the configuration test was not run
as root. I don’t see any reason in the code why that would
happen…


I’ll keep an eye on it and see if something similar occurs
again.

bug#36389: Nginx and certbot

[Andreas Enge]

Hello,

I am also experiencing problems with setting up nginx and certbot, but I
think it is more nginx that is to blame. After reconfiguring and restarting
nginx, it is still running with the old configuration. Only rebooting solves
the problem for me.

Here is what it looks like (everything as root):
$ ps -ef | grep nginx
root      2821     1  0 17:03 ?        00:00:00 nginx: master process /gnu/store/bdhfqs7sx3mal6pzz8z00hw4cpn5dj7x-nginx-1.21.4/sbin/nginx -c /gnu/store/q7bwm828r8y88sfs395n04bi8s6b7zwl-nginx.conf -p /var/run/nginx

$ guix system reconfigure ...
nginx: configuration file /gnu/store/clq2yshkq3gxpcqa6d54m8qif8i37kl9-nginx.conf test is successful

$ herd restart nginx; ps -ef | grep nginx
root      2835     1  0 17:12 ?        00:00:00 nginx: master process /gnu/store/bdhfqs7sx3mal6pzz8z00hw4cpn5dj7x-nginx-1.21.4/sbin/nginx -c /gnu/store/q7bwm828r8y88sfs395n04bi8s6b7zwl-nginx.conf -p /var/run/nginx

Notice that it is still running the old, q7b... config file!

$ reboot
$ ps -ef | grep nginx
root       188     1  0 17:13 ?        00:00:00 nginx: master process /gnu/store/bdhfqs7sx3mal6pzz8z00hw4cpn5dj7x-nginx-1.21.4/sbin/nginx -c /gnu/store/clq2yshkq3gxpcqa6d54m8qif8i37kl9-nginx.conf -p /var/run/nginx

Now the new, clq... config file is used!

So somehow nginx appears to memorise its previous configuration file even
when the service is stopped.

The problem can be solved by rebooting after each web server reconfiguration,
but this is of course not very comfortable.

Andreas

bug#36389: Nginx and certbot

[Andreas Enge]

Actually this seems to be a thing of the service, not nginx itself.

When I stop the service with the old configuration file, manually run
   /gnu/store/bdhfqs7sx3mal6pzz8z00hw4cpn5dj7x-nginx-1.21.4/sbin/nginx -p /var/run/nginx -c new_configuration_file
kill the process, and "herd restart nginx",
the herd service uses the old configuration file again.

Andreas