Nginx and certbot cervices don't play well togther

Nginx and certbot cervices don't play well togther

[Brice Waegeneire]

Hello Guix,

After an suggestion from Tobias to give a try at forcing HTTPS for
Guix's websites on berlin, I had a go at it but it was more complex that
what I was expecting.  Looking deeper at nginx and certbot services it
appear both services don't play that well together, requering a inital
dance when deploying a new HTTPS virtual server. As explained in #36389¹
you need to:
« - run system configuration with just the certbot service
- use certbot to generate your initial certificates
- reconfigure with additional nginx server configuration, pointing to
  the SSL certificates created by certbot »

Indeed, with an operating-system continaing the following services it's
impossible to sart Nginx and Certbot at once as one would expect:

--8<---------------cut here---------------start------------->8---
(service nginx-service-type)
(service php-fpm-service-type)

(service certbot-service-type
         (certbot-configuration
           (certificates
             (list (certificate-configuration
                     (domains '("test.sama.re"))
                     (deploy-hook
                       (program-file
                         "nginx-deploy-hook"
                         #~(let ((pid (call-with-input-file "/var/run/nginx.pid"
                                                            read)))
                             (kill pid SIGHUP)))))))))

(cat-avatar-generator-service
  #:configuration
  (nginx-server-configuration
    (listen '("443 ssl"))
    (server-name '("test.sama.re"))
    (ssl-certificate
      "/etc/letsencrypt/live/test.sama.re/fullchain.pem")
    (ssl-certificate-key
      "/etc/letsencrypt/live/test.sama.re/privkey.pem")))
--8<---------------cut here---------------end--------------->8---

Here is the error from reconfiguring the system:

--8<---------------cut here---------------start------------->8---
# guix system reconfigure /etc/config.sm
[...]
building /gnu/store/55cq2ja4i5489s55viv9fh50032d1ziy-switch-to-system.scm.drv...
making '/gnu/store/p2rkcmrnpls5py7x2iappf2qcbxwlb95-system' the current system...
setting up setuid programs in '/run/setuid-programs'...
populating /etc from /gnu/store/k2kb8hsq3q0dhhad4a9pjh4kx32mn4g0-etc...
/var/lib/certbot/renew-certificates may need to be run
creating nginx log directory '/var/log/nginx'
creating nginx run directory '/var/run/nginx'
creating nginx temp directories '/var/run/nginx/{client_body,proxy,fastcgi,uwsgi,scgi}_temp'
nginx: [emerg] cannot load certificate "/etc/letsencrypt/live/test.sama.re/fullchain.pem": BIO_new_file() failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen('/etc/letsencrypt/live/test.sama.re/fullchain.pem','r') error:2006D080:BIO routines:BIO_new_file:no such file)
nginx: configuration file /gnu/store/chpw631djay2w39x7agg8zz53iayy4zy-nginx.conf test failed
`/gnu/store/jyxc290q7jyhhpalski0h13h8z9zvnka-openssh-authorized-keys/bricewge' -> `/etc/ssh/authorized_keys.d/bricewge'
The following derivation will be built:
   /gnu/store/qlzbrmpx6wnhzqcpqi9yrbb6xva82kvr-install-bootloader.scm.drv

building /gnu/store/qlzbrmpx6wnhzqcpqi9yrbb6xva82kvr-install-bootloader.scm.drv...
guix system: bootloader successfully installed on '/dev/sda'
The following derivation will be built:
   /gnu/store/ikak44inrnz3b3dx8j8csdakgqafbijn-upgrade-shepherd-services.scm.drv

building /gnu/store/ikak44inrnz3b3dx8j8csdakgqafbijn-upgrade-shepherd-services.scm.drv...
shepherd: Removing service 'dbus-system'...
shepherd: Service dbus-system has been stopped.
shepherd: Done.
shepherd: Service host-name has been started.
shepherd: Service user-homes has been started.
shepherd: Service host-name has been started.
shepherd: Service term-auto could not be started.
shepherd: Service php-fpm has been started.
guix system: warning: exception caught while executing 'start' on service 'nginx':
Throw to key `%exception' with args `("#<&invoke-error program: \"/gnu/store/hn1mvgafkpf5knrnzvwpgpdlzmq553al-nginx-1.19.6/sbin/nginx\" arguments: (\"-c\" \"/gnu/store/chpw631djay2w39x7agg8zz53iayy4zy-nginx.conf\" \"-p\" \"/var/run/nginx\") exit-status: 1 term-signal: #f stop-signal: #f>")'.
guix system: warning: some services could not be upgraded
hint: To allow changes to all the system services to take effect, you will need to reboot.
--8<---------------cut here---------------end--------------->8---

What happen is Nginx won't start because the certficate related files
present in it's configuration doesn't exist and we can't get a Let's
Encrypt certificate from a HTTP-01 challenge without that web server
running.  NixOS broke that chicken and egg problem by generating a
self-signed certificate first, after that starting nginx, then
requesting a valid Lets' Encrypt certificate and finally reloading
Nginx.  That way we end up with a Nginx server using Let's Encrypt
certificate with no more that a simple system reconfiguration.  Note
that, the initial self-signed certificate will need to be at the path
were certbot will put it's own certificate.

WDYT?

¹ https://bugs.gnu.org/36389

Cheers,
- Brice