Dependency cycle between certbot and nginx is not obvious.

Dependency cycle between certbot and nginx is not obvious.

[MSavoritias]

Hey,


I spent the better part of the day today trying to debug an error that 
was happening while deploying a server with two server names in the 
nginx field.

My config was:

      (service certbot-service-type
           (certbot-configuration
            (certificates
         (list
          (certificate-configuration
           ;; The first domain provided will be the subject CN of the 
certificate, and all domains will be Subject Alternative Names on the 
certificate.
           (domains '("joinxmpp.org" "www.joinxmpp.org")))))))
      (service nginx-service-type
           (nginx-configuration
            (server-blocks
         (list (nginx-server-configuration
                (server-name '("joinxmpp.org www.joinxmpp.org"))
                (ssl-certificate "/etc/certs/joinxmpp.org/fullchain.pem")
                (ssl-certificate-key "/etc/certs/joinxmpp.org/privkey.pem")
                (locations
             (list
              (nginx-location-configuration
               (uri "/support")
               (body '("return 302 
\"https://chat.joinjabber.org/#/guest?join=support\";")))))))))))


turns out that this cant be deployed at once on the server. although it 
is correct. The reason is that certbot/letsencrypt will complain with 
the message:

Hint: The Certificate Authority failed to download the temporary 
challenge files created by Certbot. Ensure that the listed domains serve 
their content from the provided --webroot-path/-w and that files created 
there can be downloaded from the internet.


Mind you this error appeared on only one of the domains not both.

This was solved by running the system once with certbot. Then rebooting 
with the nginx service. That idea appear to me after i found a mailing 
list thread from 5 years ago saying that there is a dependency cycle 
between nginx and certbot and they cant run at the same time the first time.


The reason I am sending an email is because I am not sure if this is 
something that could be fixed somehow, or if this is unavoidable and i 
should just send a patch revising the docs slightly to suggest that if 
an nginx service is used with multiple domains in the server name, you 
should start the system once with certbot and then reboot with nginx.


MSavoritias

Re: Dependency cycle between certbot and nginx is not obvious.

[Jack Hill]

On Fri, 3 May 2024, MSavoritias wrote:

> Hey,
>
>
> I spent the better part of the day today trying to debug an error that was 
> happening while deploying a server with two server names in the nginx field.
>
> My config was:
>
>      (service certbot-service-type
>           (certbot-configuration
>            (certificates
>         (list
>          (certificate-configuration
>           ;; The first domain provided will be the subject CN of the 
> certificate, and all domains will be Subject Alternative Names on the 
> certificate.
>           (domains '("joinxmpp.org" "www.joinxmpp.org")))))))
>      (service nginx-service-type
>           (nginx-configuration
>            (server-blocks
>         (list (nginx-server-configuration
>                (server-name '("joinxmpp.org www.joinxmpp.org"))
>                (ssl-certificate "/etc/certs/joinxmpp.org/fullchain.pem")
>                (ssl-certificate-key "/etc/certs/joinxmpp.org/privkey.pem")
>                (locations
>             (list
>              (nginx-location-configuration
>               (uri "/support")
>               (body '("return 302 
> \"https://chat.joinjabber.org/#/guest?join=support\";")))))))))))
>
>
> turns out that this cant be deployed at once on the server. although it is 
> correct. The reason is that certbot/letsencrypt will complain with the 
> message:
>
> Hint: The Certificate Authority failed to download the temporary challenge 
> files created by Certbot. Ensure that the listed domains serve their content 
> from the provided --webroot-path/-w and that files created there can be 
> downloaded from the internet.
>
>
> Mind you this error appeared on only one of the domains not both.
>
> This was solved by running the system once with certbot. Then rebooting with 
> the nginx service. That idea appear to me after i found a mailing list thread 
> from 5 years ago saying that there is a dependency cycle between nginx and 
> certbot and they cant run at the same time the first time.
>
>
> The reason I am sending an email is because I am not sure if this is 
> something that could be fixed somehow, or if this is unavoidable and i should 
> just send a patch revising the docs slightly to suggest that if an nginx 
> service is used with multiple domains in the server name, you should start 
> the system once with certbot and then reboot with nginx.

Hi,

Indeed, this has been an annoyance. However, if I understand your problem 
correctly, I'm happy to report that it has now been fixed [0]! The 
start-self-signed? filed in certificate-configuration [1] should allow 
nginx to start before certbot has had a chance to get a proper cert signed 
by providing a structurally valid one for nginx to use. From the 
documentation:

> start-self-signed? (default: #t)
>  Whether to generate an initial self-signed certificate during system
>  activation. This option is particularly useful to allow nginx to start
>  before certbot has run, because certbot relies on nginx running to perform
>  HTTP challenges.

However, that says that the default is #t, but it still doesn't work 
for you. I wonder what I'm missing.

Best,
Jack

[0] https://issues.guix.gnu.org/46961#1
[1] https://guix.gnu.org/en/manual/devel/en/html_node/Certificate-Services.html

Re: Dependency cycle between certbot and nginx is not obvious.

[MSavoritias]

Another issue I just encountered seems to be that certbot domains and 
nginx server names conflict and can't be the same if you just declare 
them as strings.


Specifically the config becomes this:

user nginx nginx;
pid /var/run/nginx/pid;
error_log /var/log/nginx/error.log error;
events { }
http {
     client_body_temp_path /var/run/nginx/client_body_temp;
     proxy_temp_path /var/run/nginx/proxy_temp;
     fastcgi_temp_path /var/run/nginx/fastcgi_temp;
     uwsgi_temp_path /var/run/nginx/uwsgi_temp;
     scgi_temp_path /var/run/nginx/scgi_temp;
     access_log /var/log/nginx/access.log;
     include 
/gnu/store/fz71kvlyhzg6rq0whxr11xbxzfb18igi-nginx-1.23.3/share/nginx/conf/mime.types;

     server {
       listen 80;
       listen 443 ssl;
       server_name www.joinjabber.org joinxmpp.org www.joinxmpp.org ;
       ssl_certificate /etc/letsencrypt/live/joinjabber.org/fullchain.pem;
       ssl_certificate_key /etc/letsencrypt/live/joinjabber.org/privkey.pem;
       root /srv/http;
       index index.html ;
       server_tokens off;

       location /support {
         return 302 "https://chat.joinjabber.org/#/guest?join=support";
       }

       return 301 https://joinjabber.org$request_uri;
     }
     server {
       listen 80;
       listen [::]:80;
       server_name joinjabber.org www.joinjabber.org joinxmpp.org 
www.joinxmpp.org ;
       root /srv/http;
       index index.html ;
       server_tokens off;

       location /.well-known {
         root /var/www;
       }
       location / {
         return 301 https://$host$request_uri;
       }

     }

}

With certbot adding the second server block. I am going to open a bug 
report for all of this. so we can track it.


MSavoritias

On 5/3/24 18:43, Jack Hill wrote:
> On Fri, 3 May 2024, MSavoritias wrote:
>
>> Hey,
>>
>>
>> I spent the better part of the day today trying to debug an error 
>> that was happening while deploying a server with two server names in 
>> the nginx field.
>>
>> My config was:
>>
>>      (service certbot-service-type
>>           (certbot-configuration
>>            (certificates
>>         (list
>>          (certificate-configuration
>>           ;; The first domain provided will be the subject CN of the 
>> certificate, and all domains will be Subject Alternative Names on the 
>> certificate.
>>           (domains '("joinxmpp.org" "www.joinxmpp.org")))))))
>>      (service nginx-service-type
>>           (nginx-configuration
>>            (server-blocks
>>         (list (nginx-server-configuration
>>                (server-name '("joinxmpp.org www.joinxmpp.org"))
>>                (ssl-certificate "/etc/certs/joinxmpp.org/fullchain.pem")
>>                (ssl-certificate-key 
>> "/etc/certs/joinxmpp.org/privkey.pem")
>>                (locations
>>             (list
>>              (nginx-location-configuration
>>               (uri "/support")
>>               (body '("return 302 
>> \"https://chat.joinjabber.org/#/guest?join=support\";")))))))))))
>>
>>
>> turns out that this cant be deployed at once on the server. although 
>> it is correct. The reason is that certbot/letsencrypt will complain 
>> with the message:
>>
>> Hint: The Certificate Authority failed to download the temporary 
>> challenge files created by Certbot. Ensure that the listed domains 
>> serve their content from the provided --webroot-path/-w and that 
>> files created there can be downloaded from the internet.
>>
>>
>> Mind you this error appeared on only one of the domains not both.
>>
>> This was solved by running the system once with certbot. Then 
>> rebooting with the nginx service. That idea appear to me after i 
>> found a mailing list thread from 5 years ago saying that there is a 
>> dependency cycle between nginx and certbot and they cant run at the 
>> same time the first time.
>>
>>
>> The reason I am sending an email is because I am not sure if this is 
>> something that could be fixed somehow, or if this is unavoidable and 
>> i should just send a patch revising the docs slightly to suggest that 
>> if an nginx service is used with multiple domains in the server name, 
>> you should start the system once with certbot and then reboot with 
>> nginx.
>
> Hi,
>
> Indeed, this has been an annoyance. However, if I understand your 
> problem correctly, I'm happy to report that it has now been fixed [0]! 
> The start-self-signed? filed in certificate-configuration [1] should 
> allow nginx to start before certbot has had a chance to get a proper 
> cert signed by providing a structurally valid one for nginx to use. 
> From the documentation:
>
>> start-self-signed? (default: #t)
>>  Whether to generate an initial self-signed certificate during system
>>  activation. This option is particularly useful to allow nginx to start
>>  before certbot has run, because certbot relies on nginx running to 
>> perform
>>  HTTP challenges.
>
> However, that says that the default is #t, but it still doesn't work 
> for you. I wonder what I'm missing.
>
> Best,
> Jack
>
> [0] https://issues.guix.gnu.org/46961#1
> [1] 
> https://guix.gnu.org/en/manual/devel/en/html_node/Certificate-Services.html