Improving how NGINX modules are used and built

Improving how NGINX modules are used and built

[mirai]

Consider the following nginx-configuration modules field snippet from Guix docs:

--8<---------------cut here---------------start------------->8---
(modules
 (list
  (file-append nginx-accept-language-module "\
/etc/nginx/modules/ngx_http_accept_language_module.so")
  (file-append nginx-lua-module "\
/etc/nginx/modules/ngx_http_lua_module.so")))
--8<---------------cut here---------------end--------------->8---

It is unclear at all how an arbitrary nginx module should be added given that
one needs to append the package object with a path to the modules .so file.
This .so name can only be found by listing the files from the package object
which adds further obscurity to what should have been:

--8<---------------cut here---------------start------------->8---
(modules
 (list nginx-accept-language-module nginx-lua-module))
--8<---------------cut here---------------end--------------->8---

An oddity of how nginx modules are packaged in guix is that they all place
the .so file under /etc/nginx/modules which is an odd directory to place library object files.
Compared against other distributions, they tend to place the .so files under /usr/lib64/nginx
and a .conf file under /usr/share/nginx/modules that contains

--8<---------------cut here---------------start------------->8---
load_module "/usr/lib64/nginx/modules/ngx_foo_module.so";
--8<---------------cut here---------------end--------------->8---

My guess is that the intention behind
(file-append nginx-foo-module "/etc/nginx/modules/ngx_foo.so") is that
it was supposed to replicate the .conf behavior but inadvertently ended up being
used as a place for .so files.

Looking at how network-manager-configuration handles its vpn-plugins field, it seems doable
that a similar approach can be used here.
The existing nginx-modules should be changed to install their .so files under lib{64}/nginx
instead and they should drop a etc/nginx/modules/foo_module.conf file responsible for loading
the module from the .so file. Including modules through a .conf should be preferred as
there's no guarantee that a module is a .so file or that it is always a
_single_ .so file but in general this file should typically be a one-line .conf file containing:

--8<---------------cut here---------------start------------->8---
load_module "/gnu/store/......nginx-foo-module/usr/lib64/nginx/ngx_foo_module.so";
--8<---------------cut here---------------end--------------->8---

And nginx-configuration should serialize the modules field as a series of lines including
the module .conf files, that is:

--8<---------------cut here---------------start------------->8---
include "/gnu/store/......nginx-foo-module/etc/nginx/modules/ngx_foo_module.conf";
include "/gnu/store/......nginx-bar-module/etc/nginx/modules/ngx_bar_module.conf";
...
--8<---------------cut here---------------end--------------->8---
(note: a directory union could be used here as an alternative)


On a related note, given how nginx-modules are expected to be built and linked against the
same nginx build, a new nginx-build-system for nginx modules could be considered as the few
existing nginx modules in guix are for the most part a copy-paste of each other with some
modifications here-and-there.

Thoughts?

Regards,
Bruno

Re: Improving how NGINX modules are used and built

[Ludovic Courtès]

Hello!

mirai <mirai@makinata.eu> skribis:

> An oddity of how nginx modules are packaged in guix is that they all place
> the .so file under /etc/nginx/modules which is an odd directory to place library object files.

To me that should be treated as a bug.  Those .so files should go to
$PKG/lib/nginx instead, or something similar.

> Looking at how network-manager-configuration handles its vpn-plugins field, it seems doable
> that a similar approach can be used here.
> The existing nginx-modules should be changed to install their .so files under lib{64}/nginx
> instead and they should drop a etc/nginx/modules/foo_module.conf file responsible for loading
> the module from the .so file. Including modules through a .conf should be preferred as
> there's no guarantee that a module is a .so file or that it is always a
> _single_ .so file but in general this file should typically be a one-line .conf file containing:
>
> load_module "/gnu/store/......nginx-foo-module/usr/lib64/nginx/ngx_foo_module.so";
>
>
> And nginx-configuration should serialize the modules field as a series of lines including
> the module .conf files, that is:
>
> include "/gnu/store/......nginx-foo-module/etc/nginx/modules/ngx_foo_module.conf";
> include "/gnu/store/......nginx-bar-module/etc/nginx/modules/ngx_bar_module.conf";
> ...
>
> (note: a directory union could be used here as an alternative)

I’d say that ideally, one could extend <nginx-configuration> with
modules, and that would automatically create the ‘load_module’
statements.

> On a related note, given how nginx-modules are expected to be built and linked against the
> same nginx build, a new nginx-build-system for nginx modules could be considered as the few
> existing nginx modules in guix are for the most part a copy-paste of each other with some
> modifications here-and-there.

That would be an improvement, indeed!

Thanks,
Ludo’.

Re: Improving how NGINX modules are used and built

[Bruno Victal]

On 2023-01-09 10:51, Ludovic Courtès wrote:
> Hello!
> 
> mirai <mirai@makinata.eu> skribis:
> 
>> An oddity of how nginx modules are packaged in guix is that they all place
>> the .so file under /etc/nginx/modules which is an odd directory to place library object files.
> 
> To me that should be treated as a bug.  Those .so files should go to
> $PKG/lib/nginx instead, or something similar.

Fixing this bug is likely to cause pain to existing module users, as the path to these .so
files is explicitly passed to <nginx-configuration> and is what's documented in the Guix manual.
<... continued below ...>

>> Looking at how network-manager-configuration handles its vpn-plugins field, it seems doable
>> that a similar approach can be used here.
>> The existing nginx-modules should be changed to install their .so files under lib{64}/nginx
>> instead and they should drop a etc/nginx/modules/foo_module.conf file responsible for loading
>> the module from the .so file. Including modules through a .conf should be preferred as
>> there's no guarantee that a module is a .so file or that it is always a
>> _single_ .so file but in general this file should typically be a one-line .conf file containing:
>>
>> load_module "/gnu/store/......nginx-foo-module/usr/lib64/nginx/ngx_foo_module.so";
>>
>>
>> And nginx-configuration should serialize the modules field as a series of lines including
>> the module .conf files, that is:
>>
>> include "/gnu/store/......nginx-foo-module/etc/nginx/modules/ngx_foo_module.conf";
>> include "/gnu/store/......nginx-bar-module/etc/nginx/modules/ngx_bar_module.conf";
>> ...
>>
>> (note: a directory union could be used here as an alternative)
> 
> I’d say that ideally, one could extend <nginx-configuration> with
> modules, and that would automatically create the ‘load_module’
> statements.

A change here would really improve how modules are used (the current way of things in Guix is:
for beginners: "guess the module path", for the seasoned: "build and list the package output
directory").

Inevitably, this won't be a backward compatible change unless we yet add another
deprecation warning and a new field "temporarily" (read as: permanently). 
(or maybe using match to differentiate between strings and file-like objects?)

I'm not convinced that <nginx-configuration> should be generating load_module statements here,
these should be generated by the module-package itself into a .conf file and <nginx-configuration>
generates a include statement for it. Reason being that nothing stops a module being comprised
of several .so files or be a "pseudo-module", that is, it's a .conf snippet to be included.

I haven't encountered modules like these yet but there's nothing saying that they can't be done this way.


Cheers,
Bruno

Re: Improving how NGINX modules are used and built

[Ludovic Courtès]

Hi,

Bruno Victal <mirai@makinata.eu> skribis:

> On 2023-01-09 10:51, Ludovic Courtès wrote:
>> Hello!
>> 
>> mirai <mirai@makinata.eu> skribis:
>> 
>>> An oddity of how nginx modules are packaged in guix is that they all place
>>> the .so file under /etc/nginx/modules which is an odd directory to place library object files.
>> 
>> To me that should be treated as a bug.  Those .so files should go to
>> $PKG/lib/nginx instead, or something similar.
>
> Fixing this bug is likely to cause pain to existing module users, as the path to these .so
> files is explicitly passed to <nginx-configuration> and is what's documented in the Guix manual.

Right.  We could imagine a transition period during which nginx module
packages make their .so available under $PKG/lib/nginx primarily, but
keep $PKG/etc as a symlink to that.

[...]

>>> And nginx-configuration should serialize the modules field as a series of lines including
>>> the module .conf files, that is:
>>>
>>> include "/gnu/store/......nginx-foo-module/etc/nginx/modules/ngx_foo_module.conf";
>>> include "/gnu/store/......nginx-bar-module/etc/nginx/modules/ngx_bar_module.conf";
>>> ...
>>>
>>> (note: a directory union could be used here as an alternative)
>> 
>> I’d say that ideally, one could extend <nginx-configuration> with
>> modules, and that would automatically create the ‘load_module’
>> statements.

[...]

> I'm not convinced that <nginx-configuration> should be generating load_module statements here,
> these should be generated by the module-package itself into a .conf file and <nginx-configuration>
> generates a include statement for it. Reason being that nothing stops a module being comprised
> of several .so files or be a "pseudo-module", that is, it's a .conf snippet to be included.

Yes, that sounds even better.  From the admin’s viewpoint, it’s kind of
the same anyway: you list a bunch of packages to use as nginx modules,
and it “just works”, with the appropriate ‘load_module’ ending up
directly or indirectly in the nginx config file.

Thanks,
Ludo’.