Re: public-inbox-init with minimal info

[Eric Wong <e@80x24.org>]

Alyssa Ross <hi@alyssa.is> wrote:
> >> Since the V2 initialization isn't encapsulated in one easy command, I'm
> >> wondering what the best way to accomplish initialization without writing
> >> a config file or asking for unnecessary information is.  I could just
> >> run public-inbox-init with PI_CONFIG=/dev/null, but then it's still not
> >> clear to me what information about the mailbox the script requires to be
> >> able to initialize the mailbox.  Looking at the code, I see that at
> >> least the primary address is passed to PublicInbox::Inbox, but I'm not
> >> sure what that would actually be used for inside the inbox.
> >
> > That address is used for making commits using the
> > public-inbox-{learn,edit,purge} commands, and also matching
> > with public-inbox-mda for delivery.
> 
> I'm guessing those read from the config file, though?  I'm trying to
> figure out what configuration is stored in the inbox directory as
> opposed to the config file.

Yes, they all read from the config file because they need to
operate on inboxes.   public-inbox-{edit,purge,index,xcpdb,compact}
can also operate on inboxes outside of the config file, but -mda
and -learn needs the config.

There isn't any config stored in the inbox directories
themselves, although there's some metadata in SQLite for
incremental indexing and indexlevel for Xapian.

> > You want to provide that inbox during the post-install?
> >
> > I'm not sure if I understand what's going on (but it's been a
> > long day :x).  I'm not sure if providing an inbox on package
> > installation is necessary or even a good thing...
> >
> > Using git itself as an analogy: I wouldn't expect installing git
> > to auto-generate an empty git repo for me.  So same with
> > public-inbox, and stuff like cgit/gitweb...
> 
> I agree -- creating an inbox on package installation would be a terrible
> idea, and is not what I'm proposing to do.

Good to know :>

> Some background: in the Nix ecosystem, we have a package repository,
> Nixpkgs.  These packages are fairly typical except for unusual paths
> (containing checksums of the "inputs" -- think dependencies -- of the
> package) and the functional language they're written in.  We also have
> NixOS, which is a GNU/Linux distribution, that uses those packages but
> also adds the concept of "modules", written in the same language as the
> packages.  These modules do things like configuring the users on your
> system, setting up config files, etc.  The idea being that you can
> generate a whole Linux system from pure (in the functional programming
> sense) configuration, reproducibly, and have the system stuff be read
> only at runtime to the maximum extent possible.  I'm working on both a
> package _and_ a module for public-inbox.  The package will do exactly
> what you'd expect a package to do, but the module will let you express
> global and per-inbox configuration in the Nix language, generate a
> read-only public-inbox config file and systemd units from those, and, at
> boot time or configuration change time, initialize the inboxes defined
> by the user.

Thanks for that clarification, NixOS modules are a new concept
to me.

I'm not sure how the public-inbox config file can/should remain
read-only.  It's analogous to a config file for cgit or gitweb,
so maybe modules for those can offer some inspiration...

> As an example, Tor in NixOS looks like this:
> 
>     { ... }:
> 
>     {
>       services.tor.enable = true;
>       services.tor.hiddenServices = [
>         {
>           name = "public-inbox";
>           map = [ { port = 80; destination = 8000; } ];
>           version = 3;
>         };
>       ];
>     };
> 
> This will generate a static tor.service and tor config file -- we do as
> much as possible staticly and purely because then we know that if a
> configuration is rolled back, we can remove the service etc.  For state,
> however, like the hidden service private key (in this case -- I could
> have used a static one here if I'd wanted), it will be generated either
> at boot time or in Tor's ExecStartPre.  This is the same mechanism I
> would use to run public-inbox-init.

So that means a new tor process is spawned for every hidden
service?  (instead of one tor process running multiple
services).  It works, but it's not memory-efficient (but
could be more secure if tor has bugs).

Like gitweb and cgit, public-inbox-{httpd,nntpd} usually expects
to serve multiple inboxes off one instance.  It's possible to
start an independent -httpd/-nntpd instance for every inbox,
each with it's own config, but that's not efficient at all.

Both the WWW and NNTP code are able to scan Message-IDs across
multiple inboxes in the same process in case of cross-posts
between related inboxes.  At some point, I think it'll be useful
to define groups/relationships between inboxes for the WWW UI.

Likewise, public-inbox-watch would be expected to watch multiple
Maildirs and write to multiple inboxes, if needed.  Maybe
modules for mail downloading/notification tools such as
offlineimap/mbsync(isync) could serve as inspiration, there.

I tried to keep most of the daemon-specific config knobs in the
command-line, so it goes into .service files, which can be
read-only (well, unless somebody wants to change worker
processes via "-W $NUM").

But yeah, packaging services/modules for different systems and
use-cases is hard, everybody seems to do it differently and
want different things.  So I'm really not sure how packaging
a module would work, it seems like that's something each user
would want to manage on their own.