Questions about Public Inbox setup and configuration

Questions about Public Inbox setup and configuration

[Soft Works]

Hi,

I have set up public inbox on a Debian system, intended to mirror just 
a single mailing list, separate and independent form other PI mirrors.

Getting all information together wasn’t as easy as it could be, but eventually
I got it all working.

Though, I have a number of questions to which I couldn’t find an answer.

1. At the top of the page it says "($INBOX_DIR/description missing)", even 
   though the maildir contains a description file.
   (interestingly, several mirrors are 'suffering' for the same problem)
   How can I fix this?

2. How are the HTML views generated? Are there any templates that I could
   modify?

3. When a message cannot be found, it shows a list of mirrors to check for 
   that message. How can I disable this?
   (it's more than unlikely that any of them would have it)

4. The services are running behind NGINX which forwards the requests to 
   a local-only port. Now the www interface is showing URLs at many places 
   like http://localhost:8123/mailbox instead of the public URL.
   How can I set this up correctly?

Kind regards,
softworkz 

Re: Questions about Public Inbox setup and configuration

[Kyle Meyer]

Soft Works writes:

> Though, I have a number of questions to which I couldn’t find an answer.
>
> 1. At the top of the page it says "($INBOX_DIR/description missing)", even 
>    though the maildir contains a description file.
>    (interestingly, several mirrors are 'suffering' for the same problem)
>    How can I fix this?

Hmm, maildir?  Perhaps the issue is that you're placing this file in the
wrong place.  The description file for each inbox should be at the
directory pointed to by publicinbox.$name.inboxdir in
~/.public-inbox/config.  The first time you add it, you may need to
restart or send a SIGHUP to the httpd service, but my understanding is
that should be unnecessary as of v1.7.0 (specifically commit b9e7ecbb,
inbox: drop memoization/preload, cleanup expires caches).

> 2. How are the HTML views generated? Are there any templates that I could
>    modify?

No, I don't think there are any templates you can modify.  In order to
see how the HTML is generated, perhaps lib/PublicInbox/WWW*.pm files and
lib/PublicInbox/View.pm would be the best source files to start with.

> 3. When a message cannot be found, it shows a list of mirrors to check for 
>    that message. How can I disable this?
>    (it's more than unlikely that any of them would have it)

Those URLs are hard coded in lib/PublicInbox/ExtMsg.pm, and there's no
way to change them or disable their display via configuration yet.
However, there is a TODO comment in that file about making the list
user-configurable, so I imagine Eric would be open to a patch.

> 4. The services are running behind NGINX which forwards the requests to 
>    a local-only port. Now the www interface is showing URLs at many places 
>    like http://localhost:8123/mailbox instead of the public URL.
>    How can I set this up correctly?

Have you tried something like

  proxy_set_header    HOST $host;

?

See public-inbox's examples/nginx_proxy for an example.

RE: Questions about Public Inbox setup and configuration

[Soft Works]

> -----Original Message-----
> From: Kyle Meyer <kyle@kyleam.com>
> Sent: Sunday, December 26, 2021 9:14 PM
> To: Soft Works <softworkz@hotmail.com>
> Cc: meta@public-inbox.org
> Subject: Re: Questions about Public Inbox setup and configuration
> 
> Soft Works writes:
> 
> > Though, I have a number of questions to which I couldn’t find an answer.
> >
> > 1. At the top of the page it says "($INBOX_DIR/description missing)", even
> >    though the maildir contains a description file.
> >    (interestingly, several mirrors are 'suffering' for the same problem)
> >    How can I fix this?
> 
> Hmm, maildir?  Perhaps the issue is that you're placing this file in the
> wrong place.  The description file for each inbox should be at the
> directory pointed to by publicinbox.$name.inboxdir in
> ~/.public-inbox/config.  

Yup, I've been in fact looking at the wrong dir. I thought it must be the
right one as it already had a description file and the git inbox dir 
didn't. Maybe it should be automatically created by 'initialize'?
(given that many other mirrors seem to have the same issue)

I think the most confusing part was the $INBOX_DIR in the message, which 
makes you think it's an environment variable (which isn't set).

> The first time you add it, you may need to
> restart or send a SIGHUP to the httpd service, but my understanding is
> that should be unnecessary as of v1.7.0 (specifically commit b9e7ecbb,
> inbox: drop memoization/preload, cleanup expires caches).

Interestingly it was the other way round: it noted the change when adding
the description file, but it didn’t update on change, neither on rename
(and rename back). Not that this would matter in any way, though ;-)

Thanks a lot for pointing me in the right direction!

 
> > 2. How are the HTML views generated? Are there any templates that I could
> >    modify?
> 
> No, I don't think there are any templates you can modify.  In order to
> see how the HTML is generated, perhaps lib/PublicInbox/WWW*.pm files and
> lib/PublicInbox/View.pm would be the best source files to start with.
> 
> > 3. When a message cannot be found, it shows a list of mirrors to check for
> >    that message. How can I disable this?
> >    (it's more than unlikely that any of them would have it)
> 
> Those URLs are hard coded in lib/PublicInbox/ExtMsg.pm, and there's no
> way to change them or disable their display via configuration yet.
> However, there is a TODO comment in that file about making the list
> user-configurable, so I imagine Eric would be open to a patch.

Looking at the code files made me remember why I never got warm 
with Perl ;-)


> > 4. The services are running behind NGINX which forwards the requests to
> >    a local-only port. Now the www interface is showing URLs at many places
> >    like http://localhost:8123/mailbox instead of the public URL.
> >    How can I set this up correctly?
> 
> Have you tried something like
> 
>   proxy_set_header    HOST $host;
> 
> ?
> 
> See public-inbox's examples/nginx_proxy for an example.

I had seen the other example for running the httpd via systemd,
but I missed that one.

I think it would be really helpful when the documentation for the 
individual components would include references to the examples in that
folder. This would have saved me a lot of time as I had found the
other examples (for httpd setup) only accidentally and after quite
some time.

Anyway, I got it all set up and running well (scratch 2+3) and
big thanks to you for helping me!

Kind regards,
softworkz

Re: Questions about Public Inbox setup and configuration

[Eric Wong]

Soft Works <softworkz@hotmail.com> wrote:
> > -----Original Message-----
> > From: Kyle Meyer <kyle@kyleam.com>
> > Sent: Sunday, December 26, 2021 9:14 PM
> > To: Soft Works <softworkz@hotmail.com>
> > Cc: meta@public-inbox.org
> > Subject: Re: Questions about Public Inbox setup and configuration
> > 
> > Soft Works writes:
> > 
> > > Though, I have a number of questions to which I couldn’t find an answer.
> > >
> > > 1. At the top of the page it says "($INBOX_DIR/description missing)", even
> > >    though the maildir contains a description file.
> > >    (interestingly, several mirrors are 'suffering' for the same problem)
> > >    How can I fix this?
> > 
> > Hmm, maildir?  Perhaps the issue is that you're placing this file in the
> > wrong place.  The description file for each inbox should be at the
> > directory pointed to by publicinbox.$name.inboxdir in
> > ~/.public-inbox/config.  
> 
> Yup, I've been in fact looking at the wrong dir. I thought it must be the
> right one as it already had a description file and the git inbox dir 
> didn't. Maybe it should be automatically created by 'initialize'?
> (given that many other mirrors seem to have the same issue)
> 
> I think the most confusing part was the $INBOX_DIR in the message, which 
> makes you think it's an environment variable (which isn't set).

I suppose using `publicinbox.FOO.inboxdir' as a placeholder
instead of `$INBOX_DIR' in the error message could help.

> > The first time you add it, you may need to
> > restart or send a SIGHUP to the httpd service, but my understanding is
> > that should be unnecessary as of v1.7.0 (specifically commit b9e7ecbb,
> > inbox: drop memoization/preload, cleanup expires caches).
> 
> Interestingly it was the other way round: it noted the change when adding
> the description file, but it didn’t update on change, neither on rename
> (and rename back). Not that this would matter in any way, though ;-)
> 
> Thanks a lot for pointing me in the right direction!
> 
>  
> > > 2. How are the HTML views generated? Are there any templates that I could
> > >    modify?
> > 
> > No, I don't think there are any templates you can modify.  In order to
> > see how the HTML is generated, perhaps lib/PublicInbox/WWW*.pm files and
> > lib/PublicInbox/View.pm would be the best source files to start with.

CSS can be changed, examples are in contrib/css/.  CSS can do
all sorts of gnarly things beyond colors, but I've only done the
basic stuff which small browsers like dillo|netsurf can handle.

There's no HTML templates, but everybody who can run the Perl
code can edit+change it.  I've used Template Toolkit in other
projects, but I'm prioritizing fewer dependencies for
ease-of-installation and memory savings (it needs to be able to
stream 1000+ message threads with for travelers with temporary
Internet).

I'm actually planning to even more of the HTML generation and
buffering to be gzip-dependent to save memory on the server.

> > > 3. When a message cannot be found, it shows a list of mirrors to check for
> > >    that message. How can I disable this?
> > >    (it's more than unlikely that any of them would have it)
> > 
> > Those URLs are hard coded in lib/PublicInbox/ExtMsg.pm, and there's no
> > way to change them or disable their display via configuration yet.
> > However, there is a TODO comment in that file about making the list
> > user-configurable, so I imagine Eric would be open to a patch.
> 
> Looking at the code files made me remember why I never got warm 
> with Perl ;-)

Yeah I understand :>.  I mainly favor Perl5 because I hate
installing+updating software; and it's already installed on most
platforms (especially amongst git, Debian, OpenBSD and Linux
kernel hackers).

Python and Ruby are constant treadmills of incompatibilities and
deprecations (and slower startup time).

Using C in more places would be nice, but gcc or clang are both
gigantic dependencies; and tcc (tinycc) hardly sees releases.
It's also a goal to minimize the barrier for users becoming
developers[1]; so the only C/C++/Rust/etc code I'd accept would
be would be built just-ahead-of-time on users' systems.

> > > 4. The services are running behind NGINX which forwards the requests to
> > >    a local-only port. Now the www interface is showing URLs at many places
> > >    like http://localhost:8123/mailbox instead of the public URL.
> > >    How can I set this up correctly?
> > 
> > Have you tried something like
> > 
> >   proxy_set_header    HOST $host;
> > 
> > ?
> > 
> > See public-inbox's examples/nginx_proxy for an example.
> 
> I had seen the other example for running the httpd via systemd,
> but I missed that one.
> 
> I think it would be really helpful when the documentation for the 
> individual components would include references to the examples in that
> folder. This would have saved me a lot of time as I had found the
> other examples (for httpd setup) only accidentally and after quite
> some time.

Do you have any specific patches or updates you can propose?
There's a bunch of stuff which I consider second nature and
often assume is widely known...

I haven't used nginx in many years, now[2], and my plan is to
allow reverse proxying in public-inbox-httpd so it could be:

  [public Internet]<--   -->[tor]
                      \ /
                       V
                       |
                       v
            public-inbox-httpd (HTTPS)
                       ^
                       |
                       v
                    Varnish
                       ^
                       |
                       v
         public-inbox-httpd (PublicInbox::WWW)

There'd only need to be one public-inbox-httpd instance but it'd
talk to itself through varnish.  And perhaps varnish can be
eliminated via (optional) Cache::FastMmap, even...

> Anyway, I got it all set up and running well (scratch 2+3) and
> big thanks to you for helping me!

OK, good to know :>

[1] copyleft licenses become pointless when there's a gigantic resource
    requirement gap between running and modifying software.

[2] I currently use a Ruby HTTPS reverse proxy, but I intend to replace
    it with a Perl5 one.