watch a simple dir

watch a simple dir

[Konstantin Ryabitsev]

Hello:

I'm playing around with using public-inbox as the archiving subsystem for
mlmmj. I know it's possible to simply configure a local address to deliver to
public-inbox-mda [1], but I wonder if there's a way to reduce complexity and
simply configure public-inbox-watch to monitor mlmmj's "archive" directory for
any new files, similar to how it would monitor a maildir:

- it's a simple flat dir of numeric files corresponding to the number of the
  message in the index
- each message is a valid rfc2822 document -- in fact, if I copy them into the
  "new" folder of any maildir, I can read them with mutt
- however, if I use symlink trickery to make mlmmj deliver into "new" or "cur"
  of a maildir monitored by public-inbox-watch, they are ignored -- so things
  are not as easy as that. :) Probably, it expects to have more complex
  filenames instead of just a number, but I didn't dig too deep.

So, any way to make this work either by tricking public-inbox-watch to
recognize these as valid maildir entries, or by teaching it to monitor simple
additions to a dir as parallel scheme to maildir:?

-K

[1] adding an extra hop to deliver to public-inbox-mda, or to a maildir
monitored by public-inbox-watch adds another entry into the postfix queue,
bloats the headers by adding more Received: lines and generally seems like an
inefficient way to basically copy a file that already exists on the filesystem

Re: watch a simple dir

[Konstantin Ryabitsev]

On Fri, 26 Feb 2021 at 09:38, Konstantin Ryabitsev
<konstantin@linuxfoundation.org> wrote:
> - it's a simple flat dir of numeric files corresponding to the number of the
>   message in the index
> - each message is a valid rfc2822 document -- in fact, if I copy them into the
>   "new" folder of any maildir, I can read them with mutt
> - however, if I use symlink trickery to make mlmmj deliver into "new" or "cur"
>   of a maildir monitored by public-inbox-watch, they are ignored -- so things
>   are not as easy as that. :) Probably, it expects to have more complex
>   filenames instead of just a number, but I didn't dig too deep.

Ah, looks like this was only a limitation of v1.6.1 -- master doesn't
require that filenames in new/ contain a :2 (which is not a
requirement for maildir files in "new/" -- they merely need to be
unique; the :2, part is required for cur/ entries for adding flags).

So, this will work out just fine as long as this behaviour doesn't go
back to what is in 1.6.1 (which it shouldn't). Simply symlinking
/var/spool/mlmmj/<listname>/archive to <some-maildir-elsewhere>/new is
sufficient to automatically add mlmmj messages to the public-inbox
archive and, as an added bonus, this will properly respect
control/noarchive settings.

Me pleased.

-K

Re: watch a simple dir

[Eric Wong]

Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> Hello:
> 
> I'm playing around with using public-inbox as the archiving subsystem for
> mlmmj. I know it's possible to simply configure a local address to deliver to
> public-inbox-mda [1], but I wonder if there's a way to reduce complexity and
> simply configure public-inbox-watch to monitor mlmmj's "archive" directory for
> any new files, similar to how it would monitor a maildir:
> 
> - it's a simple flat dir of numeric files corresponding to the number of the
>   message in the index
> - each message is a valid rfc2822 document -- in fact, if I copy them into the
>   "new" folder of any maildir, I can read them with mutt

It's actually an MH directory w/o .mh_sequences.  I was actually
just looking into MH support the other night for lei(*); and
ezmlm (which mlmmj is based on) seems to do a similar thing.

> - however, if I use symlink trickery to make mlmmj deliver into "new" or "cur"
>   of a maildir monitored by public-inbox-watch, they are ignored -- so things
>   are not as easy as that. :) Probably, it expects to have more complex
>   filenames instead of just a number, but I didn't dig too deep.

> So, any way to make this work either by tricking public-inbox-watch to
> recognize these as valid maildir entries, or by teaching it to monitor simple
> additions to a dir as parallel scheme to maildir:?

Adding ":" to the end of the link should work for "new"...

public-inbox.git actually just relaxed the check for "new" to
not require the ":" which I believe is more correct.

"cur" requires a ":2,$FLAG_CHARS" suffix.

> -K
> 
> [1] adding an extra hop to deliver to public-inbox-mda, or to a maildir
> monitored by public-inbox-watch adds another entry into the postfix queue,
> bloats the headers by adding more Received: lines and generally seems like an
> inefficient way to basically copy a file that already exists on the filesystem

Understood.  I'll probably add MH support to -watch after it
hits lei.  But I've also got a lot on my plate and probably not
a lot of time to do it...


On a side note; MH is actually the least parallel-friendly of
the mail formats for writing.  I'm not sure if there's any
standardized locking scheme for it.

At least mutt has no locking for writing .mh_sequences, so
it seems possible for parallel process to clobber per-message
flags.

Choosing the next sequence number for delivery seems to require
a O(n) readdir scan for every message delivered (and "n"
increases for every new message).

mlmmj has an archive/../index file to keep the highest number,
at least; but most implementations do not.