Thoughts on search-based imap mailboxes

Thoughts on search-based imap mailboxes

[Konstantin Ryabitsev]

Hello:

While discussing something else on the kernel.org users list, the 
question of "virtual inbox folders" came up when talking about imap and 
public-inbox. Here's how I imagine it could work in a way that doesn't 
require any kind of real user management.

- any site visitor can create a saved search against any index, which 
  would be generated with a random name (e.g. guid4) and stored on the 
  server via a simple guid->(index, search terms) mapping
- if imap is enabled, any user can also create an imap "account" that 
  would also be a guid random string, perhaps with a random password as 
  well (or no password, since guid4 strings should suffice)
- users can freely associate saved searches with these accounts, perhaps 
  optionally allowing them to save them with aliases, such that the 
  record is:
  user_guid->(saved_search_guid, alias)
- when logging into imap with an actual random account name, the 
  associated inboxes would be displayed instead of the general listing
- accounts and saved searches that were not accessed within $DAYS would 
  be automatically purged

What are your thoughts about this scheme? I like it because it respects 
users' privacy, since we cannot map accounts to developers, and because 
it doesn't require any kind of central identity gatekeeping. Anyone 
setting up a public-inbox mirror can immediately start using this scheme 
without needing to set up any kind of identity management backends.

-K

Re: Thoughts on search-based imap mailboxes

[Eric Wong]

Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> Hello:
> 
> While discussing something else on the kernel.org users list, the 

Btw, is this list public?

> question of "virtual inbox folders" came up when talking about imap and 
> public-inbox. Here's how I imagine it could work in a way that doesn't 
> require any kind of real user management.
> 
> - any site visitor can create a saved search against any index, which 
>   would be generated with a random name (e.g. guid4) and stored on the 
>   server via a simple guid->(index, search terms) mapping
> - if imap is enabled, any user can also create an imap "account" that 
>   would also be a guid random string, perhaps with a random password as 
>   well (or no password, since guid4 strings should suffice)
> - users can freely associate saved searches with these accounts, perhaps 
>   optionally allowing them to save them with aliases, such that the 
>   record is:
>   user_guid->(saved_search_guid, alias)
> - when logging into imap with an actual random account name, the 
>   associated inboxes would be displayed instead of the general listing
> - accounts and saved searches that were not accessed within $DAYS would 
>   be automatically purged
> 
> What are your thoughts about this scheme? I like it because it respects 
> users' privacy, since we cannot map accounts to developers, and because 
> it doesn't require any kind of central identity gatekeeping. Anyone 
> setting up a public-inbox mirror can immediately start using this scheme 
> without needing to set up any kind of identity management backends.

I actually considered something nearly identical several years
ago with NNTP (and GUIDs or base-64-queries as NNTP group
names).

Maybe it can be done; but temporary storage + expiration
policies would require more admin overhead and possible
permissions problems if JMAP and IMAP are run as different users.
The GUID/base64 mapping might also require a dedicated
client-side tool and might not be very usable from normal MUAs.

Right now, one of the most annoying thing for me with using
public-inbox-imapd is the inability to mark messages as Seen
using normal MUAs.

A client-side tool is likely required anyways, I'm thinking
having saved search functionality in a local tool writing to
Maildir/mbox might be the best way forward as w.r.t. scalability
and offline access (and reading + saving flags from the
Maildir/mbox).

External/detached index I'm working on for global search will
also be appropriate for client-side tooling, I think, as it's
being designed with non-public-inbox data sources in mind, too.

Re: Thoughts on search-based imap mailboxes

[Konstantin Ryabitsev]

On Fri, Oct 02, 2020 at 08:08:30PM +0000, Eric Wong wrote:
> Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> > Hello:
> > 
> > While discussing something else on the kernel.org users list, the 
> 
> Btw, is this list public?

It's not, because it's supposed to be just for people with accounts on 
kernel.org and for things like maintenance and service outage 
notifications. However, discussions routinely veer into various 
directions, as one would expect.

> A client-side tool is likely required anyways, I'm thinking
> having saved search functionality in a local tool writing to
> Maildir/mbox might be the best way forward as w.r.t. scalability
> and offline access (and reading + saving flags from the
> Maildir/mbox).
> 
> External/detached index I'm working on for global search will
> also be appropriate for client-side tooling, I think, as it's
> being designed with non-public-inbox data sources in mind, too.

Instead of a "client-side" can it just be an intermediary tool that can 
run either on the client, or on any other piece of infrastructure? The 
main reason I'm interested in that is because several people I know use 
very thin clients for their work (chromebooks) and I would like to offer 
them this service as a perk of kernel.org instead of forcing them to set 
it up and maintain on their own.

-K

Re: Thoughts on search-based imap mailboxes

[Eric Wong]

Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> On Fri, Oct 02, 2020 at 08:08:30PM +0000, Eric Wong wrote:
> > A client-side tool is likely required anyways, I'm thinking
> > having saved search functionality in a local tool writing to
> > Maildir/mbox might be the best way forward as w.r.t. scalability
> > and offline access (and reading + saving flags from the
> > Maildir/mbox).
> > 
> > External/detached index I'm working on for global search will
> > also be appropriate for client-side tooling, I think, as it's
> > being designed with non-public-inbox data sources in mind, too.
> 
> Instead of a "client-side" can it just be an intermediary tool that can 
> run either on the client, or on any other piece of infrastructure? The 
> main reason I'm interested in that is because several people I know use 
> very thin clients for their work (chromebooks) and I would like to offer 
> them this service as a perk of kernel.org instead of forcing them to set 
> it up and maintain on their own.

What are the limitations of chromebooks?  Small eMMC storage?

Can they run curl, perl and git?  I could probably work with
that; maybe just curl + POSIX sh.  I get Xapian might be
stretching it, but they probably have SQLite already (though not
the Perl bindings)

CPU and RAM on Chromebooks seem to be as good or more powerful
than the ancient Thinkpads I use.

Re: Thoughts on search-based imap mailboxes

[James Bottomley]

On Sat, 2020-10-03 at 10:50 -0400, Konstantin Ryabitsev wrote:
> On Fri, Oct 02, 2020 at 08:08:30PM +0000, Eric Wong wrote:
[...]
> > A client-side tool is likely required anyways, I'm thinking
> > having saved search functionality in a local tool writing to
> > Maildir/mbox might be the best way forward as w.r.t. scalability
> > and offline access (and reading + saving flags from the
> > Maildir/mbox).

Colour me annoyed by the inability to mark email messages as Seen too.

> > External/detached index I'm working on for global search will
> > also be appropriate for client-side tooling, I think, as it's
> > being designed with non-public-inbox data sources in mind, too.
> 
> Instead of a "client-side" can it just be an intermediary tool that
> can run either on the client, or on any other piece of
> infrastructure? The main reason I'm interested in that is because
> several people I know use very thin clients for their work
> (chromebooks) and I would like to offer them this service as a perk
> of kernel.org instead of forcing them to set  it up and maintain on
> their own.

Actually, I'd like a "server" side solution to this as well because I
want the message "seen" status to be updated on all my devices when I
read a message on any of them.  Could this be implemented as something
like a pass-through imap server which has a login infrastructure for
storing the flag metadata but sends most other queries to the backend? 
If such a beast existed I'd be happy to run it locally on my cloud
system, and I'm sure Konstantin could run it for all those of use with
kernel.org addresses.

James

Re: Thoughts on search-based imap mailboxes

[Eric Wong]

James Bottomley <James.Bottomley@HansenPartnership.com> wrote:
> On Sat, 2020-10-03 at 10:50 -0400, Konstantin Ryabitsev wrote:
> > On Fri, Oct 02, 2020 at 08:08:30PM +0000, Eric Wong wrote:
> [...]
> > > A client-side tool is likely required anyways, I'm thinking
> > > having saved search functionality in a local tool writing to
> > > Maildir/mbox might be the best way forward as w.r.t. scalability
> > > and offline access (and reading + saving flags from the
> > > Maildir/mbox).
> 
> Colour me annoyed by the inability to mark email messages as Seen too.

Yup, working on it with "lei" :>  ("local email interface")

> > > External/detached index I'm working on for global search will
> > > also be appropriate for client-side tooling, I think, as it's
> > > being designed with non-public-inbox data sources in mind, too.
> > 
> > Instead of a "client-side" can it just be an intermediary tool that
> > can run either on the client, or on any other piece of
> > infrastructure? The main reason I'm interested in that is because
> > several people I know use very thin clients for their work
> > (chromebooks) and I would like to offer them this service as a perk
> > of kernel.org instead of forcing them to set  it up and maintain on
> > their own.

The add-external + "lei q --remote" example below covers that,
I think (it doesn't cache+memoize yet, but it will).

> Actually, I'd like a "server" side solution to this as well because I
> want the message "seen" status to be updated on all my devices when I
> read a message on any of them.  Could this be implemented as something
> like a pass-through imap server which has a login infrastructure for
> storing the flag metadata but sends most other queries to the backend? 
> If such a beast existed I'd be happy to run it locally on my cloud
> system, and I'm sure Konstantin could run it for all those of use with
> kernel.org addresses.

Right now the plan is to have a way to import+export keywords(*)
or entire messages to/from existing (IMAP|Maildir|mbox) mail
sources.

I don't think there needs to be new servers; existing read-write
IMAP servers should be usable as-is.

	# import any unimported messages from a personal mailbox
	# into ~/.local/share/lei/store:
	lei import imaps://user@example.com/INBOX
	# (IMAP doesn't work yet, but mboxes do, Maildir next)

	# add LKML as a (remote) external source
	lei add-external https://lore.kernel.org/lkml/

	# search for everything imported and in remote external sources
	# Anything from remote sources will be cached/memoized locally
	# by default.
	lei q --remote -o ~/tmp/results-Maildir SEARCH_TERMS...
	# "-o imaps://..." is on horizon

	# open results-Maildir (or mbox or IMAP folder)
	$YOUR_FAVOURITE_MUA ~/tmp/results-Maildir

	# lei will continue monitoring ~/tmp/results-Maildir for
	# keyword changes in background (not implemented)

	# Nothing below is implemented, yet :<

	# send keywords changes from background monitoring to IMAP
	# source (this won't put new messages in the IMAP source)
	lei export --kw-only imaps://user@example.com/INBOX

	# if your MUA interacted from an IMAP server directly,
	# just fetch keywords:
	lei import --kw-only imaps://user@example.com/INBOX

	# via ssh, assuming $HOST has lei installed, too:
	lei export --kw-only ssh://user@$HOST/

"add-external" and "q" are somewhat usable at the moment but
will get fleshed out.  "q" has yet to gain "-o imaps://..."
support.

"lei q" will eventually support saved searches + results metadata
caching.  My goal is to have "lei q" return usable results in
<100ms if rerunning a previous query (and if the results folder
didn't get clobbered by the user).

I've barely started working on "lei import" (only mboxes so
far), and haven't started on export, yet...

WARNING:

	"q" matches mairix(1) behavior by clobbering the output
	folder before writing results (-a/--augment disables it
	just as it does in mairix(1).

	As a safety feature, I will probably have "q" check and
	import any edited or uncached messages into its internal
	store.  "lei import" needs to be fleshed out, first.

	The first release will not support removing any
	imported/cached messages at all.

(*) IMAP uses "flags", JMAP calls them "keywords".  I'm trying
    to favor "keywords" since flags may be limited to the few
    flags supported by mbox/Maildir formats.  I've used "flags"
    for a long time and habits are hard-to-break.