release timelines (-extindex, JMAP, lei)

release timelines (-extindex, JMAP, lei)

[Eric Wong]

So I think the -extindex stuff might be stable and suitable for
general consumption.  The HTML WWW UI around -extindex has some
rough edges but nothing that would take too much effort to fix.

But I'm deeply worried about unleashing a new on-disk format
that's insufficient and being stuck supporting it forever
(as I am with v1 inboxes)...

* JMAP is going to be a more effort, but I think our current
  on-disk data model is OK or at least extensible enough for it.
  I might delay JMAP until 1.8.

  JMAP will be significantly less effort than inventing something
  new (and one-off) with GraphQL or REST.  And it would be less
  effort for client authors, as well; since client code can be
  used for non-public-inbox servers, too.

  I'm excited that it allows vendor extensions, which means our
  git-show|solver /$INBOX_URL/$OBJECT_ID/s/ blob-reconstruction
  endpoint can be exposed via JMAP.

* lei saved searches can probably done quick for 1.7,
  but without full keywords support for externals...

* Dealing with per-message keywords and externals in lei
  is going to be tough and delayed, I think:
  https://public-inbox.org/meta/20210224204950.GA2076@dcvr/
  ("lei: per-message keywords and externals")
  It may involve spinnning up a Python daemon to use
  custom PostingSource since Search::Xapian doesn't support it;
  and I don't expect anybody outside of FreeBSD to use SWIG Xapian.pm.

* lei has a bunch of rough edges and I'm not comfortable declaring
  it as supported, especially when there's a risk of data loss to
  users.

* .mailmap + Xapian synonym support would be nice in lei and
  HTTP/IMAP/JMAP endpoints

In any case, I need to take a few days away to clear my head.

Re: release timelines (-extindex, JMAP, lei)

[Kyle Meyer]

Eric Wong writes:

> But I'm deeply worried about unleashing a new on-disk format
> that's insufficient and being stuck supporting it forever
> (as I am with v1 inboxes)...
[...]
> * lei has a bunch of rough edges and I'm not comfortable declaring
>   it as supported, especially when there's a risk of data loss to
>   users.

What are your thoughts on marking all things lei with a big
experimental / subject-to-change / may-eat-your-data warning?

Re: release timelines (-extindex, JMAP, lei)

[Eric Wong]

Kyle Meyer <kyle@kyleam.com> wrote:
> Eric Wong writes:
> 
> > But I'm deeply worried about unleashing a new on-disk format
> > that's insufficient and being stuck supporting it forever
> > (as I am with v1 inboxes)...
> [...]
> > * lei has a bunch of rough edges and I'm not comfortable declaring
> >   it as supported, especially when there's a risk of data loss to
> >   users.
> 
> What are your thoughts on marking all things lei with a big
> experimental / subject-to-change / may-eat-your-data warning?

Yes, definitely experimental and may-eat-your-data.
I hope the subject-to-change bit can be minimized, though I'll
drop the '<>' in "lei q" JSON before release.

I'm also considering diverging from mairix in making --augment
the default behavior in "lei q".  That would make things
significantly safer, and --no-augment would be required to
clobber existing outputs.

Re: release timelines (-extindex, JMAP, lei)

[Konstantin Ryabitsev]

On Fri, Mar 05, 2021 at 10:20:19PM +0000, Eric Wong wrote:
> So I think the -extindex stuff might be stable and suitable for
> general consumption.  The HTML WWW UI around -extindex has some
> rough edges but nothing that would take too much effort to fix.

\o/

> But I'm deeply worried about unleashing a new on-disk format
> that's insufficient and being stuck supporting it forever
> (as I am with v1 inboxes)...
> 
> * JMAP is going to be a more effort, but I think our current
>   on-disk data model is OK or at least extensible enough for it.
>   I might delay JMAP until 1.8.
> 
>   JMAP will be significantly less effort than inventing something
>   new (and one-off) with GraphQL or REST.  And it would be less
>   effort for client authors, as well; since client code can be
>   used for non-public-inbox servers, too.

I don't really have any specific opinion on this matter. I don't really know
of any other provider outside of Fastmail that uses JMAP, but it *is* a
published IETF standard around RFC2822 messages, so it makes sense to use it
for this purpose.

> * lei saved searches can probably done quick for 1.7,
>   but without full keywords support for externals...

I was wondering if lei should be part of the same suite as public-inbox proper
as opposed to a standalone (or interdependent) client tool. Unless I'm
mistaken, someone running a public-inbox origin or mirror server wouldn't
necessarily be an active lei user. Decoupling them from each-other would allow
different release cadence, no?

> In any case, I need to take a few days away to clear my head.

Please take care!

Thanks for your effort on this project.

Best,
-K

Re: release timelines (-extindex, JMAP, lei)

[Eric Wong]

Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> On Fri, Mar 05, 2021 at 10:20:19PM +0000, Eric Wong wrote:
> > * lei saved searches can probably done quick for 1.7,
> >   but without full keywords support for externals...
> 
> I was wondering if lei should be part of the same suite as public-inbox proper
> as opposed to a standalone (or interdependent) client tool. Unless I'm
> mistaken, someone running a public-inbox origin or mirror server wouldn't
> necessarily be an active lei user. Decoupling them from each-other would allow
> different release cadence, no?

I've considered it, yes, but there's a lot of shared code and
writing release notes/planning/packaging for one project is
challenging enough.  As one might imagine, the non-coding
aspects of the project are the least interesting to me so I try
to avoid it as much as possible.

And I'm planning on sharing more code based on the work I've
done so far around IPC, too.

> > In any case, I need to take a few days away to clear my head.
> 
> Please take care!
> 
> Thanks for your effort on this project.

You're welcome.  Should be back to coding in a day or so,
still dealing with leaks of the non-memory variety :x