announce: my fork of alot

announce: my fork of alot

[Anton Khirnov]

Hi,

Thought I'd share with the people here the fork of alot I've been
hacking on for the past ~1.5 years, see if there is any interest in it.
The code can be found at git://git.khirnov.net/alot.

There are many changes in various places, the most user-visible ones in
the thread view mode. Specifically
- quoted blocks in the email body can now be colored and folded (this
  was probably my main motivation for starting all this)
- in upstream the thread mode shows a tree of messages, each node in the
  tree is a rendered message, that can be collapsed into a single-line
  summary;
  in my fork the thread mode is split-window - upper window for the tree
  with the thread structure, lower window for the currently selected
  message; no collapsing of messages
- attachments can be rendered inline, possibly colored with pygments
- git patches are colored with pygments
- all the parts are rendered for multipart/mixed messages, as per the
  RFCs
- encrypted/signed parts are now wrapped in a frame that indicates which
  bits of the message are actually encrypted or signed
- various architectural restructurings which were needed for the above
  or to allow for future changes (I have a large TODO list left)

The code is currently alpha quality - I am using it as my main MUA and
it works for my workflow, but any features I don't use regularly may be
broken. There is a general lack of "UX" polish (appearance and
documentation). I didn't bother updating the test suite to keep up with
all the architectural changes (plan to get to that once I consider the
code more stable). I removed some features which I considered an
impediment to progress and not worth the maintenance effort - YMMV.

Why did I not submit all this as PRs to upstream alot? The main reasons
were my lack of time and disagreement with the upstream about project
status. From what I can tell, alot maintainers consider the project to
be mature, so they prioritize stability and small incremental changes.
From my perspective, alot is lacking some critical features -- some
implemented in my fork already, some planned -- which makes it
borderline-unusable for me. As implementing those features required
large-scale architectural changes and my free time was quite limited, I
prioritized quickly implementing the things I cared about over
progressing in small incremental stable easily-reviewable steps.

At this point my tree has over 200 new commits and some ~4k changed
lines, so it's looking increasingly unlikely that I'll ever find the
free time and motivation to upstream it -- especially given alot's
glacial pace of development recently. If people are interested in using
this, I'll probably fork it "properly" under a new name.

Any comments or questions are very much welcome. I can also be reached
on IRC as elenril.

-- 
Anton Khirnov

Re: announce: my fork of alot

[Michael J Gruber]

Anton Khirnov venit, vidit, dixit 2021-05-16 12:19:45:
> Hi,
> 
> Thought I'd share with the people here the fork of alot I've been
> hacking on for the past ~1.5 years, see if there is any interest in it.

Thanks for sharing!

> The code can be found at git://git.khirnov.net/alot.

Any particular reason why this is not a fork where upstream is (GitHub)?
 
> There are many changes in various places, the most user-visible ones in
> the thread view mode. Specifically
> - quoted blocks in the email body can now be colored and folded (this
>   was probably my main motivation for starting all this)
> - in upstream the thread mode shows a tree of messages, each node in the
>   tree is a rendered message, that can be collapsed into a single-line
>   summary;
>   in my fork the thread mode is split-window - upper window for the tree
>   with the thread structure, lower window for the currently selected
>   message; no collapsing of messages
> - attachments can be rendered inline, possibly colored with pygments
> - git patches are colored with pygments
> - all the parts are rendered for multipart/mixed messages, as per the
>   RFCs
> - encrypted/signed parts are now wrapped in a frame that indicates which
>   bits of the message are actually encrypted or signed
> - various architectural restructurings which were needed for the above
>   or to allow for future changes (I have a large TODO list left)

This all sounds like getting closer to mutt's view, which is not a bad
thing at all!

> The code is currently alpha quality - I am using it as my main MUA and
> it works for my workflow, but any features I don't use regularly may be
> broken. There is a general lack of "UX" polish (appearance and
> documentation). I didn't bother updating the test suite to keep up with
> all the architectural changes (plan to get to that once I consider the
> code more stable).

I have to question this strategy. alot (upstream) suffers from a lack of
tests already. There is really no point writing tests after the fact or
once you discover bugs by chance. Especially if you go for "disruptive"
changes it's important to get the new architecture correct right from
the beginning.

> I removed some features which I considered an
> impediment to progress and not worth the maintenance effort - YMMV.
> 
> Why did I not submit all this as PRs to upstream alot? The main reasons
> were my lack of time and disagreement with the upstream about project
> status. From what I can tell, alot maintainers consider the project to
> be mature, so they prioritize stability and small incremental changes.
> From my perspective, alot is lacking some critical features -- some
> implemented in my fork already, some planned -- which makes it
> borderline-unusable for me. As implementing those features required
> large-scale architectural changes and my free time was quite limited, I
> prioritized quickly implementing the things I cared about over
> progressing in small incremental stable easily-reviewable steps.

I have a similar impression about the project status. I'm curious: What
are the architectural changes that you made?

From any notmuch-based MUA I expect a clear correspondence with
underlying notmuch concepts, such as "notmuch search results" being a
set of messages (not threads) and therefore command acting on that set
by default.

alot always confuses me in this regard.

> At this point my tree has over 200 new commits and some ~4k changed
> lines, so it's looking increasingly unlikely that I'll ever find the
> free time and motivation to upstream it -- especially given alot's
> glacial pace of development recently. If people are interested in using
> this, I'll probably fork it "properly" under a new name.
> 
> Any comments or questions are very much welcome. I can also be reached
> on IRC as elenril.

Have you tried raising these concerns with upstream before your fork?
Have you tried gathering a team around an idea and starting something
new together?

Frankly, upstream is borderline small already, and the way you started
your fork probably will not attract a team of people who want to make
that new fork their (common) own or are looking for a stronger team.

Incidentally, just yesterday I looked at neomutt's status, in search of
a substitute for alot with good notmuch support and a stronger devel
community. I suggest you read the most recent posts on their devel list
to see how much the deviating fork has burned out a small team.

Cheers
Michael

Re: announce: my fork of alot

[Anton Khirnov]

First of all - thank you for your email. I am happy to get feedback.

Quoting Michael J Gruber (2021-05-16 13:15:28)
> Anton Khirnov venit, vidit, dixit 2021-05-16 12:19:45:
> > Hi,
> > 
> > Thought I'd share with the people here the fork of alot I've been
> > hacking on for the past ~1.5 years, see if there is any interest in it.
> 
> Thanks for sharing!
> 
> > The code can be found at git://git.khirnov.net/alot.
> 
> Any particular reason why this is not a fork where upstream is (GitHub)?

Religious reasons mainly.

I dislike unnecessary centralization and prefer to run my own services
where feasible. Since I already run a git server for other things, it
was simplest to use it for alot as well.

>  
> > There are many changes in various places, the most user-visible ones in
> > the thread view mode. Specifically
> > - quoted blocks in the email body can now be colored and folded (this
> >   was probably my main motivation for starting all this)
> > - in upstream the thread mode shows a tree of messages, each node in the
> >   tree is a rendered message, that can be collapsed into a single-line
> >   summary;
> >   in my fork the thread mode is split-window - upper window for the tree
> >   with the thread structure, lower window for the currently selected
> >   message; no collapsing of messages
> > - attachments can be rendered inline, possibly colored with pygments
> > - git patches are colored with pygments
> > - all the parts are rendered for multipart/mixed messages, as per the
> >   RFCs
> > - encrypted/signed parts are now wrapped in a frame that indicates which
> >   bits of the message are actually encrypted or signed
> > - various architectural restructurings which were needed for the above
> >   or to allow for future changes (I have a large TODO list left)
> 
> This all sounds like getting closer to mutt's view, which is not a bad
> thing at all!

I used to use mutt in my computing pre-history (before discovering
notmuch over 10 years ago), so it's quite possible that was my
inspiration. Even though I barely remember what it looked like.

> 
> > The code is currently alpha quality - I am using it as my main MUA and
> > it works for my workflow, but any features I don't use regularly may be
> > broken. There is a general lack of "UX" polish (appearance and
> > documentation). I didn't bother updating the test suite to keep up with
> > all the architectural changes (plan to get to that once I consider the
> > code more stable).
> 
> I have to question this strategy. alot (upstream) suffers from a lack of
> tests already. There is really no point writing tests after the fact or
> once you discover bugs by chance.

Why "no point"? Adding tests is (almost) always good, whatever was the
trigger for writing them.

While I certainly agree that a comprehensive test suite is very
important for preventing regressions, there is a tradeoff between the
amount of time/mental effort I can spend on this and what the gain is
for me. I just don't find writing tests to be much fun, so forcing
myself to maintain the tests properly would slow down my progress quite
a lot.

I chose to implement the features I need first, and handle testing
later. You may judge me for it, but in the end what's most important to
me is how well this fulfills my requirements for a mail client.
Also, patches welcome ;)

> Especially if you go for "disruptive" changes it's important to get
> the new architecture correct right from the beginning.

I don't quite follow the reasoning here. Automated regression testing is
most useful for catching regressions. It is not that much useful for
guiding architecture design.

> 
> > I removed some features which I considered an
> > impediment to progress and not worth the maintenance effort - YMMV.
> > 
> > Why did I not submit all this as PRs to upstream alot? The main reasons
> > were my lack of time and disagreement with the upstream about project
> > status. From what I can tell, alot maintainers consider the project to
> > be mature, so they prioritize stability and small incremental changes.
> > From my perspective, alot is lacking some critical features -- some
> > implemented in my fork already, some planned -- which makes it
> > borderline-unusable for me. As implementing those features required
> > large-scale architectural changes and my free time was quite limited, I
> > prioritized quickly implementing the things I cared about over
> > progressing in small incremental stable easily-reviewable steps.
> 
> I have a similar impression about the project status. I'm curious: What
> are the architectural changes that you made?

It would be nontrivial to list them comprehensively, but off the top of
my head:
- urwidtrees is gone (this was a major obstacle to implementing quote
  handling IMO)
- thread tree structure is now fully decoupled from presentation
- stronger boundaries between various components (e.g. no more accessing
  email objects directly)
- removed some assumptions about emails that do not always hold
    * a signed/encrypted email is signed/encrypte in its entirety;
      actually this only holds on a MIME-part basis
    * a mail has one 'main' part that is to be displayed to the user as
      'the body'
      actually the body intended for display can be a concatenation of
      any number of MIME parts

> 
> From any notmuch-based MUA I expect a clear correspondence with
> underlying notmuch concepts, such as "notmuch search results" being a
> set of messages (not threads) and therefore command acting on that set
> by default.
> 
> alot always confuses me in this regard.

I'm afraid I disagree on this point. IMO one of the great points of
notmuch is its flexibility - it provides you with a versatile toolkit
that can be adapted to your workflow, rather than you adapting to
someone else's idea of what things should look like. E.g. my way of
working with emails is primarily thread-based, there is less focus on
individual emails. I would actually pefer alot to be _more_ thread-based
- e.g. when tagging in search mode.

That said, I see no fundamental reason you couldn't have both in alot
(or its hypothetical platonic ideal).

> 
> > At this point my tree has over 200 new commits and some ~4k changed
> > lines, so it's looking increasingly unlikely that I'll ever find the
> > free time and motivation to upstream it -- especially given alot's
> > glacial pace of development recently. If people are interested in using
> > this, I'll probably fork it "properly" under a new name.
> > 
> > Any comments or questions are very much welcome. I can also be reached
> > on IRC as elenril.
> 
> Have you tried raising these concerns with upstream before your fork?

1. I did submit a small number of patches to alot. The process was not
hugely satisfactory and left me with the impression that implementing
the features I want "the proper way" would require more time and effort
than I'm willing/able to spend.

2. E.g. folding quote blocks is issue number 3 in alot, opened in 2011.
Since then several people tried to work on it, but no attempt was
ultimately successful.

> Have you tried gathering a team around an idea and starting something
> new together?
> 
> Frankly, upstream is borderline small already, and the way you started
> your fork probably will not attract a team of people who want to make
> that new fork their (common) own or are looking for a stronger team.

I am way more of a "write code" person than "gather a team" person.
Writing this code was fun, improved my email workflow, and got me more
experience in new areas. That makes it worthwhile by itself.

Making it into a sustainable project used by others (either by merging
into alot or branching off into something new) would be nice, but is not
a necessary end goal for me.

> the way you started your fork

The way I see it, I have not started anything yet. I wrote some code for
my personal use and now I'm trying to gauge if others might be
interested in it. If yes, we'll see what ways forward are available. If
not, it's perfectly viable for me to just maintain it myself.

One very nice thing about alot is that it's really compact - since so
much of the heavy lifting can be done by notmuch, urwid, or the python
stdlib. It's very much possible for one person to get a decent
understanding of the entire codebase.

> 
> Incidentally, just yesterday I looked at neomutt's status, in search of
> a substitute for alot with good notmuch support and a stronger devel
> community. I suggest you read the most recent posts on their devel list
> to see how much the deviating fork has burned out a small team.

I get your point, but I see no perfect solutions here. I've been using
alot since 2014, hoping that it would at some point gain the missing
features. That never happened and its rate of development gradually slowed down
to almost nothing. I also stopped being a forever student, so using
email efficiently became a bigger priority for me.

My options were switch away from notmuch (see no better alternatives),
switch to a different notmuch frontend (of which I didn't see many,
especially as I'm a vim user), or try to improve alot. The last option
led me here.

-- 
Anton Khirnov

Re: announce: my fork of alot

[Patrick Totzke]

Hi everyone,

Of course I feel obliged to chime in and clarify, so here it goes.


Quoting Michael J Gruber (2021-05-16 12:15:28)

> Anton Khirnov venit, vidit, dixit 2021-05-16 12:19:45:
> > Hi,
> > 
> > Thought I'd share with the people here the fork of alot I've been
> > hacking on for the past ~1.5 years, see if there is any interest in it.
> 
> Thanks for sharing!

First of all many thanks to Anton for his enthusiasm in pushing the alot
project further!

 
> > The code can be found at git://git.khirnov.net/alot.
> 
> Any particular reason why this is not a fork where upstream is (GitHub)?
>  
> > There are many changes in various places, the most user-visible ones in
> > the thread view mode. Specifically
> > - quoted blocks in the email body can now be colored and folded (this
> >   was probably my main motivation for starting all this)
> > - in upstream the thread mode shows a tree of messages, each node in the
> >   tree is a rendered message, that can be collapsed into a single-line
> >   summary;
> >   in my fork the thread mode is split-window - upper window for the tree
> >   with the thread structure, lower window for the currently selected
> >   message; no collapsing of messages
> > - attachments can be rendered inline, possibly colored with pygments
> > - git patches are colored with pygments
> > - all the parts are rendered for multipart/mixed messages, as per the
> >   RFCs
> > - encrypted/signed parts are now wrapped in a frame that indicates which
> >   bits of the message are actually encrypted or signed
> > - various architectural restructurings which were needed for the above
> >   or to allow for future changes (I have a large TODO list left)
> 
> This all sounds like getting closer to mutt's view, which is not a bad
> thing at all!
> 
> > The code is currently alpha quality - I am using it as my main MUA and
> > it works for my workflow, but any features I don't use regularly may be
> > broken. There is a general lack of "UX" polish (appearance and
> > documentation). I didn't bother updating the test suite to keep up with
> > all the architectural changes (plan to get to that once I consider the
> > code more stable).
> 
> I have to question this strategy. alot (upstream) suffers from a lack of
> tests already. There is really no point writing tests after the fact or
> once you discover bugs by chance. Especially if you go for "disruptive"
> changes it's important to get the new architecture correct right from
> the beginning.
> 
> > I removed some features which I considered an
> > impediment to progress and not worth the maintenance effort - YMMV.

All this sounds very exciting and I'd be very happy to see these features in
(mainline) alot!

I agree that some of alot's underlying code is ready for refactoring
and urwid in particular has been a big drag on quickly implementing things.
Also, I'd be interested in hearing your thoughts on deprecating some "unworthy"
features in order to reduce the maintenance effort!


> > Why did I not submit all this as PRs to upstream alot? The main reasons
> > were my lack of time and disagreement with the upstream about project
> > status. From what I can tell, alot maintainers consider the project to
> > be mature, so they prioritize stability and small incremental changes.
> > From my perspective, alot is lacking some critical features -- some
> > implemented in my fork already, some planned -- which makes it
> > borderline-unusable for me. As implementing those features required
> > large-scale architectural changes and my free time was quite limited, I
> > prioritized quickly implementing the things I cared about over
> > progressing in small incremental stable easily-reviewable steps.
> 
> I have a similar impression about the project status. I'm curious: What
> are the architectural changes that you made?


Yes, the speed at which alot progresses is borderline problematic. This is of
course down to the small number of core contributors and the fact that for all
of us life goes on an priorities change.

One problem is that the project attracts many users interested in pushing what
I'd call "hotfixes" to address missing features: Often people would present
a (nicely working) proof-of concept that is not well documented, tested, and
doesn't adhere to common code conventions, only not to follow up on their
promises to "clean things up", for all too understandable reasons.
Still, I believe that just merging everything will quickly kill the project as
a) this leads to code that is very difficult and time-consuming to maintain and
b) broken features are very damaging to user's perception of the software, much
more so than missing ones.

I am not accusing you of anything here, Anton. I just wish to point out
potential long term difficulties and clarify that I tried to err on the side of
cautiousness to keep alot afloat in a usable state for most (potential) users.

> > At this point my tree has over 200 new commits and some ~4k changed
> > lines, so it's looking increasingly unlikely that I'll ever find the
> > free time and motivation to upstream it -- especially given alot's
> > glacial pace of development recently. If people are interested in using
> > this, I'll probably fork it "properly" under a new name.
> > 
> > Any comments or questions are very much welcome. I can also be reached
> > on IRC as elenril.
> 
> Have you tried raising these concerns with upstream before your fork?
> Have you tried gathering a team around an idea and starting something
> new together?
> 
> Frankly, upstream is borderline small already, and the way you started
> your fork probably will not attract a team of people who want to make
> that new fork their (common) own or are looking for a stronger team.

I share Michael's concerns about further splintering the small group of
developers and believe that this would be to the detriment of both projects.

It's no secret that I am ready to give the helm to others. I have been
maintaining this project for a while now, mainly for personal usage and as
a fun distraction. I have tried to squeeze in time to review pull requests when
possible and am grateful for the many code contributions over the years, most
notably the big steps towards pgp/mime, python3 and notmuch2, all of which I'd
have never found the time to implement myself.

It has so far been a successful, albeit slow, strategy to try and coordinate
efforts and I would very much like to see this going on, but without
sacrificing the quality of the code or the relative mature user experience.

To be clear: I still do not consider alot "mature" in the sense that I'd oppose
radical refactoring. This is reflected in its version number :)

Now, how to go on from here? It'd be great if we could coordinate future
efforts among those willing to spend time on the project!
This would require not only work on new features but also a discussion about
directions, priorities, quality control etc. So far, I found GitHub a good
place to organise these things but am happy to move on to another platform if
necessary. It'd be great to hear from the other contributors.

Best wishes,
P

Re: announce: my fork of alot

[Anton Khirnov]

Quoting Patrick Totzke (2021-05-16 17:41:49)
> Hi everyone,
> 
> All this sounds very exciting and I'd be very happy to see these features in
> (mainline) alot!
> 
> I agree that some of alot's underlying code is ready for refactoring
> and urwid in particular has been a big drag on quickly implementing things.
> Also, I'd be interested in hearing your thoughts on deprecating some "unworthy"
> features in order to reduce the maintenance effort!

That is largely a matter of perspective and personal preference. E.g.
among the things gone in my tree are:
- removing messages - I dropped that because I considered that code
  potentially dangerous, had no use for it myself, and just didn't want
  to tiptoe around it; someone actually using RemoveCommand in their
  workflow would have a different opinion
- switching to split-window layout for thread view made it simpler to
  implement quote folding, but also made sense to me since I never want
  to see more than one message at once;
  again, someone who prefers collapsing messages would see this as loss
  of functionality

https://xkcd.com/1172/ is very much in effect

> > > Why did I not submit all this as PRs to upstream alot? The main reasons
> > > were my lack of time and disagreement with the upstream about project
> > > status. From what I can tell, alot maintainers consider the project to
> > > be mature, so they prioritize stability and small incremental changes.
> > > From my perspective, alot is lacking some critical features -- some
> > > implemented in my fork already, some planned -- which makes it
> > > borderline-unusable for me. As implementing those features required
> > > large-scale architectural changes and my free time was quite limited, I
> > > prioritized quickly implementing the things I cared about over
> > > progressing in small incremental stable easily-reviewable steps.
> > 
> > I have a similar impression about the project status. I'm curious: What
> > are the architectural changes that you made?
> 
> 
> Yes, the speed at which alot progresses is borderline problematic. This is of
> course down to the small number of core contributors and the fact that for all
> of us life goes on an priorities change.
> 
> One problem is that the project attracts many users interested in pushing what
> I'd call "hotfixes" to address missing features: Often people would present
> a (nicely working) proof-of concept that is not well documented, tested, and
> doesn't adhere to common code conventions, only not to follow up on their
> promises to "clean things up", for all too understandable reasons.
> Still, I believe that just merging everything will quickly kill the project as
> a) this leads to code that is very difficult and time-consuming to maintain and
> b) broken features are very damaging to user's perception of the software, much
> more so than missing ones.
> 
> I am not accusing you of anything here, Anton. I just wish to point out
> potential long term difficulties and clarify that I tried to err on the side of
> cautiousness to keep alot afloat in a usable state for most (potential) users.

You would be very correct to accuse me of taking various shortcuts. I
would not call my changes "hotfixes", as I tried to keep continuous
future improvements in mind (and in fact see many of my changes as
cleanup and simplification). But I did make an explicit decision to
prioritise rapidly adding new functionality, at the cost of potential
regressions and loss of some features I did not need.

And again, this is a matter of perspective. If alot does what you want
it to do then of course you will value stability and consistency. But if
the lack of certain features makes it barely usable, then it makes sense
to be more radical.

> > > At this point my tree has over 200 new commits and some ~4k changed
> > > lines, so it's looking increasingly unlikely that I'll ever find the
> > > free time and motivation to upstream it -- especially given alot's
> > > glacial pace of development recently. If people are interested in using
> > > this, I'll probably fork it "properly" under a new name.
> > > 
> > > Any comments or questions are very much welcome. I can also be reached
> > > on IRC as elenril.
> > 
> > Have you tried raising these concerns with upstream before your fork?
> > Have you tried gathering a team around an idea and starting something
> > new together?
> > 
> > Frankly, upstream is borderline small already, and the way you started
> > your fork probably will not attract a team of people who want to make
> > that new fork their (common) own or are looking for a stronger team.
> 
> I share Michael's concerns about further splintering the small group of
> developers and believe that this would be to the detriment of both projects.
> 
> It's no secret that I am ready to give the helm to others. I have been
> maintaining this project for a while now, mainly for personal usage and as
> a fun distraction. I have tried to squeeze in time to review pull requests when
> possible and am grateful for the many code contributions over the years, most
> notably the big steps towards pgp/mime, python3 and notmuch2, all of which I'd
> have never found the time to implement myself.
> 
> It has so far been a successful, albeit slow, strategy to try and coordinate
> efforts and I would very much like to see this going on, but without
> sacrificing the quality of the code or the relative mature user experience.

And here is precisely the crux of the problem. My changes are pretty
drastic and 1) there WILL be bugs 2) someone WILL find them to degrade
their user experience. You cannot always satisfy everyone.

Combined with the fact that I also have a lot on my plate and don't see
myself reshaping my tree into nicely packaged atomic changes with a
ribbon on top (at least not any time soon).

> 
> To be clear: I still do not consider alot "mature" in the sense that I'd oppose
> radical refactoring. This is reflected in its version number :)

If you can find the time for it, maybe try to look at the individual
changes in my tree. Try it out, see what makes sense to you, what
doesn't, etc. I would be happy to see it all merged into alot, just
don't see how it can practically be done through the normal channels.

E.g. one thing I expect to be contentious is the removal of urwidtrees
use. I understand you are its author, so it may be unpleasant to hear,
but I found it to be a major obstacle to implementing quote folding.

-- 
Anton Khirnov

Re: announce: my fork of alot

[Patrick Totzke]

Quoting Anton Khirnov (2021-05-16 18:47:24)

> Quoting Patrick Totzke (2021-05-16 17:41:49)
> > Hi everyone,
> > 
> > All this sounds very exciting and I'd be very happy to see these features in
> > (mainline) alot!
> > 
> > I agree that some of alot's underlying code is ready for refactoring
> > and urwid in particular has been a big drag on quickly implementing things.
> > Also, I'd be interested in hearing your thoughts on deprecating some "unworthy"
> > features in order to reduce the maintenance effort!
> 
> That is largely a matter of perspective and personal preference. E.g.
> among the things gone in my tree are:
> - removing messages - I dropped that because I considered that code
>   potentially dangerous, had no use for it myself, and just didn't want
>   to tiptoe around it; someone actually using RemoveCommand in their
>   workflow would have a different opinion

Yep, same here. I never used that.

> - switching to split-window layout for thread view made it simpler to
>   implement quote folding, but also made sense to me since I never want
>   to see more than one message at once;
>   again, someone who prefers collapsing messages would see this as loss
>   of functionality
> 
> https://xkcd.com/1172/ is very much in effect

This could have been easily introduced as a separate type of buffer to keep both variants
without breaking things.


> > > > Why did I not submit all this as PRs to upstream alot? The main reasons
> > > > were my lack of time and disagreement with the upstream about project
> > > > status. From what I can tell, alot maintainers consider the project to
> > > > be mature, so they prioritize stability and small incremental changes.
> > > > From my perspective, alot is lacking some critical features -- some
> > > > implemented in my fork already, some planned -- which makes it
> > > > borderline-unusable for me. As implementing those features required
> > > > large-scale architectural changes and my free time was quite limited, I
> > > > prioritized quickly implementing the things I cared about over
> > > > progressing in small incremental stable easily-reviewable steps.
> > > 
> > > I have a similar impression about the project status. I'm curious: What
> > > are the architectural changes that you made?
> > 
> > 
> > Yes, the speed at which alot progresses is borderline problematic. This is of
> > course down to the small number of core contributors and the fact that for all
> > of us life goes on an priorities change.
> > 
> > One problem is that the project attracts many users interested in pushing what
> > I'd call "hotfixes" to address missing features: Often people would present
> > a (nicely working) proof-of concept that is not well documented, tested, and
> > doesn't adhere to common code conventions, only not to follow up on their
> > promises to "clean things up", for all too understandable reasons.
> > Still, I believe that just merging everything will quickly kill the project as
> > a) this leads to code that is very difficult and time-consuming to maintain and
> > b) broken features are very damaging to user's perception of the software, much
> > more so than missing ones.
> > 
> > I am not accusing you of anything here, Anton. I just wish to point out
> > potential long term difficulties and clarify that I tried to err on the side of
> > cautiousness to keep alot afloat in a usable state for most (potential) users.
> 
> You would be very correct to accuse me of taking various shortcuts. I
> would not call my changes "hotfixes", as I tried to keep continuous
> future improvements in mind

I understand. The question is just how long do you plan to stick around
to add improvements and provide support? If your answer is: "not at all, I'm
only sharing my code here", then don't be surprised if your efforts die the
same death and quickly disappear as so many other notmuch projects before.
If you are fine with that: great. But I suppose you wouldn't have shared your
code here unless you'd want people to join in and use it (for longer than
a day).

> (and in fact see many of my changes as
> cleanup and simplification).

I'm more than happy to push some of them.

> But I did make an explicit decision to
> prioritise rapidly adding new functionality, at the cost of potential
> regressions and loss of some features I did not need.
> 
> And again, this is a matter of perspective. If alot does what you want
> it to do then of course you will value stability and consistency. But if
> the lack of certain features makes it barely usable, then it makes sense
> to be more radical.

Yes, I agree. There are many features that I'm personally missing but
either don't yet know how to implement, or already have thought about them
and can see how much extra work it'd be.
I welcome disruption. However, it'd be constructive to introduce changes one at
a time, discuss, then implement them with all bells and whistles (read: document).

> > > > At this point my tree has over 200 new commits and some ~4k changed
> > > > lines, so it's looking increasingly unlikely that I'll ever find the
> > > > free time and motivation to upstream it -- especially given alot's
> > > > glacial pace of development recently. If people are interested in using
> > > > this, I'll probably fork it "properly" under a new name.
> > > > 
> > > > Any comments or questions are very much welcome. I can also be reached
> > > > on IRC as elenril.
> > > 
> > > Have you tried raising these concerns with upstream before your fork?
> > > Have you tried gathering a team around an idea and starting something
> > > new together?
> > > 
> > > Frankly, upstream is borderline small already, and the way you started
> > > your fork probably will not attract a team of people who want to make
> > > that new fork their (common) own or are looking for a stronger team.
> > 
> > I share Michael's concerns about further splintering the small group of
> > developers and believe that this would be to the detriment of both projects.
> > 
> > It's no secret that I am ready to give the helm to others. I have been
> > maintaining this project for a while now, mainly for personal usage and as
> > a fun distraction. I have tried to squeeze in time to review pull requests when
> > possible and am grateful for the many code contributions over the years, most
> > notably the big steps towards pgp/mime, python3 and notmuch2, all of which I'd
> > have never found the time to implement myself.
> > 
> > It has so far been a successful, albeit slow, strategy to try and coordinate
> > efforts and I would very much like to see this going on, but without
> > sacrificing the quality of the code or the relative mature user experience.
> 
> And here is precisely the crux of the problem. My changes are pretty
> drastic and 1) there WILL be bugs 2) someone WILL find them to degrade
> their user experience. You cannot always satisfy everyone.

I am actually fine with breaking things.
But it has to be justified and changes need to at least be documented somewhere.
Users will adjust or move on. But it is difficult to "re-sort" code once it
becomes inconsistent or unpredictable in my opinion.


> Combined with the fact that I also have a lot on my plate and don't see
> myself reshaping my tree into nicely packaged atomic changes with a
> ribbon on top (at least not any time soon).

That's too bad. Even small PR's would be welcome.


> > To be clear: I still do not consider alot "mature" in the sense that I'd oppose
> > radical refactoring. This is reflected in its version number :)
> 
> If you can find the time for it, maybe try to look at the individual
> changes in my tree. Try it out, see what makes sense to you, what
> doesn't, etc. I would be happy to see it all merged into alot, just
> don't see how it can practically be done through the normal channels.

Look, if you can't be bothered to go via "the normal channels",
then it's unlikely that anyone wants to spend the time to dig though your code
to find (upstream) usable nuggets. If someone does, and re-packages into a PR
then cool. I don't see myself spending much time on that I'm afraid.


> E.g. one thing I expect to be contentious is the removal of urwidtrees
> use. I understand you are its author, so it may be unpleasant to hear,
> but I found it to be a major obstacle to implementing quote folding.

No: I authored urwidtrees out of necessity. Urwid simply did not have
a widget for trees at the time and I wanted to replicate the sup/gmail
type of buffer. It is not particularly great code and I'd be happy to see it go.

Best wishes,  
P

Re: announce: my fork of alot

[Patrick Totzke]

[-- Attachment #1.1: Type: text/plain, Size: 9499 bytes --]

```
[~] git clone https://git.khirnov.net/alot.git             
Cloning into 'alot'...                        
fatal: unable to access 'https://git.khirnov.net/alot.git/': server certificate verification failed. CAfile: none CRLfile: none
[~] git clone git://git.khirnov.net/alot.git

Cloning into 'alot'...

(times out)
```



Quoting Patrick Totzke (2021-05-16 19:23:58)

> Quoting Anton Khirnov (2021-05-16 18:47:24)
> 
> > Quoting Patrick Totzke (2021-05-16 17:41:49)
> > > Hi everyone,
> > > 
> > > All this sounds very exciting and I'd be very happy to see these features in
> > > (mainline) alot!
> > > 
> > > I agree that some of alot's underlying code is ready for refactoring
> > > and urwid in particular has been a big drag on quickly implementing things.
> > > Also, I'd be interested in hearing your thoughts on deprecating some "unworthy"
> > > features in order to reduce the maintenance effort!
> > 
> > That is largely a matter of perspective and personal preference. E.g.
> > among the things gone in my tree are:
> > - removing messages - I dropped that because I considered that code
> >   potentially dangerous, had no use for it myself, and just didn't want
> >   to tiptoe around it; someone actually using RemoveCommand in their
> >   workflow would have a different opinion
> 
> Yep, same here. I never used that.
> 
> > - switching to split-window layout for thread view made it simpler to
> >   implement quote folding, but also made sense to me since I never want
> >   to see more than one message at once;
> >   again, someone who prefers collapsing messages would see this as loss
> >   of functionality
> > 
> > https://xkcd.com/1172/ is very much in effect
> 
> This could have been easily introduced as a separate type of buffer to keep both variants
> without breaking things.
> 
> 
> > > > > Why did I not submit all this as PRs to upstream alot? The main reasons
> > > > > were my lack of time and disagreement with the upstream about project
> > > > > status. From what I can tell, alot maintainers consider the project to
> > > > > be mature, so they prioritize stability and small incremental changes.
> > > > > From my perspective, alot is lacking some critical features -- some
> > > > > implemented in my fork already, some planned -- which makes it
> > > > > borderline-unusable for me. As implementing those features required
> > > > > large-scale architectural changes and my free time was quite limited, I
> > > > > prioritized quickly implementing the things I cared about over
> > > > > progressing in small incremental stable easily-reviewable steps.
> > > > 
> > > > I have a similar impression about the project status. I'm curious: What
> > > > are the architectural changes that you made?
> > > 
> > > 
> > > Yes, the speed at which alot progresses is borderline problematic. This is of
> > > course down to the small number of core contributors and the fact that for all
> > > of us life goes on an priorities change.
> > > 
> > > One problem is that the project attracts many users interested in pushing what
> > > I'd call "hotfixes" to address missing features: Often people would present
> > > a (nicely working) proof-of concept that is not well documented, tested, and
> > > doesn't adhere to common code conventions, only not to follow up on their
> > > promises to "clean things up", for all too understandable reasons.
> > > Still, I believe that just merging everything will quickly kill the project as
> > > a) this leads to code that is very difficult and time-consuming to maintain and
> > > b) broken features are very damaging to user's perception of the software, much
> > > more so than missing ones.
> > > 
> > > I am not accusing you of anything here, Anton. I just wish to point out
> > > potential long term difficulties and clarify that I tried to err on the side of
> > > cautiousness to keep alot afloat in a usable state for most (potential) users.
> > 
> > You would be very correct to accuse me of taking various shortcuts. I
> > would not call my changes "hotfixes", as I tried to keep continuous
> > future improvements in mind
> 
> I understand. The question is just how long do you plan to stick around
> to add improvements and provide support? If your answer is: "not at all, I'm
> only sharing my code here", then don't be surprised if your efforts die the
> same death and quickly disappear as so many other notmuch projects before.
> If you are fine with that: great. But I suppose you wouldn't have shared your
> code here unless you'd want people to join in and use it (for longer than
> a day).
> 
> > (and in fact see many of my changes as
> > cleanup and simplification).
> 
> I'm more than happy to push some of them.
> 
> > But I did make an explicit decision to
> > prioritise rapidly adding new functionality, at the cost of potential
> > regressions and loss of some features I did not need.
> > 
> > And again, this is a matter of perspective. If alot does what you want
> > it to do then of course you will value stability and consistency. But if
> > the lack of certain features makes it barely usable, then it makes sense
> > to be more radical.
> 
> Yes, I agree. There are many features that I'm personally missing but
> either don't yet know how to implement, or already have thought about them
> and can see how much extra work it'd be.
> I welcome disruption. However, it'd be constructive to introduce changes one at
> a time, discuss, then implement them with all bells and whistles (read: document).
> 
> > > > > At this point my tree has over 200 new commits and some ~4k changed
> > > > > lines, so it's looking increasingly unlikely that I'll ever find the
> > > > > free time and motivation to upstream it -- especially given alot's
> > > > > glacial pace of development recently. If people are interested in using
> > > > > this, I'll probably fork it "properly" under a new name.
> > > > > 
> > > > > Any comments or questions are very much welcome. I can also be reached
> > > > > on IRC as elenril.
> > > > 
> > > > Have you tried raising these concerns with upstream before your fork?
> > > > Have you tried gathering a team around an idea and starting something
> > > > new together?
> > > > 
> > > > Frankly, upstream is borderline small already, and the way you started
> > > > your fork probably will not attract a team of people who want to make
> > > > that new fork their (common) own or are looking for a stronger team.
> > > 
> > > I share Michael's concerns about further splintering the small group of
> > > developers and believe that this would be to the detriment of both projects.
> > > 
> > > It's no secret that I am ready to give the helm to others. I have been
> > > maintaining this project for a while now, mainly for personal usage and as
> > > a fun distraction. I have tried to squeeze in time to review pull requests when
> > > possible and am grateful for the many code contributions over the years, most
> > > notably the big steps towards pgp/mime, python3 and notmuch2, all of which I'd
> > > have never found the time to implement myself.
> > > 
> > > It has so far been a successful, albeit slow, strategy to try and coordinate
> > > efforts and I would very much like to see this going on, but without
> > > sacrificing the quality of the code or the relative mature user experience.
> > 
> > And here is precisely the crux of the problem. My changes are pretty
> > drastic and 1) there WILL be bugs 2) someone WILL find them to degrade
> > their user experience. You cannot always satisfy everyone.
> 
> I am actually fine with breaking things.
> But it has to be justified and changes need to at least be documented somewhere.
> Users will adjust or move on. But it is difficult to "re-sort" code once it
> becomes inconsistent or unpredictable in my opinion.
> 
> 
> > Combined with the fact that I also have a lot on my plate and don't see
> > myself reshaping my tree into nicely packaged atomic changes with a
> > ribbon on top (at least not any time soon).
> 
> That's too bad. Even small PR's would be welcome.
> 
> 
> > > To be clear: I still do not consider alot "mature" in the sense that I'd oppose
> > > radical refactoring. This is reflected in its version number :)
> > 
> > If you can find the time for it, maybe try to look at the individual
> > changes in my tree. Try it out, see what makes sense to you, what
> > doesn't, etc. I would be happy to see it all merged into alot, just
> > don't see how it can practically be done through the normal channels.
> 
> Look, if you can't be bothered to go via "the normal channels",
> then it's unlikely that anyone wants to spend the time to dig though your code
> to find (upstream) usable nuggets. If someone does, and re-packages into a PR
> then cool. I don't see myself spending much time on that I'm afraid.
> 
> 
> > E.g. one thing I expect to be contentious is the removal of urwidtrees
> > use. I understand you are its author, so it may be unpleasant to hear,
> > but I found it to be a major obstacle to implementing quote folding.
> 
> No: I authored urwidtrees out of necessity. Urwid simply did not have
> a widget for trees at the time and I wanted to replicate the sup/gmail
> type of buffer. It is not particularly great code and I'd be happy to see it go.
> 
> Best wishes,  
> P

[-- Attachment #1.2: Type: text/html, Size: 19659 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: announce: my fork of alot

[Michael J Gruber]

Patrick Totzke venit, vidit, dixit 2021-05-17 09:08:31:
> ```
> [~] git clone https://git.khirnov.net/alot.git             
> Cloning into 'alot'...                        
> fatal: unable to access 'https://git.khirnov.net/alot.git/': server certificate verification failed. CAfile: none CRLfile: none
> [~] git clone git://git.khirnov.net/alot.git
> 
> Cloning into 'alot'...
> 
> (times out)
> ```

It is unauthenticated git protocol:

git clone git://git.khirnov.net/alot.git

I can fully understand why someone wants to host their personal stuff
for themselves. I still don't understand why a git-fork of a
github-project which asks for upstreaming or collaboration is not at
github - and yes, you can 's/github/yourchoice/' in that sentence ;)

Cheers
Michael

Re: announce: my fork of alot

[Michael J Gruber]

Michael J Gruber venit, vidit, dixit 2021-05-17 09:33:13:
> Patrick Totzke venit, vidit, dixit 2021-05-17 09:08:31:
> > ```
> > [~] git clone https://git.khirnov.net/alot.git             
> > Cloning into 'alot'...                        
> > fatal: unable to access 'https://git.khirnov.net/alot.git/': server certificate verification failed. CAfile: none CRLfile: none
> > [~] git clone git://git.khirnov.net/alot.git
> > 
> > Cloning into 'alot'...
> > 
> > (times out)
> > ```
> 
> It is unauthenticated git protocol:
> 
> git clone git://git.khirnov.net/alot.git
> 

Just for fun, I'll throw in a few git commands, but maybe other
notmuch-ml users would prefer us to continue somewhere else:

git merge-base upstream/master akhirnov/master
f1ceccaa58bb36cac73e6886a0b14230e5518fda

git one f1ceccaa
f1ceccaa ("envelope: add setting for custom Message-ID domain", 2019-11-23)

git rev-list --count --left-right --cherry-mark upstream/master...akhirnov/master
108     229	0

(There is one cherry-pick which this misses due to non-equal patch
hashes.)

git diff --shortstat upstream/master akhirnov/master
 99 files changed, 4135 insertions(+), 5475 deletions(-)

Michael

Re: announce: my fork of alot

[Anton Khirnov]

Quoting Michael J Gruber (2021-05-17 09:33:13)
> Patrick Totzke venit, vidit, dixit 2021-05-17 09:08:31:
> > ```
> > [~] git clone https://git.khirnov.net/alot.git             
> > Cloning into 'alot'...                        
> > fatal: unable to access 'https://git.khirnov.net/alot.git/': server certificate verification failed. CAfile: none CRLfile: none
> > [~] git clone git://git.khirnov.net/alot.git
> > 
> > Cloning into 'alot'...
> > 
> > (times out)
> > ```
> 
> It is unauthenticated git protocol:
> 
> git clone git://git.khirnov.net/alot.git
> 
> I can fully understand why someone wants to host their personal stuff
> for themselves. I still don't understand why a git-fork of a
> github-project which asks for upstreaming or collaboration is not at
> github - and yes, you can 's/github/yourchoice/' in that sentence ;)

Because at this point it is still "my personal stuff". If and when it
actually becomes "upstreaming or collaboration", I can also push it
elsewhere, as needed. I am not super fundamentalist in my religion.

But for the time being I see no big difference between people cloning
from my git server vs people cloning my repo on github.

-- 
Anton Khirnov

Re: announce: my fork of alot

[Anton Khirnov]

Quoting Patrick Totzke (2021-05-17 09:08:31)
> ```
> [~] git clone https://git.khirnov.net/alot.git             
> Cloning into 'alot'...                        
> fatal: unable to access 'https://git.khirnov.net/alot.git/': server certificate verification failed. CAfile: none CRLfile: none

no HTTP(S) cloning set up, sorry

> [~] git clone git://git.khirnov.net/alot.git
> 
> Cloning into 'alot'...
> 
> (times out)

this should work, unless you managed to trigger my firewall's autoban
(cleared it, just in case)

-- 
Anton Khirnov

Re: announce: my fork of alot

[Patrick Totzke]

Quoting Anton Khirnov (2021-05-17 09:19:21)

> Quoting Patrick Totzke (2021-05-17 09:08:31)
> > ```
> > [~] git clone https://git.khirnov.net/alot.git             
> > Cloning into 'alot'...                        
> > fatal: unable to access 'https://git.khirnov.net/alot.git/': server certificate verification failed. CAfile: none CRLfile: none
> 
> no HTTP(S) cloning set up, sorry
> 
> > [~] git clone git://git.khirnov.net/alot.git
> > 
> > Cloning into 'alot'...
> > 
> > (times out)
> 
> this should work, unless you managed to trigger my firewall's autoban
> (cleared it, just in case)


Thank you, this works now.
I had to radically edit my config but I can run it now.
The folding stuff is indeed nice. It'd be great if that was implemented simply on alots Message widget because that'd mean it could easily be backported. 
I struggle to move into messages that appear in mid-thread.

Anyways, glad to see some great new ideas. I'd love to see some work on getting this into alot mainline.
By the way, you are not the only one who wrote a different thread buffer...
P

Re: announce: my fork of alot

[Anton Khirnov]

Quoting Patrick Totzke (2021-05-17 10:28:14)
> Quoting Anton Khirnov (2021-05-17 09:19:21)
> 
> > Quoting Patrick Totzke (2021-05-17 09:08:31)
> > > ```
> > > [~] git clone https://git.khirnov.net/alot.git             
> > > Cloning into 'alot'...                        
> > > fatal: unable to access 'https://git.khirnov.net/alot.git/': server certificate verification failed. CAfile: none CRLfile: none
> > 
> > no HTTP(S) cloning set up, sorry
> > 
> > > [~] git clone git://git.khirnov.net/alot.git
> > > 
> > > Cloning into 'alot'...
> > > 
> > > (times out)
> > 
> > this should work, unless you managed to trigger my firewall's autoban
> > (cleared it, just in case)
> 
> 
> Thank you, this works now.
> I had to radically edit my config but I can run it now.
> The folding stuff is indeed nice. It'd be great if that was implemented simply on alots Message widget because that'd mean it could easily be backported. 

Sadly, I couldn't implement it in a satisfactory way without big other
changes. Otherwise I would have done so and sent a PR.

> I struggle to move into messages that appear in mid-thread.

I agree. Some ideas in my TODO list for that are:
- numbering the messages in the thread list
- number-prefixing commands (like in vim) to operate on nth message

-- 
Anton Khirnov

Re: announce: my fork of alot

[Anton Khirnov]

Quoting Patrick Totzke (2021-05-16 20:23:58)
> Quoting Anton Khirnov (2021-05-16 18:47:24)
> > - switching to split-window layout for thread view made it simpler to
> >   implement quote folding, but also made sense to me since I never want
> >   to see more than one message at once;
> >   again, someone who prefers collapsing messages would see this as loss
> >   of functionality
> > 
> > https://xkcd.com/1172/ is very much in effect
> 
> This could have been easily introduced as a separate type of buffer to keep both variants
> without breaking things.

I don't think it's that simple. Having two buffers for viewing threads
means:
- both of them now have to be maintained
- keeping feature parity between them would be nontrivial
- not keeping feature parity would be confusing for the users
It can easily lead to an unmaintainable mess and madness.

Will reply to other bits of this email later.

-- 
Anton Khirnov

Re: announce: my fork of alot

[Johannes Larsen]

[-- Attachment #1.1: Type: text/plain, Size: 3357 bytes --]

2021-05-16 12:19:45, Anton Khirnov <anton@khirnov.net> wrote:
> Thought I'd share with the people here the fork of alot I've been
> hacking on for the past ~1.5 years, see if there is any interest in it.
> The code can be found at git://git.khirnov.net/alot.

This looks very promising, and at least for me personally it is very
interesting. I tried it, and for my workflow at least it seems to work
as a drop-in for the mainline alot version. The only thing I have
noticed that broke was the now unnecessary ANSI-coloring I used in
mailcap scripts as a workaround to get rudimentary quote coloring.

I have been using sup/notmuch/alot (and forks thereof) for the last
decade or so, and I am very grateful for the work people have put into
this notmuch ecosystem of MUAs. None of the UIs have ever been perfect,
but that is okay, because for me at least most of them are still much
better than the alternatives (e.g. mutt, GMail, Thunderbird, Zimbra).


> - quoted blocks [...]
> - [...] thread mode is split-window

This new view is very nice, and for large threads (especially ones with
deep reply indentation) splitting it up like this gives a much better
overview of the threads than alots indented tree view.

> - all the parts are rendered for multipart/mixed messages, as per the
>   RFCs
> - encrypted/signed parts are now wrapped in a frame that indicates which
>   bits of the message are actually encrypted or signed

But I actually think this new way of dealing with multipart messages is
the most novel feature. It should be mentioned that mainline merged in a
similar togglemimepart/tree feature about a year ago (i.e. later than
where you forked), but this seems to be a more thoughtful design. This
is actually the first UI I have seen that intuitively presents the
different parts of complicated mails.


> - various architectural restructurings which were needed for the above
>   or to allow for future changes (I have a large TODO list left)

I for one at least would be very interested in hearing what sort of
feature you plan in the road map ahead.


> At this point my tree has over 200 new commits and some ~4k changed
> lines, so it's looking increasingly unlikely that I'll ever find the
> free time and motivation to upstream it -- especially given alot's
> glacial pace of development recently. If people are interested in using
> this, I'll probably fork it "properly" under a new name.

I really get this. I too have custom forks of e.g. alot tweaking small
annoyances in ways often considered more hacky than production ready.
For me at least these are tools I am using all day, so if I end of
supporting whatever fork I settled on for my own use long after the that
fork / the mainline dies away, that is fine by me.

Regardless of what happens to this with regards the mainline alot, I
just want to point out how nice it is that we have notmuch as a shared
backend for all these UIs. This makes it so much easier to port between
them as newer alternatives pops up. It also means that none of the UIs
needs to be feature complete to be useful, because for e.g. features
people needs once in a blue moon, there is no problem having a tailored
UIs/CLIs around for dealing with those particular things.


Best wishes,

-- johs (Johannes Larsen), (+47) 41435451

[-- Attachment #1.2: signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

-----BEGIN PGP SIGNATURE-----

iQIzBAABCAAdFiEEpTer3kqGU3pr882G07zxebJvqyYFAmChmAoACgkQ07zxebJv
qyaI1w//atQT3ym7K1C6kLPR1f+2LT6rBdyCZGEpvsZQnkA++powpwIuhEhbBvUB
P6bVAXtcNQt0hygUk1yTNyGzNQlFwXywyNgN57/QQcsbjs6mxmLqGZI6H3Z2Cq42
HrRreyH7xEklcLB8V+8epLgjZhlHi06cdFMMLWoyHrUC0YFrtHjM7veL7DHScOGD
mWCn6nF+WtqpQ+hMkjTeUWCuuBSXo6v3tynmJwY0yBM/xUSmBd6cVqy5HH5z62dI
+1EHk/gjyPgMR8uiii+9O4bYO7P2ucEZvOW+QLyEH4bM4t9JJ7Wfsih9GS7OhuJ/
71crd9KqFdAcTCF7CwMQbywTOZbZeR29UkDMYJ+n+w8/EcrI5UQ5owYpQuOgl2mE
AcJY4gkTXC20W7jWpeDzgxNaCZOzzGjz9SryJh9+ZdBFdQM9MLsmkPFCboa6q+vm
lVcAYLK/1S45JHqqAbVZ3u5SZ7eyFFSpRzIf/g9iDU7bX8B3OBQ0v9ynOlA+F5AN
qG/zqAiClbj4sUqMUxrAWFsURE8/MdIy/Vnf2/9RNH6i5GbzA963rAYmYe/pKjKr
EogAz7ZB5YI5ElI41f5ZxINpqsSXtJ06oNnXv8/xsTsamJ1aeYbGQQirpSWLsv5w
JmeUxvGxxyJiGN4RhtT6gM0LrEL6RH/g4O2nWYb3KnHdLTU6ZV4=
=gUcL
-----END PGP SIGNATURE-----

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: announce: my fork of alot

[Felipe Contreras]

On Sun, May 16, 2021 at 5:16 PM Johannes Larsen <mail@johslarsen.net> wrote:

> I have been using sup/notmuch/alot (and forks thereof) for the last
> decade or so, and I am very grateful for the work people have put into
> this notmuch ecosystem of MUAs. None of the UIs have ever been perfect,
> but that is okay, because for me at least most of them are still much
> better than the alternatives (e.g. mutt, GMail, Thunderbird, Zimbra).

Me too, but precisely because I found all of them lacking (including
alot), I started my own. Last one I vaguely remember using is sup.

Have you tried notmuch-vim?

-- 
Felipe Contreras

Re: announce: my fork of alot

[Johannes Larsen]

[-- Attachment #1.1: Type: text/plain, Size: 832 bytes --]

2021-05-17 00:32:09, Felipe Contreras <felipe.contreras@gmail.com> wrote:
> Have you tried notmuch-vim?

Yes, I have tried it, but I did not like its thread view though. Maybe
that mbox-like view of all messages in a thread works fine if there few
messages per thread, but I interact with a lot of mailing-lists where it
is not uncommon with 20+ messages in a thread, and then I kind of need
some sort of hierarchic view of messages to keep my bearings.

Also, I have been using vim as my primary editor/IDE for decades, so I
am kind of scared of bundling even more features into that setup. When I
am dealing with dozens of plugins and thousands of lines with vimscript
configuration already, then it is not so uncommon for some of them to
interact badly with each other.

-- 
johs (Johannes Larsen), (+47) 41435451

[-- Attachment #1.2: signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

-----BEGIN PGP SIGNATURE-----

iQIyBAABCAAdFiEEpTer3kqGU3pr882G07zxebJvqyYFAmChpx0ACgkQ07zxebJv
qyZCcQ/4vrrcyfPJelU/FTs0tdCKfA9QUFTtLyDO1ERJI2zAooBhKBg0wyq7cRO+
v/DfnEF0/ylwsP5nZ9L/QbyWqO8faexJWytF2CXJDYQV8NhQ+1mQ1R2pMV5y2VC1
oG05DwzeJaPKUu0yVoIySekDq8DdLI0LeZqcW893YuS9WsFoe7QEYIx2RCAh7Lsq
qgDS36x2GniuEeTYgGoRI5Gk2sLu6Xkslz1KX1U9gTbcZv58FIbYzz+I8BrRQhNo
fOu/96z0cFlp6jeIyK1V+dMe2Z1p90Qubeq9IjemMEXSkMr5CqXAwv935z6iU6gG
5EQaYkEugYeYSjKEnj+sLawcPepMqbirz7gkN/5hE0HYbdGqdGv9MRpokLqwIa3S
/GB7ORRC4ShXGmHctt5fRmV7/oAHVncr0WfIQreDpR4xOP0r22ghbZq9Q/c1yTVv
5mxYKPEixnHCWdJ8XI3l4pGQKgKxpP7Pd6qqDu2iGgpB80kpwcbjoV0bF6+YtgOs
JkzMJf4bAlK9Je81CBe4QoGsuDv5k7it8L4UinINMIFDn5bsm9Fob/XNvVOZtOIF
iyX2iNvOh8fRdl/ZfV15YcNeqCfGs9i/JzrgKAAcqdgB6V5nkJ+3G2OFEhQsfmAe
HOt0xZdgDMI4HKpD8oKhU4WMNGIL1pgDYQ+M/lB/Dd4vuvZDzA==
=U8Cq
-----END PGP SIGNATURE-----

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: announce: my fork of alot

[Felipe Contreras]

On Sun, May 16, 2021 at 6:13 PM Johannes Larsen <mail@johslarsen.net> wrote:
>
> 2021-05-17 00:32:09, Felipe Contreras <felipe.contreras@gmail.com> wrote:
> > Have you tried notmuch-vim?
>
> Yes, I have tried it, but I did not like its thread view though. Maybe
> that mbox-like view of all messages in a thread works fine if there few
> messages per thread, but I interact with a lot of mailing-lists where it
> is not uncommon with 20+ messages in a thread, and then I kind of need
> some sort of hierarchic view of messages to keep my bearings.

I hear you.

I do interact with such mailing lists, but _usually_ I only deal with
the new messages on the thread, not *all* messages. In that scenario
the thread-view works fine.

However, when I'm looking at old threads that I neglected, then it is
a bit cumbersome. I agree with you.

I'm not unfamiliar with big threads: my last big thread is of
yesterday: 45 messages [1].

I do have some experimental code in which I started rewriting
notmuch-vim from scratch, and one of the first thoughts I had was: why
not use a hierarchical view? In fact I don't think it would be that
difficult; we could use vim folds in order to expand/contract
subthreads.

There's a lot of other things to do first though.

> Also, I have been using vim as my primary editor/IDE for decades, so I
> am kind of scared of bundling even more features into that setup. When I
> am dealing with dozens of plugins and thousands of lines with vimscript
> configuration already, then it is not so uncommon for some of them to
> interact badly with each other.

Sure, but in modern versions of vim (since 7.0?) there is an autoload
feature [2]. With that feature nothing actually happens unless you (or
some code) calls that function. There's absolutely no way autoload
code can interact with another plugin if you have not issued a
:NotMuch command specifically.

Granted: the current version of notmuch-vim doesn't use this autoload
feature, although I suspect much of the code normally is hidden from
view (until you call :NotMuch). I'm not entirely sure, and it's better
to make sure using autoload.

This is something I could do in probably less than 30 minutes, if it's
truly a blocker for you... But again... I think there's a lot of
things to do first; that's why I haven't done it yet.

Of course, any issues you find you are more than welcome to report on
the issue tracker.

Cheers.

[1] https://lore.kernel.org/git/20210511222754.417371-1-felipe.contreras@gmail.com/
[2] https://vimhelp.org/eval.txt.html#autoload

-- 
Felipe Contreras

Re: announce: my fork of alot

[Anton Khirnov]

Quoting Johannes Larsen (2021-05-17 00:09:14)
> > - various architectural restructurings which were needed for the above
> >   or to allow for future changes (I have a large TODO list left)
> 
> I for one at least would be very interested in hearing what sort of
> feature you plan in the road map ahead.

Here's my current roadmap:

general
-------
* statusbar notification about outstanding async tasks
* more async
    - pass current context to commands:
      now that write operations are async, it is possible for a user to execute a command,
      then move focus
      since the command retrieves the focus (and hence the item to act on) by itself,
      the focus item by the time the command executes might different from the one the user
      intended to target
    - commands themselves should probably be synchronous and only launch
      async tasks where appropriate
      this is necessary to prevent races where a later command conflicts
      with an earlier one
    - indicator of outstanding async operations in statusbar
    - db operations (in a thread)
    - buffer/widget construction
    - message loading/processing
* atomic complex retagging commands
  e.g. add+remove in the same command
* refactor command implementation
    - make them tied to a buffer instead of global
    - number-prefixing commands (like in vim),
      e.g. to allow operating on message/thread <N>
* eliminate helper - it's a mess of random unrelated stuff that should be
  properly organized
* better mouse support
* do something more useful with status bar -- display tabs?
* running external commands in a separate tmux pane/terminal
    - tmux does not have a simple way of "wait for pane to exit"
      but tmux wait-for might be usable
* rethink keypress handling
* improve logging
    - trace and verbose levels
    - logger per object
    - more comprehensive
    - log errors/warnings as ui notifications automatically
* look into the use of email.charset, it looks suspicious
* rename and fork?
* tests
* documentation
* look into dropping twisted
    - replace with asyncio event loop?
      there are urwid performance issues, due to urwid idle horror
* review hooks
    - kill off those that get access to too much private state (e.g. ui)
    - enums for hooks
* look at prompt command
* look at https://github.com/pazz/alot/pull/1546

thread
------
- search+highlight
- properly handle focus switching between the thread tree and message body
    * message select vs attachment open
    * better styling for which one is focused
- better attachment styling
    * kill off highlightable widgets
- elinks encoding fuckery
  id:1f6d7111-4b01-42de-93da-7c3536715527@atl1s07mta1444.xt.local
- auto-page copiousoutput attachments
- color improvements
    * allow disabling all coloring
    * custom highlighting hooks
    * color HTML rendered by elinks/w3m
    * pygments
        * terminal256
        * configurable theming
        * more configurable lexer detection (custom hook?)
    * assign colors to authors
      for large/many-author threads only?
- folding improvements
    * better styling for folded part
    * smarter quote level detection
        - consider messages in references header?
    * some indicator for current foldlevel?
    * individual folds
    * folding sections of git patches
    * other filetypes?
    * custom folding detection hooks
    * folding in git patch commit messages
    * test case
      167e6c3a-a7f7-4b75-b2dd-00fbface9c33@www.fastmail.com
      060F97E7-6D39-4184-8F05-D59A9F6F3495@googlemail.com
- message exporting sanitization
    decoded - strip transfer-encoding headers
    raw-decrypted?
- better handling of matching vs whole thread
    * make the thread aware of the query it was spawned from (if any)
    * different theming for matching messages?
- better and customizable styling for MIME parts decorations
- prettify headers
    * styling for addresses (name vs mailbox part)
    * styling for subject (styling for [TAG])
    * styling for Date (display in local-preferred format)
- displaying message source should be in the to-be-added message view buffer
- changing indentation levels
- make refresh only refresh changed stuff
    * keep focus
- numbered messages in the thread tree
- scrollbars/scroll indicators
- fix arrow/tree structure theming
  when the tree node spans more than one line
- handle multipart/report better
  sample 20210129131012.A16FD24068A@mail0.khirnov.net
- cleaner way to pass rendering settings through the call stack
- folding parts of the thread tree
  useful for very large and complex threads
- pipeto refactoring
- decoration-less mode for copying long links (like alt-l in weechat)

search
------
- operations on whole threads, not just matching messages
- numbered lines, ability to operate on line <num> directly
- refine is useless - replace with "open modified query in a new buffer"
- retagging tag ordering improvements
    * selected tags to display first/last
- dim threads that no longer match the query
- threadline structure is in the theme, wtf

completion
----------
* commandline history completion by commands
* completing tags with arguments (e.g. --all) is broken
* remove should only complete tags that are present

namedqueries
------------
* count threads
* completion for select

compose
-------
* in a separate tmux pane
* reply including original attachments
* factor out reply code into a module under mail/
* honor_follow_up_to default should most likely be True

new buffer for displaying a single message
------------------------------------------
* full MIME structure
* toggleable rendering for parts

theme files
-----------
* make more user friendly
* inherit from default
* includes?
* referring to previously defined colors

db
--
* use threads+async
    - queries
    - message loading
* refreshing threads is really suspect and probably fragile
    * signal changes and have interested code re-create thread objects
* deduplicate tagging operations code in manager/thread/message
* thorough exception catching for database operations
* better missing email file handling

config
------
* apparently errors on parsing defaults file are swallowed

crypto:
-------
* signature parsing for <CABKe4Msh89xu=xeYEuTNy44xu5+MB07hZ6wz3kSPd15C3OWeeA@mail.gmail.com>
* refactor crypto.py
* report missing keys as such and not as "bad signature"
* special theming for missing keys?

links
-----
* urwid
    * http://urwid.org/reference/widget.html
    * http://urwid.org/reference/main_loop.html
    * https://github.com/urwid/urwid/issues/226
    * https://github.com/rndusr/stig/blob/master/stig/tui/scroll.py
    * https://wiki.goffi.org/w/index.php/Urwid-satext/en
    * https://github.com/wackywendell/ipyurwid/blob/master/urwidpygments.py
* https://cr.yp.to/proto/replyto.html
  look at Mail-Followup-To in detail

-- 
Anton Khirnov

Re: announce: my fork of alot

[Johannes Larsen]

[-- Attachment #1.1: Type: text/plain, Size: 1792 bytes --]

2021-05-17 10:54:09, Anton Khirnov <anton@khirnov.net> wrote:
> Here's my current roadmap:

Lots if interesting features/fixes!


>     - commands themselves should probably be synchronous [...] to
>       prevent races where a later command conflicts with an earlier one

Yeah, that race condition is a annoyance. Quitting alot helps at least.

> * atomic complex retagging commands
>   e.g. add+remove in the same command

sup had something like that. The retag operation when you had multiple
messages used +/- prefixes to add/remove tags from those messages whilst
leaving other tags on them untouched. I do not know if this was atomic,
but at least it was idempotent.

> * running external commands in a separate tmux pane/terminal

That is a good idea. Quitting the editor and using ";" to temporary
switching to another buffer (e.g. reading something elsewhere in a
thread) works, but is a hassle. [vim-dispatch] does something similar
for e.g. external compilation/tests from the editor, so maybe they
figured out some indicator for when tmux panes/windows completes.

[vim-dispath]: https://github.com/tpope/vim-dispatch

> - search+highlight

Yeah, I am still automatically trying "/" in alot (the hotkey that
searched like that in sup) to search in the buffer.

> - properly handle focus switching between the thread tree and message body

Mapping something to "move toggle" worked okay to switch focus between
these panes. Was kind of a surprise when scrolling within a message body
scrolled to the next message though.

> - pipeto refactoring

Yeah, as it is now I only use it with less to read the raw body. I think
pipeto based on a selected body/attachment for instance would be useful.

-- 
johs (Johannes Larsen), (+47) 41435451

[-- Attachment #1.2: signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

-----BEGIN PGP SIGNATURE-----

iQIzBAABCAAdFiEEpTer3kqGU3pr882G07zxebJvqyYFAmCiYkwACgkQ07zxebJv
qyZZiw//fK3+u3WnYSbcuAv7zhFLudEkm4GDMMR83bkegUhevhcjlPXSVSqh1NIj
Tv4Vvh4lwl5fZyLnx+YON40TBzfuPOrMX2nfjOw8Uy5Qkw48lDhfTTfBTasuOWKG
8pEtFPwRIW4eYJMc8PjY9blqZ6WkcdAeZVl0mFrmT1JqZ3tL7z8njksrpSjkH416
0Wk6E3Ok7hSGcmmzgkvQP0i8oH8+QAu52pM8RCILTLqsBt9M8/8W7ktAT6VlGCkI
HLpkCEotF9jlrmEvQ0OYUZ1QkrOdcEAYiR6yswhF/gJ/dC+NQtVVuZvasSFK4cDe
UZHTyIFkxIY7KNYuEXggECqYoHIO/Tgo1imii8ep/RTSsUhvQ0LtcbBtOH2IyOBi
YRT3eVOaMsh7SPjDihrajGMR9wi5O/Cc/FNqQUKQTov6l7vG5MXM+ZyRdWhnrRnn
5wrO/6wvfsJ0EmPrKsOKCh2OSNLgXZ/kKh5TJry5DiwE4sCNdnZyQI3qbQdqATR4
2xbxrmgEVofg5of/kH4SETP5P/p62Sxthf21U9kKGZJp/c/fHb1caBAYqQH42bjO
lHkxe0v9d5bzh+CnbZdnnQRIZ4OSK7plbJSHLWgXnggxKugyVSDi9N2Oq6yT8NjO
6m8p0xn5Qb8zljsZomhiajKU4+tL30CC/784l1TI3+ySGL/tvhY=
=w//Z
-----END PGP SIGNATURE-----

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: announce: my fork of alot

[Anton Khirnov]

Quoting Johannes Larsen (2021-05-17 14:32:12)
> 2021-05-17 10:54:09, Anton Khirnov <anton@khirnov.net> wrote:
> > Here's my current roadmap:
> 
> Lots if interesting features/fixes!

Of course keep in mind that it's largely a wishlist, so don't expect
these to be done
- at all
- any time soon
- in this order

> >     - commands themselves should probably be synchronous [...] to
> >       prevent races where a later command conflicts with an earlier one
> 
> Yeah, that race condition is a annoyance. Quitting alot helps at least.

Did you actually hit it? I don't think I actually saw it happen more
than a couple times, but even so I'd like to fix it before adding more
async.

> 
> > * running external commands in a separate tmux pane/terminal
> 
> That is a good idea. Quitting the editor and using ";" to temporary
> switching to another buffer (e.g. reading something elsewhere in a
> thread) works, but is a hassle. [vim-dispatch] does something similar
> for e.g. external compilation/tests from the editor, so maybe they
> figured out some indicator for when tmux panes/windows completes.
> 
> [vim-dispath]: https://github.com/tpope/vim-dispatch

Seems that just periodically polls the pane list, which is suboptimal.
Seems tmux has a 'wait-for' command that could be useful for
implementing this.

> > - properly handle focus switching between the thread tree and message body
> 
> Mapping something to "move toggle" worked okay to switch focus between
> these panes. Was kind of a surprise when scrolling within a message body
> scrolled to the next message though.

Yeah, that's a known bug and I'm sure I'll get to fixing it SomeDay(tm),
since it's rather annoying.

-- 
Anton Khirnov

Re: announce: my fork of alot

[Johannes Larsen]

[-- Attachment #1.1: Type: text/plain, Size: 658 bytes --]

2021-05-17 14:55:57, Anton Khirnov <anton@khirnov.net> wrote:
>> >     - commands themselves should probably be synchronous [...] to
>> >       prevent races where a later command conflicts with an earlier one
>> 
>> Yeah, that race condition is a annoyance. Quitting alot helps at least.
>
> Did you actually hit it? I don't think I actually saw it happen more
> than a couple times, but even so I'd like to fix it before adding more
> async.

Maybe not as a race condition, but out of habit I tend to try aborting
prompts with Ctrl-C (instead of Esc), and that triggers this conflict
you describe.

-- 
johs (Johannes Larsen), (+47) 41435451

[-- Attachment #1.2: signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

-----BEGIN PGP SIGNATURE-----

iQIzBAABCAAdFiEEpTer3kqGU3pr882G07zxebJvqyYFAmCivfEACgkQ07zxebJv
qybCFRAArlcqpGjHFK1sOs1f1xkr0v/6hgZJMcUUg0Qut4CVHbrtZbde6WArUmVs
D2ZMwExak+Dj2Z/w+xZf5FpMIhD41rgrjOoYn1kay/O93vBe1CnKOOpkiA5JyVVk
wppxO9+kY+WVZBZZo0nLrqbdjiAoR/AskfqOfKPiNd7qj20a9Su+op2oEX4nofud
1miBxBiuTm/6hEoAVVTN0mZECTUqdZVBiWMlSey995oJuIP5LwKnTXIoE4OvD6k8
xXr4eMCqK8L5uNW98yiO822BEz4BJSk4nP9xg8zzj+H03SEjJUlOKcS3snxZv0rY
+bbuXAhxSCmGust93QjWupHCTcQbgsFB5CShG37c28hdZS0Ft5xlO2h4NKf+DwgP
VrkXB9x0J5OZPXvR0XjldA1hp7KutU7Kd0aRVC14zOLLGJFxallrARMSSNvNm/Ro
Y0dA0UEcuN/FP/upR097FDLoudxxniuU+E5CrPR/wzquKCKhxyeWaAyekenembIS
hZ/9fn++nhUwtWXAwwGpFSe6avVHwFLQp7ViueEAOjgdmoq8bX+wEhXY0k72MxZq
8j7Ag0DnaXJp40jrthQWo/VGwrt0/ya8ossF2pnc5sxtSRvuTbrUfTalAwGks+kQ
A8t5pqRE7BpZ9NCrnpo1bnE/q3d5JhUuaSCScodpW/mILFE5rrc=
=lf+G
-----END PGP SIGNATURE-----

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]