Plans for the 0.2 release (this week)

Plans for the 0.2 release (this week)

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 1552 bytes --]

We've obviously got a lot of interest in notmuch, and a huge pile of
features that need to be merged.

I think that means we're in a state where we can have extremely regular
releases, with continually improving feature sets. I'm thinking releases
once per week or so.

With each release, I expect people to remind me of their favorite
features that haven't been merged, (while I'll continue to work through
any backlog on my own).

For the upcoming 0.2 release, here are some things that I would like to
have in place:

  * Any further changes from the Sebastian's repository. Sebastian, I
    worked through one list I saw recently. Do you have another list to
    propose yet?

  * The big batch of emacs-client improvements from David E.'s
    repository. David, do you have particular things to recommend here?

  * Changes to indexing, (addition of body:, folder:, list:, etc.). This
    is stuff that I'll work on.

  * Some library additions (move_to_first for the iterators, and perhaps
    a notmuch_database_add_message_with_data which Srinivasa requested
    to support integration of notmuch into evolution). I'll work on
    these as well, (I know that there are patches for some of these on
    the list already).

  * Anything else that people want, (especially things that already
    exist and that you're already using). Support for maildir flags on
    import would be great here. I'm still waiting to see a complete
    solution I think.

So keep the patches coming, and the pointers to patches that you want me
to look at.

-Carl

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Plans for the 0.2 release (this week)

[James Westby]

On Wed, 07 Apr 2010 15:12:44 -0700, Carl Worth <cworth@cworth.org> wrote:
>   * Anything else that people want, (especially things that already
>     exist and that you're already using). Support for maildir flags on
>     import would be great here. I'm still waiting to see a complete
>     solution I think.

id:1268515677-12692-1-git-send-email-jw+debian@jameswestby.net

I've been running with this locally since then with no apparent issues.

It will make anyone who suffers from this problem regularly pretty
happy, as they can return to the world of a thread-based mail client.

Thanks,

James

Re: Plans for the 0.2 release (this week)

[Sebastian Spaeth]

First of all, thanks for the great work Carl. I have to admit I was
getting nervous about the backlog of patches, but your recent committing
binge (you did say your work patterns are bursty :-)) made me very happy.

That having said, I am glad to meet your expectation: "I expect people
 to remind me of their favorite features that haven't been merged..."
 :-)

>   * Any further changes from the Sebastian's repository. Sebastian, I
>     worked through one list I saw recently. Do you have another list to
>     propose yet?

One of the things I have seen no comment on yet, is the
"json_quote_str should handle non-ASCII characters" patch in 
id:1267697893-sup-4538@sam.mediasupervision.de.

Right now, every char > 127 will simply be suppressed, this patch fixes
this for me but I cannot comment on its correctness. I rebased the patch
to current here:
http://github.com/spaetz/notmuch-all-feature/commit/3b5423fbdda5e78414e6f2e441cad5394173e155

Would that be an acceptable fix?

Also, as part of getting the useful backlog down, I propose these smallish
patches that are in my tree in the feature/misc branch (that I all grabbed from the mailing list):

* Add count command to manual page, Sandra Snan
  mail id:4ba29058.0f67f10a.2b59.1a73@mx.google.com
  commit 2f5ea4c76dc9688b2520c8d64ac8617ba8a62f86
  (it was really missing)

* Fix typo in notmuch.h documentation regarding database open modes 
  mail id:1269628757-24857-1-git-send-email-michael@obberon.com
  michaelforney, comit 3262b39f58ab4865c0c0cada69955ad65fccdfd9
  (our beloved notmuch.h is WRONG!!! :-)

* Prevent data loss caused by SIGINT during notmuch new
  mail id:1269937403-8495-1-git-send-email-sojkam1@fel.cvut.cz
  wentasah, commit dcb7a957027837f187f06d31fdcddb3740b2c20b
  (preventable data loss, has happened to him at least once)

* Do not segfault on empty mime parts, 
  mail id:1267543888-18134-1-git-send-email-madduck@madduck.net
  martin f. krafft, commit bb310c6397a94f6f59b6f93b232bedd833f96090
  (probably a bug in gmime, but we should not crash)

>   * Some library additions (move_to_first for the iterators,
+1!

All the rest souds very good.

Sebastian

Re: Plans for the 0.2 release (this week)

[David Edmondson]

On Wed, 07 Apr 2010 15:12:44 -0700, Carl Worth <cworth@cworth.org> wrote:
>   * The big batch of emacs-client improvements from David E.'s
>     repository. David, do you have particular things to recommend here?

It's necessary for me to merge with your latest batch of changes. That
won't happen until next week, when I can set aside a few hours. (The
merge is somewhat painful, as I have modified versions of some of the
recently applied patches in my tree.)

I worry about committing the JSON based Emacs UI and then immediately
producing a release - it would be useful to have more people test it
from git HEAD before dropping it on an unsuspecting public.

dme.
-- 
David Edmondson, http://dme.org

Re: Plans for the 0.2 release (this week)

[Anthony Towns]

On Thu, Apr 8, 2010 at 17:47, Sebastian Spaeth <Sebastian@sspaeth.de> wrote:
> * Prevent data loss caused by SIGINT during notmuch new
>  mail id:1269937403-8495-1-git-send-email-sojkam1@fel.cvut.cz
>  wentasah, commit dcb7a957027837f187f06d31fdcddb3740b2c20b
>  (preventable data loss, has happened to him at least once)

That seems to be a different id to
http://patchwork.notmuchmail.org/patch/445/ but "me too" on that,
fwiw. (Haven't had data loss, did have confusing behaviour that I
attribute to this, and the logic for the patch seems sound to me)

Cheers,
aj

-- 
Anthony Towns <aj@erisian.com.au>

Re: Plans for the 0.2 release (this week)

[Jameson Rollins]

[-- Attachment #1: Type: text/plain, Size: 622 bytes --]

Hey, Carl.  I'm very encouraged by the recent patch-apply fest, and the
proposal for more regular releases.  I have some new improvements I'll
try to get in at some point, but in the mean time...

> So keep the patches coming, and the pointers to patches that you want me
> to look at.

I would really like to see the patch in spaetz/issue15-handle-fcc-bcc
applied soon.  This is the lingering issue of bcc'ing the primary email
address in notmuch replies, which I think really needs to be removed.

Let me know if there are any issues.  I'm pretty sure this will apply
cleanly against the head without a rebase.

jamie.


[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: Plans for the 0.2 release (this week)

[Jameson Rollins]

[-- Attachment #1: Type: text/plain, Size: 799 bytes --]

On Wed, 07 Apr 2010 15:12:44 -0700, Carl Worth <cworth@cworth.org> wrote:
> For the upcoming 0.2 release, here are some things that I would like to
> have in place:
> 
>   * Changes to indexing, (addition of body:, folder:, list:, etc.). This
>     is stuff that I'll work on.

Also, fwiw, the folder: indexing is probably the new feature that I'm
most eagerly awaiting.  I've got all these ideas for ways to handle sent
mail and drafts if we can get this working.

Presumably others must be annoyed about having to manually "read" and
archive all their sent mail, unless there's some other way that people
having been dealing with this that I'm not aware of.

And as for drafts, they could be easily indexed and managed by the UI if
the folder: search term is available.

jamie.

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: Plans for the 0.2 release (this week)

[Michal Sojka]

On Wed, 07 Apr 2010, Carl Worth wrote:
> For the upcoming 0.2 release, here are some things that I would like to
> have in place:
> 
>   * Anything else that people want, (especially things that already
>     exist and that you're already using). Support for maildir flags on
>     import would be great here. I'm still waiting to see a complete
>     solution I think.

This would be useful: id:m1my03gsmu.fsf@watt.gilman.jhu.edu. I use it
occasionally.

-Michal

Re: Plans for the 0.2 release (this week)

[Mike Kelly]

On Wed, Apr 07, 2010 at 11:23:14PM +0100, James Westby wrote:
> On Wed, 07 Apr 2010 15:12:44 -0700, Carl Worth <cworth@cworth.org> wrote:
> >   * Anything else that people want, (especially things that already
> >     exist and that you're already using). Support for maildir flags on
> >     import would be great here. I'm still waiting to see a complete
> >     solution I think.
> 
> id:1268515677-12692-1-git-send-email-jw+debian@jameswestby.net

I haven't tested yet, but this is definitely a feature I want/need (due
to the... unique behavior of a certain bug tracker I use at work).

-- 
Mike Kelly

Re: Plans for the 0.2 release (this week)

[Dirk Hohndel]

On Thu, 08 Apr 2010 09:13:20 +0100, David Edmondson <dme@dme.org> wrote:
> On Wed, 07 Apr 2010 15:12:44 -0700, Carl Worth <cworth@cworth.org> wrote:
> >   * The big batch of emacs-client improvements from David E.'s
> >     repository. David, do you have particular things to recommend here?
> 
> It's necessary for me to merge with your latest batch of changes. That
> won't happen until next week, when I can set aside a few hours. (The
> merge is somewhat painful, as I have modified versions of some of the
> recently applied patches in my tree.)

I hope that this will be a less frequent thing to happen as Carl is
quicker in picking up patches. I'm very eager to see the emacs client
improvements, though.
 
> I worry about committing the JSON based Emacs UI and then immediately
> producing a release - it would be useful to have more people test it
> from git HEAD before dropping it on an unsuspecting public.

Not sure how much of an "unsuspecting public" we have at this
point... but in general this is of course valid. The question is "what
are releases". Or what does it mean to "release 0.2". I don't think we
have the notion of a stable branch and a development cycle,
yet. Everything is development (correct me if I'm wrong, Carl)

/D

-- 
Dirk Hohndel
Intel Open Source Technology Center

Re: Plans for the 0.2 release (this week)

[Dirk Hohndel]

On Thu, 08 Apr 2010 09:52:21 -0400, Jameson Rollins <jrollins@finestructure.net> wrote:
> > So keep the patches coming, and the pointers to patches that you want me
> > to look at.
> 
> I would really like to see the patch in spaetz/issue15-handle-fcc-bcc
> applied soon.  This is the lingering issue of bcc'ing the primary email
> address in notmuch replies, which I think really needs to be removed.

+1

The FCC solution is much more sane. Especially with the From address
based path selection that I proposed (but haven't been able to implement
for lack of Lisp skills)

/D

-- 
Dirk Hohndel
Intel Open Source Technology Center

Re: Plans for the 0.2 release (this week)

[Dirk Hohndel]

On Thu, 08 Apr 2010 10:03:15 -0400, Jameson Rollins <jrollins@finestructure.net> wrote:
> On Wed, 07 Apr 2010 15:12:44 -0700, Carl Worth <cworth@cworth.org> wrote:
> > For the upcoming 0.2 release, here are some things that I would like to
> > have in place:
> > 
> >   * Changes to indexing, (addition of body:, folder:, list:, etc.). This
> >     is stuff that I'll work on.
> 
> Also, fwiw, the folder: indexing is probably the new feature that I'm
> most eagerly awaiting.  I've got all these ideas for ways to handle sent
> mail and drafts if we can get this working.

Maybe I am missing some context here - what exactly does 'indexing' mean
here?
 
> Presumably others must be annoyed about having to manually "read" and
> archive all their sent mail, unless there's some other way that people
> having been dealing with this that I'm not aware of.

my import/tagging script deals with +sent on emails that I've sent and
removes -inbox -unread on them. The FCC logic gets them filed into a
Sent folder (right now just one - see previous comment about making this
From address dependent)
 
> And as for drafts, they could be easily indexed and managed by the UI if
> the folder: search term is available.

Hmm - haven't even thought about drafts, yet. How would the UI deal with
those? 

/D

-- 
Dirk Hohndel
Intel Open Source Technology Center

Re: Plans for the 0.2 release (this week)

[Michal Sojka]

On Thu, 08 Apr 2010, Dirk Hohndel wrote:
> On Thu, 08 Apr 2010 10:03:15 -0400, Jameson Rollins <jrollins@finestructure.net> wrote:
> > On Wed, 07 Apr 2010 15:12:44 -0700, Carl Worth <cworth@cworth.org> wrote:
> > > For the upcoming 0.2 release, here are some things that I would like to
> > > have in place:
> > > 
> > >   * Changes to indexing, (addition of body:, folder:, list:, etc.). This
> > >     is stuff that I'll work on.
> > 
> > Also, fwiw, the folder: indexing is probably the new feature that I'm
> > most eagerly awaiting.  I've got all these ideas for ways to handle sent
> > mail and drafts if we can get this working.
> 
> Maybe I am missing some context here - what exactly does 'indexing' mean
> here?

Indexing means calling notmuch new. The folder: patch, adds "folder:"
search prefix which allows searching for messages based on the folder
where the message is stored e.g. folder:sent.

-Michal

Re: Plans for the 0.2 release (this week)

[Mike Kelly]

On Thu, Apr 08, 2010 at 10:03:15AM -0400, Jameson Rollins wrote:
> On Wed, 07 Apr 2010 15:12:44 -0700, Carl Worth <cworth@cworth.org> wrote:
> > For the upcoming 0.2 release, here are some things that I would like to
> > have in place:
> > 
> >   * Changes to indexing, (addition of body:, folder:, list:, etc.). This
> >     is stuff that I'll work on.
> 
> Also, fwiw, the folder: indexing is probably the new feature that I'm
> most eagerly awaiting.  I've got all these ideas for ways to handle sent
> mail and drafts if we can get this working.

For me this, along with handling maildir flags, are pretty much my top
"must haves" before I think I would be able to begin to replace
mutt/whatever with notmuch.

-- 
Mike Kelly

draft handling [was: Re: Plans for the 0.2 release (this week)]

[Jameson Rollins]

[-- Attachment #1: Type: text/plain, Size: 1012 bytes --]

On Thu, 08 Apr 2010 07:58:47 -0700, Dirk Hohndel <hohndel@infradead.org> wrote:
> > And as for drafts, they could be easily indexed and managed by the UI if
> > the folder: search term is available.
> 
> Hmm - haven't even thought about drafts, yet. How would the UI deal with
> those? 

My idea is the following:

Currently for me drafts are stored in ~/.mail/drafts.  If we have
folder: searching, then I can starting indexing my drafts directory and
apply the following post-new filter:

notmuch tag +draft -inbox -unread -- folder:drafts

I'll then make a simple elisp "resume" function that is available for
resuming draft messages, that would be available when viewing messages
From a draft search results.

One could also imagine a "notmuch-draft" emacs mode that would provide
functions specifically for viewing and resuming draft messages.

Anyway, my plan was to work on all of this when some sane way of
indexing draft messages was available (ie. folder: searching).

jamie.

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: Plans for the 0.2 release (this week)

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 3883 bytes --]

On Thu, 08 Apr 2010 07:58:47 -0700, Dirk Hohndel <hohndel@infradead.org> wrote:
> On Thu, 08 Apr 2010 10:03:15 -0400, Jameson Rollins <jrollins@finestructure.net> wrote:
> > Also, fwiw, the folder: indexing is probably the new feature that I'm
> > most eagerly awaiting.  I've got all these ideas for ways to handle sent
> > mail and drafts if we can get this working.
> 
> Maybe I am missing some context here - what exactly does 'indexing' mean
> here?

Indexing here means shoving information into the database to make it
searchable.

I was a bit vague in my description of the features I want to enable
here. So let me start making it a bit more clear. I'd like to add
support for the following:

  1. A "folder:" prefix in the search syntax that will match components
     of the directory in which mail files are stored. So, if you've got
     mail filed into directories, (from using a folder-based email
     program, or from doing an imap sync from gmail that turns tags into
     directories), then you could search for things like "folder:spam",
     "folder:important", "folder:arbitrary/subset/of/some/path", etc.

  2. A "list:" prefix to match content from headers that identify
     mailing list. My perception is that there are likely a handful of
     different headers that have been used, and they should all be
     indexed so that "list:" will search any of them.

  3. Some prefix that can be used to match typical headers added by
     spam-filtering software, (maybe this would be a general
     prefix---see below).

I think the above are probably the three I can think of that pretty much
everybody has asked for, so that should be indexed by default.

Beyond that, I'd like to be able to provide support for arbitrary
headers in the email. I had envisioned allowing the user to configure
specific headers to index. Alternately, I had imagined having a
blacklist of "uninteresting" headers and indexing everything
else. (Though, now Dirk is very interested in Received and its perhaps
provides the most content of any of the headers I originally thought
would be "uninteresting".)

One experiment I should do is to measure how much my database would grow
if I were to index all of the header content. I don't really know what
kind of a percentage we're talking about.

Finally, we'd need a syntax for searching all of these headers. Rather
than trying to map each header name to a custom prefix, I think what
might be best would be a general thing that could look like:

	header:"X-Spam-Flag: YES"

We don't currently have the ability to tie a search to the beginning of
a line, but it occurs to me that we could do something fancy like index
each header with a specific term to indicate line beginnings. Then, with
a custom query parser we could use the common symbol of '^' to map to
that. That would enable something like:

	header:"^X-Spam-Flag: YES"

That's a lot of work though, (and perhaps not as important as it's
probably uncommon for a header name to appear within the value of some
other header).

> Hmm - haven't even thought about drafts, yet. How would the UI deal with
> those? 

I would imagine another "folder" alongside the others that would list
all drafts, and selecting any such message would bring up the message in
a mode to edit it.

Sup also displayed draft messages in their proper location in the
threads containing the message being replied to, (and highlighted such
threads in the search view).

All of that sounds quite easy to do by simply saving the draft within
the mailstore and giving it a "draft" tag.

Finally, many mail interfaces prompt from the "compose new message"
command whether an existing draft should be continued. That's useful to
help the user avoid forgetting to complete and send a draft that was
forgotten.

-Carl



[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Plans for the 0.2 release (this week)

[Jameson Rollins]

[-- Attachment #1: Type: text/plain, Size: 317 bytes --]

On Thu, 08 Apr 2010 14:28:20 -0700, Carl Worth <cworth@cworth.org> wrote:
> All of that sounds quite easy to do by simply saving the draft within
> the mailstore and giving it a "draft" tag.

The only way I can see to give drafts a "draft" tag is via 'folder:'
indexing.  Do you have any ideas of other ways?

jamie.

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: Plans for the 0.2 release (this week)

[Mike Kelly]

[-- Attachment #1: Type: text/plain, Size: 476 bytes --]

On Thu, 08 Apr 2010 14:28:20 -0700
Carl Worth <cworth@cworth.org> wrote:

>   2. A "list:" prefix to match content from headers that identify
>      mailing list. My perception is that there are likely a handful of
>      different headers that have been used, and they should all be
>      indexed so that "list:" will search any of them.

Almost all modern mailing list software seems to use List-Id:

    https://tools.ietf.org/html/rfc2919

-- 
Mike Kelly


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Plans for the 0.2 release (this week)

[Anthony Towns]

On Fri, Apr 9, 2010 at 00:03, Jameson Rollins
<jrollins@finestructure.net> wrote:
> Presumably others must be annoyed about having to manually "read" and
> archive all their sent mail, unless there's some other way that people
> having been dealing with this that I'm not aware of.

I haven't switched over to notmuch properly yet, but in my initial
import of my mail I made a couple of patches to notmuch new that let
me get the tags right first. One was to make:

    notmuch new --initial-tags=list,unread

set the initial tags for newly found messages to "list" and "unread"
instead of the default "inbox" and "unread".

The other was to make:

    find Mail/.Saved/ -type f | notmuch new --initial-tags=archived --file-list

only process the mail files listed on stdin (via find), and give them
the explicit tags I specify. That way I could import all my existing
archived mail without it appearing in my inbox or as unread.

Unfortunately the second one ended up complicated and a bit slow (I
think because I'm doing a talloc() for every line on stdin, and
talloc_free() by context every time the base directory changes; that
sort of behaviour was necessary in order to do duplicate checking in a
sane way)

But anyway, that would let you do:

    find Mail/.Drafts/ -type f | notmuch new --initial-tags=draft --file-list
    notmuch new

to get drafts correctly tagged.

(I don't have the patches handy at the moment; but can certainly dig
them up if there's interest)

Cheers,
aj

-- 
Anthony Towns <aj@erisian.com.au>

Re: Plans for the 0.2 release (this week)

[Michal Sojka]

On Fri, 09 Apr 2010, Anthony Towns wrote:
> On Fri, Apr 9, 2010 at 00:03, Jameson Rollins
> <jrollins@finestructure.net> wrote:
> > Presumably others must be annoyed about having to manually "read" and
> > archive all their sent mail, unless there's some other way that people
> > having been dealing with this that I'm not aware of.
> 
> I haven't switched over to notmuch properly yet, but in my initial
> import of my mail I made a couple of patches to notmuch new that let
> me get the tags right first. One was to make:
> 
>     notmuch new --initial-tags=list,unread
> 
> set the initial tags for newly found messages to "list" and "unread"
> instead of the default "inbox" and "unread".
> 
> The other was to make:
> 
>     find Mail/.Saved/ -type f | notmuch new --initial-tags=archived --file-list
> 
> only process the mail files listed on stdin (via find), and give them
> the explicit tags I specify. That way I could import all my existing
> archived mail without it appearing in my inbox or as unread.

Hmm, quite interesting feature. I think that several people asked for
this in the past. I'd be happy to see the patches.

-Michal

Re: Plans for the 0.2 release (this week)

[Michal Sojka]

On Wed, 07 Apr 2010, Carl Worth wrote:
>   * Anything else that people want, (especially things that already
>     exist and that you're already using). Support for maildir flags on
>     import would be great here. I'm still waiting to see a complete
>     solution I think.
> 
> So keep the patches coming, and the pointers to patches that you want me
> to look at.

"Decode headers in reply"
(id:1267602656-24940-1-git-send-email-sojkam1@fel.cvut.cz) is another
patch, which is quite essential to me. Am I the only one here, who needs
to reply to messages with non-ASCII characters?

-Michal

Re: Plans for the 0.2 release (this week)

[Sebastian Spaeth]

On 2010-04-09, Michal Sojka wrote:
> "Decode headers in reply"
> (id:1267602656-24940-1-git-send-email-sojkam1@fel.cvut.cz) is another
> patch, which is quite essential to me. Am I the only one here, who needs
> to reply to messages with non-ASCII characters?

Nope, and it looks currently very ugly. Having a good solution for that
problem would be nice indeed.

Perhaps Carl should get more Nørwég¡añ friends, :-).

spaetz

Re: Plans for the 0.2 release (this week)

[micah anderson]

[-- Attachment #1: Type: text/plain, Size: 571 bytes --]

On 2010-04-08, micah anderson wrote:
> On Wed, 07 Apr 2010 15:12:44 -0700, Carl Worth <cworth@cworth.org> wrote:
> > For the upcoming 0.2 release, here are some things that I would like to
> > have in place:
> > 
> >   * Changes to indexing, (addition of body:, folder:, list:, etc.). This
> >     is stuff that I'll work on.
> 
> Also, fwiw, the folder: indexing is probably the new feature that I'm
> most eagerly awaiting.  I've got all these ideas for ways to handle sent
> mail and drafts if we can get this working.

+1 on folder: capability!

micah

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: Plans for the 0.2 release (this week)

[Dirk Hohndel]

On Fri, 09 Apr 2010 09:35:07 +0200, "Sebastian Spaeth" <Sebastian@SSpaeth.de> wrote:
> On 2010-04-09, Michal Sojka wrote:
> > "Decode headers in reply"
> > (id:1267602656-24940-1-git-send-email-sojkam1@fel.cvut.cz) is another
> > patch, which is quite essential to me. Am I the only one here, who needs
> > to reply to messages with non-ASCII characters?
> 
> Nope, and it looks currently very ugly. Having a good solution for that
> problem would be nice indeed.
> 
> Perhaps Carl should get more Nørwég¡añ friends, :-).

Or Görmän or 中文

Yes, I ran into that myself as my brother's first name is Jürgen and he
complained about my emails to him suddenly being mangled...

But then, Sebastian doesn't even spell his own last name correctly :-)

/D

-- 
Dirk Hohndel
Intel Open Source Technology Center

Re: Plans for the 0.2 release (this week)

[Carl n∅tmuch 䚳 Worth]

[-- Attachment #1: Type: text/plain, Size: 561 bytes --]

> On Fri, 09 Apr 2010 09:35:07 +0200, "Sebastian Spaeth" <Sebastian@SSpaeth.de> wrote:
> > On 2010-04-09, Michal Sojka wrote:
> > Perhaps Carl should get more Nørwég¡añ friends, :-).
> 
> Or Görmän or 中文

Are you all trying to show a problem here? All of the above comes
through fine. Perhaps it's only with non-ASCII in the From line being
replied to? Let's test that here...

-Carl

PS. How about this for something interesting from Unicode:

䚳 Definition in English: do not know, to know nothing about, quickly;
fast, sharp; keen

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Plans for the 0.2 release (this week)

[Dirk Hohndel]

here's what's going wrong. Look at the To: line...

/D

On Fri, 09 Apr 2010 20:06:09 -0700, Carl n∅tmuch 䚳 Worth <cworth@cworth.org> wrote:
> > On Fri, 09 Apr 2010 09:35:07 +0200, "Sebastian Spaeth" <Sebastian@SSpaeth.de> wrote:
> > > On 2010-04-09, Michal Sojka wrote:
> > > Perhaps Carl should get more Nørwég¡añ friends, :-).
> > 
> > Or Görmän or 中文
> 
> Are you all trying to show a problem here? All of the above comes
> through fine. Perhaps it's only with non-ASCII in the From line being
> replied to? Let's test that here...
> 
> -Carl
> 
> PS. How about this for something interesting from Unicode:
> 
> 䚳 Definition in English: do not know, to know nothing about, quickly;
> fast, sharp; keen
Non-text part: application/pgp-signature

-- 
Dirk Hohndel
Intel Open Source Technology Center

Re: Plans for the 0.2 release (this week)

[Dirk Hohndel]

On Fri, 09 Apr 2010 21:44:02 -0700, Dirk Hohndel <hohndel@infradead.org> wrote:
> 
> here's what's going wrong. Look at the To: line...

Carl =?UTF-8?b?buKIhXRtdWNoIOSasw==?= Worth <cworth@cworth.org>,

that's not pretty... nor readable.

/D

-- 
Dirk Hohndel
Intel Open Source Technology Center

Re: Plans for the 0.2 release (this week)

[Michal Sojka]

On Sat, 10 Apr 2010, Carl n∅tmuch 䚳 Worth wrote:
> > On Fri, 09 Apr 2010 09:35:07 +0200, "Sebastian Spaeth" <Sebastian@SSpaeth.de> wrote:
> > > On 2010-04-09, Michal Sojka wrote:
> > > Perhaps Carl should get more Nørwég¡añ friends, :-).
> > 
> > Or Görmän or 中文
> 
> Are you all trying to show a problem here? All of the above comes
> through fine. Perhaps it's only with non-ASCII in the From line being
> replied to? 

Yes and also in Subject line. You can test this with
id:87r5o1etjb.fsf@steelpick.localdomain.

Michal

Re: Plans for the 0.2 release (this week)

[Jameson Rollins]

[-- Attachment #1: Type: text/plain, Size: 626 bytes --]

On Fri, 09 Apr 2010 21:47:20 -0700, Dirk Hohndel <hohndel@infradead.org> wrote:
> On Fri, 09 Apr 2010 21:44:02 -0700, Dirk Hohndel <hohndel@infradead.org> wrote:
> > 
> > here's what's going wrong. Look at the To: line...
> 
> Carl =?UTF-8?b?buKIhXRtdWNoIOSasw==?= Worth <cworth@cworth.org>,

The To: line shows up fine in notmuch-show for me (in emacs).  It only
shows up like this when editing.  I think this is actually a general
mail issue, and is not a notmuch issue.  I don't understand all the
issues, but I think non-ASCII characters in headers have to be encoded,
and that's what you're seeing.

jamie.

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: Plans for the 0.2 release (this week)

[Jameson Rollins]

[-- Attachment #1: Type: text/plain, Size: 671 bytes --]

On Sat, 10 Apr 2010 11:02:19 +0200, Michal Sojka <sojkam1@fel.cvut.cz> wrote:
> On Sat, 10 Apr 2010, Carl n∅tmuch 䚳 Worth wrote:
> > Are you all trying to show a problem here? All of the above comes
> > through fine. Perhaps it's only with non-ASCII in the From line being
> > replied to? 
> 
> Yes and also in Subject line. You can test this with
> id:87r5o1etjb.fsf@steelpick.localdomain.

Again, this subject line shows up fine for me in notmuch-show in emacs.
I believe the non-ASCII characters in headers have to be escaped for
mail handlers to properly handle them, and the reader should translate.
I need to find a reference for this...

jamie.

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: Plans for the 0.2 release (this week)

[Jameson Rollins]

[-- Attachment #1: Type: text/plain, Size: 646 bytes --]

On Sat, 10 Apr 2010 09:06:54 -0400, Jameson Rollins <jrollins@finestructure.net> wrote:
> Again, this subject line shows up fine for me in notmuch-show in emacs.
> I believe the non-ASCII characters in headers have to be escaped for
> mail handlers to properly handle them, and the reader should translate.
> I need to find a reference for this...

I think this is the relevant RFC:

http://tools.ietf.org/html/rfc5335

Basically, I think that emacs is doing the right thing here.  This isn't
a notmuch issue at all.  The headers have to be encoded this way for the
MTAs.  Your reader (ie. emacs) should be doing the translation for you.

jamie.

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: Plans for the 0.2 release (this week)

[Sebastian Spaeth]

On 2010-04-10, Carl n∅tmuch 䚳 Worth wrote:
> Are you all trying to show a problem here? All of the above comes
> through fine. Perhaps it's only with non-ASCII in the From line being
> replied to? Let's test that here...

I think there are 2 issues being discussed here. One is the encoding in
the subject line (which does not look pretty in compose mail, but seems
to be the standard). I did not refer to that.

What I referred to was the json output as the encode_as_json_string (or
however it is called), simply drops chars >127, leading to missing
letters in e.g. the author names etc. I could see that in the web
frontends that are making use of json already, and once (if?) emacs uses
the json output too, this will become an issue there too.

Certainly not urgent, but it looked quite weird in the web frontends.

Sebastian

Re: Plans for the 0.2 release (this week)

[Jameson Rollins]

[-- Attachment #1: Type: text/plain, Size: 243 bytes --]

On Sat, 10 Apr 2010 15:52:04 +0200, "Sebastian Spaeth" <Sebastian@SSpaeth.de> wrote:
> frontends that are making use of json already, and once (if?) emacs uses
> the json output too, this will become an issue there too.

once, not if!

jamie.

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: Plans for the 0.2 release (this week)

[Michael Elkins]

On Sat, Apr 10, 2010 at 09:06:54AM -0400, Jameson Rollins wrote:
> Again, this subject line shows up fine for me in notmuch-show in emacs.
> I believe the non-ASCII characters in headers have to be escaped for
> mail handlers to properly handle them, and the reader should translate.
> I need to find a reference for this...

http://tools.ietf.org/html/rfc2047

The problem is not the MTA, but the fact that the sender and recipient
may be using different character sets.  There is no way to decode it
properly unless the sender specifies the character set.

me

Re: Plans for the 0.2 release (this week)

[Dirk Hohndel]

On Sat, 10 Apr 2010 09:05:00 -0400, Jameson Rollins <jrollins@finestructure.net> wrote:
> On Fri, 09 Apr 2010 21:47:20 -0700, Dirk Hohndel <hohndel@infradead.org> wrote:
> > On Fri, 09 Apr 2010 21:44:02 -0700, Dirk Hohndel <hohndel@infradead.org> wrote:
> > > 
> > > here's what's going wrong. Look at the To: line...
> > 
> > Carl =?UTF-8?b?buKIhXRtdWNoIOSasw==?= Worth <cworth@cworth.org>,
> 
> The To: line shows up fine in notmuch-show for me (in emacs).  It only
> shows up like this when editing.  I think this is actually a general
> mail issue, and is not a notmuch issue.  I don't understand all the
> issues, but I think non-ASCII characters in headers have to be encoded,
> and that's what you're seeing.

Yes, I understand the RFCs and encoding fairly well. That wasn't my
point. The point is that for some reason we display the encoded text
shown above in the Messages Mode buffer, instead of displaying the
non-ASCII characters as we (IMHO) should.

Maybe this is an issue of different versions of Emacs behaving slightly
differently as Carl seems not to have this issue...

I'm on GNU Emacs 23.1.1 (i386-redhat-linux-gnu, GTK+ Version 2.18.6)
 of 2010-01-14 on x86-02.phx2.fedoraproject.org

/D

-- 
Dirk Hohndel
Intel Open Source Technology Center

Re: Plans for the 0.2 release (this week)

[Dirk Hohndel]

On Sat, 10 Apr 2010 15:52:04 +0200, "Sebastian Spaeth" <Sebastian@SSpaeth.de> wrote:
> On 2010-04-10, Carl n∅tmuch 䚳 Worth wrote:
> > Are you all trying to show a problem here? All of the above comes
> > through fine. Perhaps it's only with non-ASCII in the From line being
> > replied to? Let's test that here...
> 
> I think there are 2 issues being discussed here. One is the encoding in
> the subject line (which does not look pretty in compose mail, but seems
> to be the standard). I did not refer to that.

That's the issue I refer to. Yes, it needs to be decoded for the actual
mail transport. But it should be displayed correctly not only in the
summary view but also in the message buffer when responding to an email.
 
> What I referred to was the json output as the encode_as_json_string (or
> however it is called), simply drops chars >127, leading to missing
> letters in e.g. the author names etc. I could see that in the web
> frontends that are making use of json already, and once (if?) emacs uses
> the json output too, this will become an issue there too.

Separate issue that almost certainly has a different cause.

/D

-- 
Dirk Hohndel
Intel Open Source Technology Center

Re: Plans for the 0.2 release (this week)

[Michal Sojka]

On Sat, 10 Apr 2010, Jameson Rollins wrote:
> On Sat, 10 Apr 2010 11:02:19 +0200, Michal Sojka <sojkam1@fel.cvut.cz> wrote:
> > On Sat, 10 Apr 2010, Carl n∅tmuch 䚳 Worth wrote:
> > > Are you all trying to show a problem here? All of the above comes
> > > through fine. Perhaps it's only with non-ASCII in the From line being
> > > replied to? 
> > 
> > Yes and also in Subject line. You can test this with
> > id:87r5o1etjb.fsf@steelpick.localdomain.
> 
> Again, this subject line shows up fine for me in notmuch-show in emacs.
> I believe the non-ASCII characters in headers have to be escaped for
> mail handlers to properly handle them, and the reader should translate.
> I need to find a reference for this...

Yes. It is displayed correctly in notmuch-show mode, but try to reply
(press 'r') to the message and you will see. If you do not see it in
emacs, try running

  notmuch reply id:87r5o1etjb.fsf@steelpick.localdomain

and you will see the encoded headers. There is no problem with sending
such a mail. All recipients will see the headers decoded correctly. The
problem is, that I (the one who writes the reply) do not know what is
written in subject and what are the names of non-ascii named recipients.

If notmuch reply command decodes the headers, as implemented by the
patch, then Emacs (or some other part of mail sending chain?) will
encode the headers again before the message is actually sent.

-Michal

Re: Plans for the 0.2 release (this week)

[Mike Kelly]

On Thu, Apr 08, 2010 at 10:34:59AM -0400, Mike Kelly wrote:
> On Wed, Apr 07, 2010 at 11:23:14PM +0100, James Westby wrote:
> > On Wed, 07 Apr 2010 15:12:44 -0700, Carl Worth <cworth@cworth.org> wrote:
> > >   * Anything else that people want, (especially things that already
> > >     exist and that you're already using). Support for maildir flags on
> > >     import would be great here. I'm still waiting to see a complete
> > >     solution I think.
> > 
> > id:1268515677-12692-1-git-send-email-jw+debian@jameswestby.net
> 
> I haven't tested yet, but this is definitely a feature I want/need (due
> to the... unique behavior of a certain bug tracker I use at work).

I've now tested this patch, and it appears to work properly. Thanks!

-- 
Mike Kelly

Re: Plans for the 0.2 release (this week)

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 1514 bytes --]

On Thu, 08 Apr 2010 09:47:00 +0200, "Sebastian Spaeth" <Sebastian@SSpaeth.de> wrote:
> First of all, thanks for the great work Carl. I have to admit I was
> getting nervous about the backlog of patches, but your recent committing
> binge (you did say your work patterns are bursty :-)) made me very happy.

You're quite welcome. And the big backlog was making me nervous too.

I'm not sure that the backlog has gotten any smaller yet, but I'm really
happy with where things are going. I seem to have created a code hydra,
where every time I merge one patch, people get excited to and send me
three more patches.

> That having said, I am glad to meet your expectation: "I expect people
>  to remind me of their favorite features that haven't been merged..."
>  :-)

Thanks for the very useful list, Sebastian. These are all merged now,
(as mentioned in the replies I've made to the original messages with the
patches).

> >   * Some library additions (move_to_first for the iterators,
> +1!

That one I've got noted on a separate list.

> All the rest souds very good.

I really do need a thread-splitting feature in order to use notmuch as
my feature tracker. A list like the above of "6 trivial patches to
merge" is easy to handle as a single item. I can send a message like
this one saying "I'm done with this list".

But several different people did reply to my original "plans for 0.2"
call for features. So the single thread I have for this in my "notmuch
merge window" search is not doing the trick.

-Carl

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Plans for the 0.2 release (this week)

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 904 bytes --]

On Thu, 08 Apr 2010 09:52:21 -0400, Jameson Rollins <jrollins@finestructure.net> wrote:
> I would really like to see the patch in spaetz/issue15-handle-fcc-bcc
> applied soon.  This is the lingering issue of bcc'ing the primary email
> address in notmuch replies, which I think really needs to be removed.
> 
> Let me know if there are any issues.  I'm pretty sure this will apply
> cleanly against the head without a rebase.

I'm happy to remove the Bcc thing, but only *after* we have an Fcc
approach integrated into the emacs client by default, (without users
having to configure it, download python scripts, etc.).

And yes, I know the current Bcc approach is still quite useless, because
it only triggers on replies, (and not when composing a new message from
the emacs client).

I'm looking forward to seeing this all fixed, but I haven't yet seen a
"packaged" solution.

-Carl

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Plans for the 0.2 release (this week)

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 1887 bytes --]

On Wed, 07 Apr 2010 15:12:44 -0700, Carl Worth <cworth@cworth.org> wrote:
> For the upcoming 0.2 release, here are some things that I would like to
> have in place:
> 
>   * Any further changes from the Sebastian's repository. Sebastian, I
>     worked through one list I saw recently. Do you have another list to
>     propose yet?

Sebastian sent a list of these, and they are merged now.

>   * The big batch of emacs-client improvements from David E.'s
>     repository. David, do you have particular things to recommend
>     here?

David has no followed up with a rebase of his entire branch. It looks
*very* nice still but won't be in 0.2. I plan to land this early and
make it the focus of the 0.3 release.

>   * Changes to indexing, (addition of body:, folder:, list:, etc.). This
>     is stuff that I'll work on.

I haven't gotten to this yet. It's a big change, (in that it requires
everyone to rebuild their database). So this will need some careful
testing. And I'd like to get as many changes in at once so we won't all
have to rebuild are databases in the following release. This stuff will
be the focus of the 0.4 release.

>   * Some library additions (move_to_first for the iterators, and perhaps
>     a notmuch_database_add_message_with_data which Srinivasa requested
>     to support integration of notmuch into evolution). I'll work on
>     these as well, (I know that there are patches for some of these on
>     the list already).

I'll take a look at the existing patches for the iterators. That's still
0.2 material. The add_message_with_data feature will come later.

>   * Anything else that people want, (especially things that already
>     exist and that you're already using).

Thanks for several suggestions in reply to this message. I think they
are all merged now. Let me know if I missed anything.

-Carl

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]