Introducing myself

Introducing myself

[Adrian Perez de Castro]

[-- Attachment #1: Type: text/plain, Size: 2083 bytes --]


Hello to all,

I have just heard about Not Much today in some random Linux-related news
site (LWN?), my name is Adrian Perez and I work as systems administrator
(although I can do some code as well :P). I have always thought that the
ideas behind Sup were great, but after some time using it, I got tired of
the oddities that it has. I also do not like doing things like having to
install Ruby just for reading and sorting mails. Some time ago I thought
about doing something like Not Much and in fact I played a bit with the
Python+Xapian and the Python+Whoosh combinations, because I find relaxing
to code things in Python when I am not working and also it is installed
by default on most distribution. I got to have some mailboxes indexed and
basic searching working a couple of months ago. Lately I have been very
busy and had no time for coding, and them... boom! Not Much appears -- and
it is almost exactly what I was trying to do, but faster. I have been
playing a bit with Not Much today, and I think it has potential.

Also, I would like to share one idea I had in mind, that you might find
interesting: One thing I have found very annoying is having to re-tag my
mail when the indexes get b0rked (it happened a couple of times to me while
using Sup), so I was planning to mails as read/unread and adding the tags
not just to the index, but to the mail text itself, e.g. by adding a
"X-Tags" header field or by reusing the "Keywords" one. This way, the index
could be totally recreated by re-reading the mail directories, and this
would also allow to a tools like OfflineIMAP [1] to get the mails into a
local maildir, tagging and indexing the mails with the e-mail reader and
then syncing back the messages with the "X-Tags" header to the IMAP server.
This would allow to use the mail reader from a different computer and still
have everything tagged finely.

Best regards,


---
[1] http://software.complete.org/software/projects/show/offlineimap

-- 
Adrian Perez de Castro <aperez@igalia.com>
Igalia - Free Software Engineering

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Introducing myself

[Keith Packard]

On Wed, 18 Nov 2009 00:20:59 +0100, Adrian Perez de Castro <aperez@igalia.com> wrote:

> Some time ago I thought
> about doing something like Not Much and in fact I played a bit with the
> Python+Xapian and the Python+Whoosh combinations, because I find relaxing
> to code things in Python when I am not working and also it is installed
> by default on most distribution. I got to have some mailboxes indexed and
> basic searching working a couple of months ago.

Sup certainly started a lot of people thinking...

> Also, I would like to share one idea I had in mind, that you might find
> interesting: One thing I have found very annoying is having to re-tag my
> mail when the indexes get b0rked (it happened a couple of times to me while
> using Sup), so I was planning to mails as read/unread and adding the tags
> not just to the index, but to the mail text itself, e.g. by adding a
> "X-Tags" header field or by reusing the "Keywords" one.

Easier than that, notmuch (and sup too), provide a 'dump' command which
just lists all of the message IDs and their associated tags. Makes
saving tags easy and doesn't involve rewriting messages. I do this once
a day just before my computer is backed up to an external drive.

If the index is destroyed, you can reindex the messages and then reapply
all of the tags with 'notmuch restore'.

--
keith.packard@intel.com

Re: Introducing myself

[Carl Worth]

On Wed, 18 Nov 2009 00:20:59 +0100, Adrian Perez de Castro <aperez@igalia.com> wrote:
> I have just heard about Not Much today in some random Linux-related news
> site (LWN?), my name is Adrian Perez and I work as systems administrator

Welcome to notmuch, Adrian! We're glad to have you here.

> by default on most distribution. I got to have some mailboxes indexed and
> basic searching working a couple of months ago. Lately I have been very
> busy and had no time for coding, and them... boom! Not Much appears -- and
> it is almost exactly what I was trying to do, but faster. I have been
> playing a bit with Not Much today, and I think it has potential.

It's funny, because I had the exact same experience with sup a couple of
months ago. I had been frustrated for years with email programs, and had
been thinking about how I'd like things to work n the back of my mind
for a long time, (but never *quite* getting to the point where I would
commit to writing an email system myself).

And then... boom! I found sup and was instantly hooked. It had so much
of what I had imagined, (and much of what I hadn't yet imagined) that I
was quite delighted.

It was really quite by accident that I ended up inventing a different
system. I had started out just trying to speedup index creation for sup.
If I hadn't run into the problem that it was very difficult[*] to create a
sup-compatible index from C code, I might have stopped there.

So I'd written a bunch of functional code, only to find myself stuck at
the very last step, (hooking it up to the existing sup interface). Then
Keith suggested emacs and it all seemed pretty easy since I'd already
done all the Xapian work. So it's funny, I was only willing to commit to
this project because I wasn't consciously aware I was working on it.
Otherwise it would have seemed to overwhelming to start. :-)

Anyway, that's a lot of off-topic rambling off of your introduction. But
I'm glad that notmuch can now give that same "boom!" to others, and I'm
glad you see potential in it.

> Also, I would like to share one idea I had in mind, that you might find
> interesting: One thing I have found very annoying is having to re-tag my
> mail when the indexes get b0rked (it happened a couple of times to me while
> using Sup), so I was planning to mails as read/unread and adding the tags
> not just to the index, but to the mail text itself, e.g. by adding a
> "X-Tags" header field or by reusing the "Keywords" one. This way, the index
> could be totally recreated by re-reading the mail directories, and this
> would also allow to a tools like OfflineIMAP [1] to get the mails into a
> local maildir, tagging and indexing the mails with the e-mail reader and
> then syncing back the messages with the "X-Tags" header to the IMAP server.
> This would allow to use the mail reader from a different computer and still
> have everything tagged finely.

It is an interesting idea. But there's also something really comforting
about the email indexed never modifying the mail files. If you're
reading the notmuch commit logs closely you'll see that I'm not actually
careful enough to be trusted with your mail (but I try). So I like that
I don't even have to trust myself---the worst that happens is that I
have to recreate my index.

And as Keith mentioned, we've got the "notmuch dump; notmuch restore"
idea working exactly as it did in sup. (Though I am thinking of also
adding thread IDs to that now---more on that later.)

The big annoyance I had with sup index creation, (I ended up having to
do it more than once too), was that it takes *forever*. Right now,
notmuch is a little bit faster, but not a lot faster. And I've got some
ideas to fix that. It would be really nice if index creation were pain
free. (And maybe it is for some user with small amounts of mail---oh, to
have only 40000 messages to have to index!).

-Carl

[*] The problem here is that sup puts serialized ruby data structures
into the data field of its Xapian documents. So being compatible with
sup means being able to recreate serialized data structures for a
particular version of ruby.

Re: Introducing myself

[Adrian Perez de Castro]

[-- Attachment #1: Type: text/plain, Size: 1648 bytes --]

On Tue, 17 Nov 2009 18:03:17 -0800, Keith wrote:

> On Wed, 18 Nov 2009 00:20:59 +0100, Adrian Perez de Castro <aperez@igalia.com> wrote:
>
> > Also, I would like to share one idea I had in mind, that you might find
> > interesting: One thing I have found very annoying is having to re-tag my
> > mail when the indexes get b0rked (it happened a couple of times to me while
> > using Sup), so I was planning to mails as read/unread and adding the tags
> > not just to the index, but to the mail text itself, e.g. by adding a
> > "X-Tags" header field or by reusing the "Keywords" one.
> 
> Easier than that, notmuch (and sup too), provide a 'dump' command which
> just lists all of the message IDs and their associated tags. Makes
> saving tags easy and doesn't involve rewriting messages. I do this once
> a day just before my computer is backed up to an external drive.
> 
> If the index is destroyed, you can reindex the messages and then reapply
> all of the tags with 'notmuch restore'.

I already knew about "dump" and "restore", just wanted to share my
(wild) thought. I have been thinking a bit more, and if I understand
correctly the philosophy of Not Much is that it will only read my mail
and that it will never write to my mailboxes. So my idea makes no sense.
The nice thing is that I could make a script to read the index (or the
output of "notmuch dump") to add the "X-Tags" headers and mark mail :P
Taking into account this and the "do one thing well" motto: there is no
point in adding that to Not Much :)

Regards,

-- 
Adrian Perez de Castro <aperez@igalia.com>
Igalia - Free Software Engineering

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Introducing myself

[Adrian Perez de Castro]

[-- Attachment #1: Type: text/plain, Size: 3881 bytes --]

On Wed, 18 Nov 2009 03:15:31 -0800, Carl wrote:

> So I'd written a bunch of functional code, only to find myself stuck at
> the very last step, (hooking it up to the existing sup interface). Then
> Keith suggested emacs and it all seemed pretty easy since I'd already
> done all the Xapian work. So it's funny, I was only willing to commit to
> this project because I wasn't consciously aware I was working on it.
> Otherwise it would have seemed to overwhelming to start. :-)

I tried the "notmuch.el" Emacs mode, and have just installed Emacs in
order to be able to try it out. As I have been using Vim for the last ~9
years, probably that was the reason why I found it not very comfortable.
Even so, I am able of searching, sending and tagging mail quite happily.
Apart from it being a bit Spartan, the only issue I have found so far is
that I keep pushing the Escape key a lot -- which is my fault as that is a
compulsive behaviour derived of using a Vi-like editor :D

(Side note: I would not like to start an editor-religion debate, I just
want you to know that most probably it is my fault when I say that I am
not very comfortable with the Emacs mode)

I showed the thing to my work mates this morning and most of them said
that they would like to have Mutt-like keybindings *with* Not Much's
search and tagging running behind it. I also think it would be great to
have a curses-based UI, so this may be a good opportunity to start doing
some serious coding with it in my spare time.
 
> > Also, I would like to share one idea [...] adding a "X-Tags" header
> > field or by reusing the "Keywords" one. This way, the index could
> > be totally recreated by re-reading the mail directories, and this
> > would also allow to a tools like OfflineIMAP [1] to get the mails into a
> > local maildir, tagging and indexing the mails with the e-mail reader and
> > then syncing back the messages with the "X-Tags" header to the IMAP server.
> > This would allow to use the mail reader from a different computer and still
> > have everything tagged finely.
> 
> It is an interesting idea. But there's also something really comforting
> about the email indexed never modifying the mail files. If you're
> reading the notmuch commit logs closely you'll see that I'm not actually
> careful enough to be trusted with your mail (but I try). So I like that
> I don't even have to trust myself---the worst that happens is that I
> have to recreate my index.

I already made a comment about this in another reply, I could make the
modifications e.g. from a script, so no need to add this in Not Much.
Moreover, I agree that if one of the goals is to never modify mailboxes,
then the current behaviour is just perfect.

> And as Keith mentioned, we've got the "notmuch dump; notmuch restore"
> idea working exactly as it did in sup. (Though I am thinking of also
> adding thread IDs to that now---more on that later.)
> 
> The big annoyance I had with sup index creation, (I ended up having to
> do it more than once too), was that it takes *forever*. Right now,
> notmuch is a little bit faster, but not a lot faster. And I've got some
> ideas to fix that. It would be really nice if index creation were pain
> free. (And maybe it is for some user with small amounts of mail---oh, to
> have only 40000 messages to have to index!).

Wow, that is a lot of mail. Definitely people having as much mail as you
(er... I mean "_not much_ mail as you") would benefit from faster
indexing.

> [*] The problem here is that sup puts serialized ruby data structures
> into the data field of its Xapian documents. So being compatible with
> sup means being able to recreate serialized data structures for a
> particular version of ruby.

Scary...

-- 
Adrian Perez de Castro <aperez@igalia.com>
Igalia - Free Software Engineering

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]