Re: notmuchsync: handling of the deleted tag

Re: notmuchsync: handling of the deleted tag

[Sebastian Spaeth]

On 2010-09-18, Rob Browning wrote:
> I've been trying notmuchsync, and it's quite helpful, but I just ran
> across something that caused me a bit of concern.  I noticed that some
> messages were marked deleted that I'm fairly certain I didn't intend to
> delete.
> 
> My suspicion is that this is because I received two copies of the
> message and I marked one of them for deletion (expiry) in Gnus, but not
> the other one.  Since notmuch doesn't normally distinguish between two
> messages with the same id, I suspected that notmuchsync might not notice
> either.  Is that possible?

Yes, that is certainly possible. notmuchsync checks if a file should be
deleted ("pruning") like this:

1)Run notmuchsync --revsync (or -r) and it will examine your mail
files. If a mail is trashed (expired) according to maildir flags in its
filename, it will add the tag "deleted" to the mail's id in the notmuch
database. If you have multiple copies with the same message that tag is
now associated with the mail id, not the actual file.

2) Run notmuchsync --prune which runs the query: "tag:delete or
tag:deleted or tag:maildir::trashed" by default. 

To which notmuch returns a list of messages and a corresponding filename
for each id. If there are multiple messages with the same id, I think it
will return the first filename that was associated with that mail id, so
it is kind of random which file location gets returned.  The returned
file names are associated with the "deleted" tag - and thus - unlinked.

I am unsure how to handle this in a better way. What should notmuchsync
--prune do if it finds multiple mail files that are associated with a
"deleted" tag? And what should --revsync do when it finds a mail file
that is marked as expired. As long as notmuch only offers one data entry
per mail id, there will remain some ambiguity as to how to handle these.


If notmuch gave me at least all filenames that are associated with a
mail id, I could introduce a command line option "--prune --safe" which would
Sebastian

P.S. CC'ing the notmuch list as this is potentially an issue that
affects people.

Re: notmuchsync: handling of the deleted tag

[Rob Browning]

Sebastian Spaeth <Sebastian@SSpaeth.de> writes:

> I am unsure how to handle this in a better way. What should notmuchsync
> --prune do if it finds multiple mail files that are associated with a
> "deleted" tag?

Conceptually what I'd like for it to do, is reference count -- only mark
the message deleted if every occurrence (across all maildirs) is marked
trashed (T).

Though even there I can imagine corner cases: imagine that notmuch
doesn't initially see all your maildirs -- perhaps because you're using
a folder filter in offlineimap, and so there are untrashed copies in the
maildirs it hasn't seen yet.  But reference counting of everything you
can see is probably still better than what we have now, if it's
feasible.

> And what should --revsync do when it finds a mail file that is marked
> as expired.

Actually revsync is all I use so far, so that's what I'm talking about
above.

> If notmuch gave me at least all filenames that are associated with a
> mail id, I could introduce a command line option "--prune --safe" which would
> Sebastian

Looks like you got cut off there.

Thanks
-- 
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

Re: notmuchsync: handling of the deleted tag

[Sebastian Spaeth]

[-- Attachment #1: Type: text/plain, Size: 2432 bytes --]

On 2010-09-21, Rob Browning wrote:
> Conceptually what I'd like for it to do, is reference count -- only mark
> the message deleted if every occurrence (across all maildirs) is marked
> trashed (T).

Right, but that is trickier than might appear at first sight.

I parse those file names which notmuch is giving me (by default for
mails from the last 30 days, as reparsing ALL your mails every time
would be horribly expensive and often unneeded). 

notmuch is only able to give me one file name (+path) per mail id, so
that is what I examine. If that is the one copy that has the mail dir
flag "expired/trashed", I tag the message as "deleted". 

There is little else that notmuchsync can do here. I ask for new
messages and their file path and if they are expired mark it as such. I
have no way of finding out if there are other mails with the same mail
id from notmuch (unless I am very much mistaken). 

Doing reference counting would require me/notmuchsync to parse ALL your
mails by itself and finding out the often horribly mail id from the mail
headers myself... something that notmuchsync does not want to get into.

See the problem? I could do reference counting if notmuch were able to
tell me how many file names/paths are associated with a mail id.


> Though even there I can imagine corner cases: imagine that notmuch
> doesn't initially see all your maildirs -- perhaps because you're using
> a folder filter in offlineimap, and so there are untrashed copies in the
> maildirs it hasn't seen yet.

Right that would be a problem. But I cannot do reference counting unless
notmuch can give me the number of copies it knows about for a given mail
id (and internally it does know all associated file paths, so it would
be a simple API extension a la, "get_all_message_file_paths" or similar,
or a get_number_of_mail_files(mailid) to start with.

> > And what should --revsync do when it finds a mail file that is marked
> > as expired.

What should notmuch do BTW if there are 2 copies and 1 is expired and 1 not?
Mark as "deleted" or not?

> Looks like you got cut off there.

Right, it was 5pm and I left the computer :). I had intented to rant
about the deficiencies of the notmuch 1 document per mail id approach
here, but I don't see a better approach. All that would be useful from
the notmuch side is to get all associated filenames with a mail id.

sebastian

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: notmuchsync: handling of the deleted tag

[Mike Kelly]

On Tue, 21 Sep 2010 11:44:29 +0200
Sebastian Spaeth <Sebastian@SSpaeth.de> wrote:

> Right that would be a problem. But I cannot do reference counting
> unless notmuch can give me the number of copies it knows about for a
> given mail id (and internally it does know all associated file paths,
> so it would be a simple API extension a la,
> "get_all_message_file_paths" or similar, or a
> get_number_of_mail_files(mailid) to start with.

I agree, and such an API would be useful for other things too. For
example, I could probably implement a rudamentary gmail syncer with
that. I may just try to make a patch to do this soon if nobody beats me
to it.

-- 
Mike Kelly

Re: notmuchsync: handling of the deleted tag

[Jameson Rollins]

[-- Attachment #1: Type: text/plain, Size: 420 bytes --]

On Tue, 21 Sep 2010 09:15:44 -0400, Mike Kelly <pioto@pioto.org> wrote:
> I agree, and such an API would be useful for other things too. For
> example, I could probably implement a rudamentary gmail syncer with
> that. I may just try to make a patch to do this soon if nobody beats me
> to it.

I think this is related to the requests that I've made to have access to
all versions of duplicate messages as well.

jamie.

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: notmuchsync: handling of the deleted tag

[Sebastian Spaeth]

[-- Attachment #1: Type: text/plain, Size: 267 bytes --]

On 2010-09-21, Jameson Rollins wrote:
> I think this is related to the requests that I've made to have access to
> all versions of duplicate messages as well.

Absolutely yes. This would allow solving your problem with duplicate
messages as well, I think.

Sebastian

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: notmuchsync: handling of the deleted tag

[Rob Browning]

Sebastian Spaeth <Sebastian@SSpaeth.de> writes:

> On 2010-09-21, Rob Browning wrote:
>> Conceptually what I'd like for it to do, is reference count -- only mark
>> the message deleted if every occurrence (across all maildirs) is marked
>> trashed (T).
>
> Right, but that is trickier than might appear at first sight.

In general, I think that until/unless notmuchsync can be more assured of
doing the right thing, and in particular, if the deleted tag is likely
to become official, notmuchsync should default to not setting it.

...or at least, I'd prefer that.  Then I can add --tag-deleted if/when I
want to.  Of course defaulting to --tag-deleted would also be OK, as
long as there's a --no-tag-deleted.

Thanks
-- 
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

Re: notmuchsync: handling of the deleted tag

[Sebastian Spaeth]

[-- Attachment #1: Type: text/plain, Size: 1602 bytes --]

On 2010-09-22, Rob Browning wrote:
> In general, I think that until/unless notmuchsync can be more assured of
> doing the right thing, and in particular, if the deleted tag is likely
> to become official, notmuchsync should default to not setting it.
> 
> ...or at least, I'd prefer that.  Then I can add --tag-deleted if/when I
> want to.  Of course defaulting to --tag-deleted would also be OK, as
> long as there's a --no-tag-deleted.

In my workflow it does the right thing as I only "expire" a message if I
am happy that it actually will be deleted. The situation is a bit less
scary that it seemed to me: if a notmuch --revsync examines a mail file,
it is the exact copy that notmuch will point to, and it will mark that
as "deleted" (and eventually "--prune" can delete that very copy). A new
run by notmuchsync --revsync will then be told to look at the second
copy if the first is gone and it will discover that it does not have the
maildir flag "expired" and thus remove the "deleted" tag in the notmuch
db. As such, the tag should reflect the actual status of the mail file
that notmuch points to.

But I can imagine circumstances in which there would be an erroneous tag
be set and I don't want notmuchsync to accidently delete anyones emails,
so I'll disable the "deleted" tag synchronization by default and only
make it an option (that I will use). Solving this in a proper way will
require support from the notmuch API though.

Sebastian
P.S. cworth made clear that he preferred if there were no "official" tags
or some tags that are more queal than others.

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: notmuchsync: handling of the deleted tag

[Sebastian Spaeth]

[-- Attachment #1: Type: text/plain, Size: 421 bytes --]

On 2010-09-22, Rob Browning wrote:
> In general, I think that until/unless notmuchsync can be more assured of
> doing the right thing, and in particular, if the deleted tag is likely
> to become official, notmuchsync should default to not setting it.

BTW, I consider --revsync as a kludge anyway that belongs into notmuch
proper. There are patches to achieve that kind of synchronization on the
mailing list.

Sebastian

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

notmuchsync default behavior change (was: notmuchsync: handling of the deleted tag)

[Sebastian Spaeth]

[-- Attachment #1: Type: text/plain, Size: 1838 bytes --]

On 2010-09-22, Rob Browning wrote:
> In general, I think that until/unless notmuchsync can be more assured of
> doing the right thing, and in particular, if the deleted tag is likely
> to become official, notmuchsync should default to not setting it.
> ...or at least, I'd prefer that.  Then I can add --tag-deleted if/when I
> want to.  Of course defaulting to --tag-deleted would also be OK, as
> long as there's a --no-tag-deleted.

notmuchsync does not want to delete anyones' e-mails by accident and
although I believe that syncing the "deleted" tag should usually be
safe, I can think of situations with multiple copies of a mail where
things could go wrong. I therefore disabled syncing of the maildir flag
"trashed" and the notmuch tag "deleted" by default. It has to be
explicitly be enabled with the new --sync-deleted option.

Sebastian

------------------------------------------------------------------
For reference, this could be a szenario where things turn out bad:

- Send a mail with FCC to your Sent folder and BCC yourself. You have
then 2 mail copies with the same mail id (1 notmuch entry)

- "Expire" your BCC copy via some mail/webclient, it gets maildir flag
"T"

- notmuch --revsync will examine new mails and in case the BCC copy is
what notmuch points to, notmuchsync will then add the notmuch tag
"deleted" to that mail id.

- notmuchsync --prune deletes the BCC mail copy but does not remove the
  "deleted" tag in the notmuch db (it probably should for safety
  reasons).

- 'notmuch new' will now recognize the BCC copy as removed and point to the
  "Sent/FCC" copy as the current file, but it still carries the "deleted"
  tag for the mail.

- notmuchsync --prune will happily delete your Sent/FCC copy of the
  mail, or notmuchsync --sync will mark your Sent/FCC copy with the
  "expired" maildir flag.

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: notmuchsync: handling of the deleted tag

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 1688 bytes --]

On Mon, 20 Sep 2010 21:25:13 -0500, Rob Browning <rlb@defaultvalue.org> wrote:
> Sebastian Spaeth <Sebastian@SSpaeth.de> writes:
> Conceptually what I'd like for it to do, is reference count -- only mark
> the message deleted if every occurrence (across all maildirs) is marked
> trashed (T).

[I'm going back through this old notmuchsync thread after having worked
to merge and improve some related synchronization featured.]

The above idea sounds quite interesting and would be possible in the
current approach. The only thing I don't like is that we would have some
very detailed knowledge about the semantics of "deleted" hard-coded deep
in the library.

I already don't like the fact that the library has a hard-coded list
with a handful of mappings between flags and tags.

So, what we probably need here is for the user to be able to configure
the mapping and in a fairly sophisticated way:

	'R' on _any_ filename  -> "replied" tag gets added
	'T' on _all_ filenames -> "deleted" tag gets added
	'S' on _any_ filename  -> "unread" tag gets removed

That would be expressible with just a couple of pieces of punctuation:

	R=replied;T*=deleted;S=~unread

So maybe something like that?

> > If notmuch gave me at least all filenames that are associated with a
> > mail id, I could introduce a command line option "--prune --safe"
> > which would

This feature is at least implemented in libnotmuch at least.

Hopefully things just keep getting better and better.

I am looking forward to some feedback from users of notmuchsync who try
out the new maildir.synchronize_flags support in notmuch itself.

-Carl

-- 
carl.d.worth@intel.com

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: notmuchsync: handling of the deleted tag

[Sebastian Spaeth]

[-- Attachment #1: Type: text/plain, Size: 1033 bytes --]

On Thu, 11 Nov 2010 17:27:34 -0800, Carl Worth <cworth@cworth.org> wrote:
> So, what we probably need here is for the user to be able to configure
> the mapping and in a fairly sophisticated way:
> 
> 	'R' on _any_ filename  -> "replied" tag gets added
> 	'T' on _all_ filenames -> "deleted" tag gets added
> 	'S' on _any_ filename  -> "unread" tag gets removed
> So maybe something like that?

Maybe, but that sounds like a horribly complex configuration, in which
the user has to really think through what he wants (and can still make
blunders). :)
 
> > > If notmuch gave me at least all filenames that are associated with a
> > > mail id, I could introduce a command line option "--prune --safe"
> > > which would

Right, you pushed the ball in my court. The only problem is that -- with
the arrival of maildir sync -- I lost my motivation to work on
notmuchsync. Seriously, what does notmuchsync still provide that notmuch
cannot do? I wonder if I shouldn't stick a "deprecated" warning on it.

Sebastian

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: notmuchsync: handling of the deleted tag

[Dirk Hohndel]

On Fri, 12 Nov 2010 08:30:36 +0100, Sebastian Spaeth <Sebastian@SSpaeth.de> wrote:
> On Thu, 11 Nov 2010 17:27:34 -0800, Carl Worth <cworth@cworth.org> wrote:
> > So, what we probably need here is for the user to be able to configure
> > the mapping and in a fairly sophisticated way:
> > 
> > 	'R' on _any_ filename  -> "replied" tag gets added
> > 	'T' on _all_ filenames -> "deleted" tag gets added
> > 	'S' on _any_ filename  -> "unread" tag gets removed
> > So maybe something like that?
> 
> Maybe, but that sounds like a horribly complex configuration, in which
> the user has to really think through what he wants (and can still make
> blunders). :)
>  
> > > > If notmuch gave me at least all filenames that are associated with a
> > > > mail id, I could introduce a command line option "--prune --safe"
> > > > which would
> 
> Right, you pushed the ball in my court. The only problem is that -- with
> the arrival of maildir sync -- I lost my motivation to work on
> notmuchsync. Seriously, what does notmuchsync still provide that notmuch
> cannot do? I wonder if I shouldn't stick a "deprecated" warning on it.

Please don't! I use it all the time:

I use it to archive mail that I don't want on an imap server any more,
but in a local maildir. Can't do that with notmuch.

I use it to prune mail that has the deleted tag. Again not something
that notmuch supports.

So please don't stop supporting your very important (to me) tool!

/D

Re: notmuchsync: handling of the deleted tag

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 3225 bytes --]

On Fri, 12 Nov 2010 13:04:16 -0800, Dirk Hohndel <hohndel@infradead.org> wrote:
> > notmuchsync. Seriously, what does notmuchsync still provide that notmuch
> > cannot do? I wonder if I shouldn't stick a "deprecated" warning on it.
> 
> Please don't! I use it all the time:

Excellent. I love getting more feedback from users so that we can
understand things better.

> I use it to archive mail that I don't want on an imap server any more,
> but in a local maildir. Can't do that with notmuch.

Can you describe this use case in a bit more detail? I'd like to
understand what needs to be done to support this.

> I use it to prune mail that has the deleted tag. Again not something
> that notmuch supports.

This one is a simple one-liner as of notmuch 0.4:

    notmuch search --output=files tag:deleted -print0 | xargs -0 rm

And that's something we could even add to a keybinding in the emacs
interface[*].

Meanwhile, I know people have asked for keybindings to even just add the
"deleted" tag before. What I still want for that is good support for
being able to configure tags that cause mails to be automatically
omitted from search results---unless the user is explicitly searching
for mails with that tag. So for example, I would add "deleted" and
"spam" to such a list and never see mails with that tag unless I go
looking for a mis-tagged email with a search like "from:someone and
tag:spam".

I think the above should be fairly easy. One way to do it is to just
implicitly add "and not (tag:deleted or tag:spam)" to all searches. And
then simply omit the relevant portion of this if the original search
terms happen to mention tag:deleted or tag:spam. This is what sup does,
for example.

-Carl

[*] Personally, I don't want a keybinding for actual deletion of mail
files. As long as nothing appears in searches, then I'd prefer the
tagged messages not be deleted right away. Instead, I'd like a cron job
to delete appropriately tagged messages after some safety interval (a
month, say). For that, I'd like better support for specifying message
dates of course, (that's been on my todo list for some time). But
really, I care about the date of the tag, not the date of the
message. We don't save information like that, but the cron script could
do something cheesy like implement counters in tags:

  # XXX: Entirely untested!

  # Days after a mail is tag:deleted when the mail will be deleted
  EXPIRATION=30
  for days in $(seq 0 $((EXPIRATION - 1))); do
    notmuch tag +expire-$((days)) -expire-$((days + 1)) tag:expire-$(days + 1))
  done

  notmuch search --output=files tag:expire-0 and tag:deleted -print0 | xargs -0 rm
  notmuch tag -expire-0 tag:expire-0

  notmuch tag +expire-$EXPIRATION +expiring tag:deleted and not tag:expiring
  notmuch tag -expiring not tag:deleted

OK. Looking at that---that's *really* cheesy. Perhaps we should come up
with a sane way to do this. Hooks on the addition or deletion of the
"deleted" tag that set/cleared a date value in the message document in
the database might be much cleaner.

But the above might actually work for now. Anyone brave enough to test
it?

-- 
carl.d.worth@intel.com

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: notmuchsync: handling of the deleted tag

[Sebastian Spaeth]

[-- Attachment #1: Type: text/plain, Size: 426 bytes --]

On Fri, 12 Nov 2010 16:52:51 -0800, Carl Worth wrote:
> On Fri, 12 Nov 2010 13:04:16 -0800, Dirk Hohndel <hohndel@infradead.org> wrote:
> > Please don't! I use it all the time:

Thanks :).
> This one is a simple one-liner as of notmuch 0.4:
> 
>     notmuch search --output=files tag:deleted -print0 | xargs -0 rm

Uhh, that is a neat line, and a neat trick. (is it on the emacstips page
already? ;-))

Sebastian

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: notmuchsync: handling of the deleted tag

[Jameson Rollins]

[-- Attachment #1: Type: text/plain, Size: 417 bytes --]

On Sun, 14 Nov 2010 23:21:41 +0100, Sebastian Spaeth <Sebastian@SSpaeth.de> wrote:
> > This one is a simple one-liner as of notmuch 0.4:
> > 
> >     notmuch search --output=files tag:deleted -print0 | xargs -0 rm
> 
> Uhh, that is a neat line, and a neat trick. (is it on the emacstips page
> already? ;-))

It's not an emacs-specific tip!  I guess the web page needs a "general
tips" section...

jamie.

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: notmuchsync: handling of the deleted tag

[Jeff Richards]

On Sun, 14 Nov 2010 23:21:41 +0100, Sebastian Spaeth <Sebastian@SSpaeth.de> wrote:
> This one is a simple one-liner as of notmuch 0.4:
> > 
> >     notmuch search --output=files tag:deleted -print0 | xargs -0 rm
> 

Having just switched from notmuchsync the synchronizing power of 0.5 and
being as how I'm missing the pruning feature I gave this a try and found
that I ended up with rm complaining that the file name length being too
long or simply file not found.  Seems like the  
xargs -0 rm 
portion was choking as all the file names were being contcatenated.
Perhaps the null character line ending is missing?  At this point I
haven't investigated further.    

So for anyone else who is stuck adjusting the one liner like 
notmuch search --output=files tag:deleted -print0 | xargs -d '\n' rm

worked for me.

PS Thanks to Sebastian and Carl and everyone else who has built notmuch
I'm really enjoying using it! :)

Jeff


> Uhh, that is a neat line, and a neat trick. (is it on the emacstips page
> already? ;-))


> 
> Sebastian
Non-text part: application/pgp-signature
> _______________________________________________
> notmuch mailing list
> notmuch@notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch

-- 
Jeff Richards
jrichards@revenuewire.com

Re: notmuchsync: handling of the deleted tag

[Jameson Rollins]

[-- Attachment #1: Type: text/plain, Size: 1089 bytes --]

On Sun, 14 Nov 2010 17:23:57 -0800, Jeff Richards <jrichards@revenuewire.com> wrote:
> Having just switched from notmuchsync the synchronizing power of 0.5 and
> being as how I'm missing the pruning feature I gave this a try and found
> that I ended up with rm complaining that the file name length being too
> long or simply file not found.  Seems like the  
> xargs -0 rm 
> portion was choking as all the file names were being contcatenated.
> Perhaps the null character line ending is missing?  At this point I
> haven't investigated further.    

I think the 'print0' might have been a mistake on Carl's part.  I don't
think that option is actually supported by notmuch.  In which case xargs
-0 isn't going to work as expected because there are no null characters
in the input stream to use as delimiters.

> So for anyone else who is stuck adjusting the one liner like 
> notmuch search --output=files tag:deleted -print0 | xargs -d '\n' rm

This is working because the input stream is newline delimited.  So I
think the -print0 is confusing the issue.

jamie.

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: notmuchsync: handling of the deleted tag

[servilio]

On 14 November 2010 22:03, Jameson Rollins <jrollins@finestructure.net> wrote:
[...]
> I think the 'print0' might have been a mistake on Carl's part.  I don't
> think that option is actually supported by notmuch.  In which case xargs
> -0 isn't going to work as expected because there are no null characters
> in the input stream to use as delimiters.
>
>> So for anyone else who is stuck adjusting the one liner like
>> notmuch search --output=files tag:deleted -print0 | xargs -d '\n' rm
>
> This is working because the input stream is newline delimited.  So I
> think the -print0 is confusing the issue.

And "xargs -L1 ..." will solve the issue in a nicer way. So the above would be:

notmuch search --output=files tag:deleted | xargs -L1 rm

Servilio

Re: notmuchsync: handling of the deleted tag

[Daniel Kahn Gillmor]

[-- Attachment #1: Type: text/plain, Size: 436 bytes --]

On 11/15/2010 10:24 AM, servilio wrote:
> And "xargs -L1 ..." will solve the issue in a nicer way. So the above would be:
> 
> notmuch search --output=files tag:deleted | xargs -L1 rm

Assuming none of your mail filenames have newlines or trailing
whitespace, that is.

I'd like to make that kind of assumption, myself.  But I've certainly
seen files that violate it (not from any maildir implementation, though).

	--dkg


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 900 bytes --]

Re: notmuchsync: handling of the deleted tag

[Jameson Rollins]

[-- Attachment #1: Type: text/plain, Size: 482 bytes --]

On Fri, 12 Nov 2010 16:52:51 -0800, Carl Worth <cworth@cworth.org> wrote:
> This one is a simple one-liner as of notmuch 0.4:
> 
>     notmuch search --output=files tag:deleted -print0 | xargs -0 rm

Is -print0 actually a command that notmuch accepts?  It's not documented
as far as I can tell.  And it doesn't seem to affect the output of the
command, i.e. whether or not it's used the output is still newline
delimited.  I'm not seeing it in the source either.

jamie.

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: notmuchsync: handling of the deleted tag

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 1265 bytes --]

On Sun, 14 Nov 2010 22:01:09 -0500, Jameson Rollins <jrollins@finestructure.net> wrote:
> On Fri, 12 Nov 2010 16:52:51 -0800, Carl Worth <cworth@cworth.org> wrote:
> > This one is a simple one-liner as of notmuch 0.4:
> > 
> >     notmuch search --output=files tag:deleted -print0 | xargs -0 rm
> 
> Is -print0 actually a command that notmuch accepts?  It's not documented
> as far as I can tell.  And it doesn't seem to affect the output of the
> command, i.e. whether or not it's used the output is still newline
> delimited.  I'm not seeing it in the source either.

No, it's not something implemented. I'm surprised I spouted such
nonsense above. I think what happened is that I first typed a command
line which would actually work, (for filenames without embedded newline
characters):

        notmuch search --output=files tag:deleted | xargs rm

And then a habit kicked in from typing many find/xargs commands and I
retrofitted the command with -print0 and -0 as I originally sent it out.

Should we perhaps add a -0 option so that one could actually reliably
handle filenames with newlines in a case like this? Or can we just
assert that such filenames are insane and *should* cause problems?

-Carl

-- 
carl.d.worth@intel.com

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: notmuchsync: handling of the deleted tag

[Daniel Kahn Gillmor]

[-- Attachment #1: Type: text/plain, Size: 723 bytes --]

On 11/15/2010 02:58 PM, Carl Worth wrote:
> Should we perhaps add a -0 option so that one could actually reliably
> handle filenames with newlines in a case like this? Or can we just
> assert that such filenames are insane and *should* cause problems?

it's not just newlines.  Without using the -L1 flag to xargs (suggested
earlier on this list), it's actually any filename with whitespace.

with -L1, newlines and trailing whitespace are still problematic.

newlines and trailing whitespace are unarguably insane in file names.
But they happen in the wild, and it would be a shame if notmuch broke
with them.

i think offering -print0 or -0 to provide null byte delimiters is a good
proposal.

	--dkg


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 900 bytes --]

Re: notmuchsync: handling of the deleted tag

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 2085 bytes --]

On Fri, 12 Nov 2010 08:30:36 +0100, Sebastian Spaeth <Sebastian@SSpaeth.de> wrote:
> On Thu, 11 Nov 2010 17:27:34 -0800, Carl Worth <cworth@cworth.org> wrote:
> > 	'R' on _any_ filename  -> "replied" tag gets added
> > 	'T' on _all_ filenames -> "deleted" tag gets added
> > 	'S' on _any_ filename  -> "unread" tag gets removed
> > So maybe something like that?
> 
> Maybe, but that sounds like a horribly complex configuration, in which
> the user has to really think through what he wants (and can still make
> blunders). :)

Yes. But a common user configurations could be as simple as:

	R:replied; t:todo; d:done;

or so. And that would be hugely powerful because it would allow for
remote synchronization of tags with tools that exist *today* (and
perhaps[*] even through imap servers where the user cannot install
custom software).

[*] This is a point I'm not clear on. Would a tool like offlineimap be
able to push custom filenames through to the remote end with standard
imap servers?

But we could also have a well-documented default configuration
that happens to include two of the more complex cases:

	R:replied; T* : deleted; S : ~unread;

So questions of syntax aside, I'm not convinced that this idea is
insane. And if it covers all of the synchronization needs that people
really have.

> Right, you pushed the ball in my court. The only problem is that -- with
> the arrival of maildir sync -- I lost my motivation to work on
> notmuchsync.

Hehe. Oh well, someone else will have to go play with
notmuch_message_get_filenames. I think the people interested in writing
a "gmail importer" want this as well.

> Seriously, what does notmuchsync still provide that notmuch
> cannot do? I wonder if I shouldn't stick a "deprecated" warning on it.

That's actually a question I wanted to ask you. If notmuch 0.5 actually
does everything that notmuchsync did, then that's great news. If there
is any missing functionality, then let's get that into notmuch.

Thanks for the discussion.

-Carl

-- 
carl.d.worth@intel.com

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: notmuchsync: handling of the deleted tag

[Sebastian Spaeth]

[-- Attachment #1: Type: text/plain, Size: 1556 bytes --]

On Fri, 12 Nov 2010 13:16:53 -0800, Carl Worth wrote:
> [*] This is a point I'm not clear on. Would a tool like offlineimap be
> able to push custom filenames through to the remote end with standard
> imap servers?

Not sure. If it worked this would be really nice. But as most IMAP
server support arbitrary flags to be added to a message, I would still
think the best is to code a tool that talks IMAP and synchronizes
notmuch tags with the IMAP server. I've never done it before and
python's IMAP support is pretty crappy or I would alredy have given it a
go :).

> So questions of syntax aside, I'm not convinced that this idea is
> insane. And if it covers all of the synchronization needs that people
> really have.

Yes, it would cover a lot of my needs too. Altough I am still aiming at
propagating my notmuch flags via the IMAP server and back :).

> That's actually a question I wanted to ask you. If notmuch 0.5 actually
> does everything that notmuchsync did, then that's great news. If there
> is any missing functionality, then let's get that into notmuch.

Pruning used to be done nicely with notmuchsync, but you have given a
nice example how that could be done with notmuch now.

The one thing I can see now that notmuchsync still does is moving
maessages to a folder when they match a tag. But that would be an
equally short shell script that notmuch can do now. So, while I might
still fix bugs, I do not think that notmuchsync will evolve a lot
further (until I make it speak IMAP which I don't see coming in the near
future)

Sebastian

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: notmuchsync: handling of the deleted tag

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 944 bytes --]

On Sun, 14 Nov 2010 23:30:13 +0100, Sebastian Spaeth <Sebastian@SSpaeth.de> wrote:
> Not sure. If it worked this would be really nice. But as most IMAP
> server support arbitrary flags to be added to a message, I would still
> think the best is to code a tool that talks IMAP and synchronizes
> notmuch tags with the IMAP server. I've never done it before and
> python's IMAP support is pretty crappy or I would alredy have given it a
> go :).

Right. A tool that did synchronize notmuch tags by talking to an IMAP
server would be something of interest to many people. But, from all the
rumors I've heard, writing code to implement IMAP isn't much fun.

So we'll have to wait for someone brave, I guess.

In the meantime, I'll be happy enough to synchronize my mail with rsync,
so I may very well implement the support for custom mapping of
maildir-style filename flags to notmuch tags.

-Carl

-- 
carl.d.worth@intel.com

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]