I don't understand the results of this search query, is this a bug?

I don't understand the results of this search query, is this a bug?

[Renaud B.]

[-- Attachment #1.1: Type: text/plain, Size: 1651 bytes --]

Hello,

If I do:

notmuch search --output=files path:rb@kosmopolis.ca/**

I get the following results:

/home/rb/.mail/rb@kosmopolis.ca/Inbox/cur/1715564710.62398_2.tp,U=2:2,S
/home/rb/.mail/
contact@renaudbussieres.com/Inbox/cur/1715541329.42568_1.tp,U=2:2,S
/home/rb/.mail/rb@kosmopolis.ca/Sent/cur/1715564710.62398_6.tp,U=4:2,S
/home/rb/.mail/
contact@renaudbussieres.com/Inbox/cur/1714698543.61892_1.tp,U=1:2,S
/home/rb/.mail/rb@kosmopolis.ca/Inbox/cur/1715564710.62398_1.tp,U=1:2,S
/home/rb/.mail/rb@kosmopolis.ca/Trash/cur/1715564710.62398_7.tp,U=1:2,S
/home/rb/.mail/rb@kosmopolis.ca/Sent/cur/1715564710.62398_5.tp,U=3:2,S
/home/rb/.mail/rb@kosmopolis.ca/Sent/cur/1715564710.62398_4.tp,U=2:2,S
/home/rb/.mail/rb@kosmopolis.ca/Sent/cur/1715564710.62398_3.tp,U=1:2,S

As you can see, two of these files are located under "~/.mail/
contact@renaudbussieres.com/". It shouldn't be the case, because of the "
path:rb@kosmopolis.ca/**" which means explicitly "everything under the
~/.mail/rb@kosmopolis.ca/ directory" (if I understand correctly).

Likewise, if I do:

notmuch search --output=files path:contact@renaudbussieres.com/**

It returns:

/home/rb/.mail/
contact@renaudbussieres.com/Inbox/cur/1715541329.42568_1.tp,U=2:2,S
/home/rb/.mail/rb@kosmopolis.ca/Sent/cur/1715564710.62398_6.tp,U=4:2,S
/home/rb/.mail/
contact@renaudbussieres.com/Inbox/cur/1714698543.61892_1.tp,U=1:2,S
/home/rb/.mail/rb@kosmopolis.ca/Inbox/cur/1715564710.62398_1.tp,U=1:2,S

Again, there are 2 false results.

Do you have any idea what's happening? My goal is simply to get all emails
under each directories in my "~/.mail" folder, one at a time.

Thanks,
Renaud B.

[-- Attachment #1.2: Type: text/html, Size: 3352 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: I don't understand the results of this search query, is this a bug?

[Carl Worth]

Hi Renaud,

I was able to see similar behavior in my own mail store. And I agree
that this behavior is confusing!

The documentation for the --files option of notmuch search documents the
cause (and predicts that this will be confusing):

        Note that each message may have multiple filenames associ‐
        ated  with it. All of them are included in the output (un‐
        less limited with the --duplicate=N option). This  may  be
        particularly  confusing for folder: or path: searches in a
        specified directory, as the messages may  have  duplicates
        in  other directories that are included in the output, al‐
        though these files alone would not match the search.

For my case, I ran a little shell-script one-liner to verify that the
duplicated messages where causing this behavior. This one-liner will
loop over each message-id matched by the path: pattern and will print
all filenames for each:

for id in $(notmuch search --output=messages path:YOUR_PATTERN_HERE); do \
    echo "=== $id ==="; \
    notmuch search --output=files $id; \
    echo; \
done

If you run the above with your own pattern substituted for
YOUR_PATTERN_HERE I imagine you'll see the same thing I did, (that for
any filename paths that don't match the pattern, they have a message-id
which _also_ appears in the database for another path that _does_ match
the pattern).

In general, I'm not a fan of software documenting "this may be
confusing". That suggests the authors of the documentation know that the
software is not behaving as the user intends, so it would be preferable
for the software to behave as intended. That said, I also understand the
implementation details that lead to this behavior. So I wouldn't be
opposed to improving the behavior of notmuch to reduce this behavior,
(but that implementation might not be trivial or even fully feasible).

Hopefully, the duplicate message-id issue explains what you're
seeing. And further, I hope that you just being aware of this issue
gives you a way to cleanly solve your problem, (either by filtering the
output of "notmuch search --output=files" or by instead doing something
along the lines of the "notmuch search --output=messages" approach I
show above).

Let us know if you need anything further. And good luck!

-Carl

On Mon, May 13 2024, Renaud B. wrote:
> Hello,
>
> If I do:
>
> notmuch search --output=files path:rb@kosmopolis.ca/**
>
> I get the following results:
>
> /home/rb/.mail/rb@kosmopolis.ca/Inbox/cur/1715564710.62398_2.tp,U=2:2,S
> /home/rb/.mail/
> contact@renaudbussieres.com/Inbox/cur/1715541329.42568_1.tp,U=2:2,S
> /home/rb/.mail/rb@kosmopolis.ca/Sent/cur/1715564710.62398_6.tp,U=4:2,S
> /home/rb/.mail/
> contact@renaudbussieres.com/Inbox/cur/1714698543.61892_1.tp,U=1:2,S
> /home/rb/.mail/rb@kosmopolis.ca/Inbox/cur/1715564710.62398_1.tp,U=1:2,S
> /home/rb/.mail/rb@kosmopolis.ca/Trash/cur/1715564710.62398_7.tp,U=1:2,S
> /home/rb/.mail/rb@kosmopolis.ca/Sent/cur/1715564710.62398_5.tp,U=3:2,S
> /home/rb/.mail/rb@kosmopolis.ca/Sent/cur/1715564710.62398_4.tp,U=2:2,S
> /home/rb/.mail/rb@kosmopolis.ca/Sent/cur/1715564710.62398_3.tp,U=1:2,S
>
> As you can see, two of these files are located under "~/.mail/
> contact@renaudbussieres.com/". It shouldn't be the case, because of the "
> path:rb@kosmopolis.ca/**" which means explicitly "everything under the
> ~/.mail/rb@kosmopolis.ca/ directory" (if I understand correctly).
>
> Likewise, if I do:
>
> notmuch search --output=files path:contact@renaudbussieres.com/**
>
> It returns:
>
> /home/rb/.mail/
> contact@renaudbussieres.com/Inbox/cur/1715541329.42568_1.tp,U=2:2,S
> /home/rb/.mail/rb@kosmopolis.ca/Sent/cur/1715564710.62398_6.tp,U=4:2,S
> /home/rb/.mail/
> contact@renaudbussieres.com/Inbox/cur/1714698543.61892_1.tp,U=1:2,S
> /home/rb/.mail/rb@kosmopolis.ca/Inbox/cur/1715564710.62398_1.tp,U=1:2,S
>
> Again, there are 2 false results.
>
> Do you have any idea what's happening? My goal is simply to get all emails
> under each directories in my "~/.mail" folder, one at a time.
>
> Thanks,
> Renaud B.
> _______________________________________________
> notmuch mailing list -- notmuch@notmuchmail.org
> To unsubscribe send an email to notmuch-leave@notmuchmail.org\r

Re: I don't understand the results of this search query, is this a bug?

[Michael J Gruber]

Am Mo., 13. Mai 2024 um 23:38 Uhr schrieb Carl Worth <cworth@cworth.org>:
>
> Hi Renaud,
>
> I was able to see similar behavior in my own mail store. And I agree
> that this behavior is confusing!
>
> The documentation for the --files option of notmuch search documents the
> cause (and predicts that this will be confusing):
>
....
> In general, I'm not a fan of software documenting "this may be
> confusing". That suggests the authors of the documentation know that the

:-)

> software is not behaving as the user intends, so it would be preferable
> for the software to behave as intended. That said, I also understand the
> implementation details that lead to this behavior. So I wouldn't be
> opposed to improving the behavior of notmuch to reduce this behavior,
> (but that implementation might not be trivial or even fully feasible).

I wouldn't call it an implementation detail in this case, though,
rather than the guiding principle of notmuch: it is all about
*messages* as identified by a mid.

Consequently, notmuch stores information by message, searches by
message and outputs information by message. This in turn has
consequences, for better or worse, e.g. when different mail files with
the same mid have different (maildir) flags. But without grasping the
main guiding principle you'll get confused sooner or later.

As soon as you introduce "do what I mean" into the CLI design the
outcome depends on the "I" implementing it, who may "mean" very
different things compared to the "I" using the CLI. This creates
confusion which cannot be resolved by pointing out a guiding
principle, but rather "when we do x it is often convenient to imply y
and that's why do z". You can witness that to some extent in git's
CLI.

Also, dwim'ing in the case at hand seems difficult - you'd have to
extract "path:" tokens from a possibly complex query, track logical
operators applying to them and filter the output accordingly. Compare
that to "find -type f dirWhichIWant" which would have solved OP's use
case ...

Cheers
Michael

Re: I don't understand the results of this search query, is this a bug?

[Renaud B.]

[-- Attachment #1.1: Type: text/plain, Size: 4031 bytes --]

Hello,

Thanks a lot for your answers, I'm happy to have received such thoughtful
replies. Like you both pointed out, and from what I get, Notmuch works with
message-ids, and since it may happen that several files in different
folders have the same message-id, it explains what I was seeing.

My goal was to try to sync somehow how emails on my computer are organized
using Notmuch and its tags with folders on the IMAP servers of the many
email accounts I'm syncing to my ~/mail dir, using mbsync. I guess this is
a common endeavour some people want, so for example if you browse one of
your email account using a webmail interface, or an app on your phone, then
there is some continuity between how your emails are sorted on your
computer and how you find them on other interfaces. But I also see that
this isn't that simple to implement. Being centered on message-ids, Notmuch
doesn't seem to easily be able to recognize on which email account an email
is (like in cases where one of my email account send an email to another of
my account, or when some of my accounts receive the same email from
somewhere else... basically, when there are duplicates). I also found out
that moving email files might disrupt mbsync, so it is another issue I
would need to look at.

I also understand the important differences between organizing emails using
tags, and using folders. I just wanted to see if, with some shell
scripting, I could arrange something somewhat similar. But it seems harder
to do than I hoped.

I am still new to Notmuch, I use it on Emacs, maybe my efforts weren't
necessary, and so I will keep using it and see how my needs and preferences
go.

Thanks again for your replies, much appreciated!
Renaud


Le mar. 14 mai 2024, à 04 h 19, Michael J Gruber <
michaeljgruber+grubix+git@gmail.com> a écrit :

> Am Mo., 13. Mai 2024 um 23:38 Uhr schrieb Carl Worth <cworth@cworth.org>:
> >
> > Hi Renaud,
> >
> > I was able to see similar behavior in my own mail store. And I agree
> > that this behavior is confusing!
> >
> > The documentation for the --files option of notmuch search documents the
> > cause (and predicts that this will be confusing):
> >
> ....
> > In general, I'm not a fan of software documenting "this may be
> > confusing". That suggests the authors of the documentation know that the
>
> :-)
>
> > software is not behaving as the user intends, so it would be preferable
> > for the software to behave as intended. That said, I also understand the
> > implementation details that lead to this behavior. So I wouldn't be
> > opposed to improving the behavior of notmuch to reduce this behavior,
> > (but that implementation might not be trivial or even fully feasible).
>
> I wouldn't call it an implementation detail in this case, though,
> rather than the guiding principle of notmuch: it is all about
> *messages* as identified by a mid.
>
> Consequently, notmuch stores information by message, searches by
> message and outputs information by message. This in turn has
> consequences, for better or worse, e.g. when different mail files with
> the same mid have different (maildir) flags. But without grasping the
> main guiding principle you'll get confused sooner or later.
>
> As soon as you introduce "do what I mean" into the CLI design the
> outcome depends on the "I" implementing it, who may "mean" very
> different things compared to the "I" using the CLI. This creates
> confusion which cannot be resolved by pointing out a guiding
> principle, but rather "when we do x it is often convenient to imply y
> and that's why do z". You can witness that to some extent in git's
> CLI.
>
> Also, dwim'ing in the case at hand seems difficult - you'd have to
> extract "path:" tokens from a possibly complex query, track logical
> operators applying to them and filter the output accordingly. Compare
> that to "find -type f dirWhichIWant" which would have solved OP's use
> case ...
>
> Cheers
> Michael
>

[-- Attachment #1.2: Type: text/html, Size: 4801 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]