[PATCH] TODO: date range queries - check

[PATCH] TODO: date range queries - check

[Jani Nikula]

---
 devel/TODO |    9 ---------
 1 file changed, 9 deletions(-)

diff --git a/devel/TODO b/devel/TODO
index eb757af..277a997 100644
--- a/devel/TODO
+++ b/devel/TODO
@@ -163,15 +163,6 @@ vs. tag-when-all-files-flagged (* above)).
 Add an interface to accept a "key" and a byte stream, rather than a
 filename.
 
-Provide a sane syntax for date ranges. First, we don't want to require
-both endpoints to be specified. For example it would be nice to be
-able to say things like "since:2009-01-1" or "until:2009-01-1" and
-have the other endpoint be implicit. Second we'd like to support
-relative specifications of time such as "since:'2 months ago'". To do
-any of this we're probably going to need to break down an write our
-own parser for the query string rather than using Xapian's QueryParser
-class.
-
 Make failure to read a file (such as a permissions problem) a warning
 rather than an error (should be similar to the existing warning for a
 non-mail file).
-- 
1.7.10.4

Re: [PATCH] TODO: date range queries - check

[Jameson Graef Rollins]

[-- Attachment #1: Type: text/plain, Size: 1125 bytes --]

On Thu, Dec 06 2012, Jani Nikula <jani@nikula.org> wrote:
> ---
>  devel/TODO |    9 ---------
>  1 file changed, 9 deletions(-)
>
> diff --git a/devel/TODO b/devel/TODO
> index eb757af..277a997 100644
> --- a/devel/TODO
> +++ b/devel/TODO
> @@ -163,15 +163,6 @@ vs. tag-when-all-files-flagged (* above)).
>  Add an interface to accept a "key" and a byte stream, rather than a
>  filename.
>  
> -Provide a sane syntax for date ranges. First, we don't want to require
> -both endpoints to be specified. For example it would be nice to be
> -able to say things like "since:2009-01-1" or "until:2009-01-1" and
> -have the other endpoint be implicit. Second we'd like to support
> -relative specifications of time such as "since:'2 months ago'". To do
> -any of this we're probably going to need to break down an write our
> -own parser for the query string rather than using Xapian's QueryParser
> -class.
> -

Has this TODO really been resolved?  Do searches like "since:2009-01-1"
or "until:2009-01-1" really now work?  As far as I can tell they don't.
Nor are they documented if they do.

jamie.

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: [PATCH] TODO: date range queries - check

[Jani Nikula]

On Fri, 07 Dec 2012, Jameson Graef Rollins <jrollins@finestructure.net> wrote:
> On Thu, Dec 06 2012, Jani Nikula <jani@nikula.org> wrote:
>> ---
>>  devel/TODO |    9 ---------
>>  1 file changed, 9 deletions(-)
>>
>> diff --git a/devel/TODO b/devel/TODO
>> index eb757af..277a997 100644
>> --- a/devel/TODO
>> +++ b/devel/TODO
>> @@ -163,15 +163,6 @@ vs. tag-when-all-files-flagged (* above)).
>>  Add an interface to accept a "key" and a byte stream, rather than a
>>  filename.
>>  
>> -Provide a sane syntax for date ranges. First, we don't want to require
>> -both endpoints to be specified. For example it would be nice to be
>> -able to say things like "since:2009-01-1" or "until:2009-01-1" and
>> -have the other endpoint be implicit. Second we'd like to support
>> -relative specifications of time such as "since:'2 months ago'". To do
>> -any of this we're probably going to need to break down an write our
>> -own parser for the query string rather than using Xapian's QueryParser
>> -class.
>> -
>
> Has this TODO really been resolved?  Do searches like "since:2009-01-1"
> or "until:2009-01-1" really now work?  As far as I can tell they don't.
> Nor are they documented if they do.

Well, it does say "for example" and "things like" for since: and until:,
and it's not like the TODO is a strict spec anyway. All of the
functionality is there. You can use "date:2009-01-01.." or
"date:..2009-01-01" and the other endpoint is implicit (since Xapian
1.2.1). Relative specification of time is there. Subjectively, with a
sane syntax too. And it's all documented.

BR,
Jani.

Re: [PATCH] TODO: date range queries - check

[David Bremner]

Jameson Graef Rollins <jrollins@finestructure.net> writes:


> Has this TODO really been resolved?  Do searches like "since:2009-01-1"
> or "until:2009-01-1" really now work?  As far as I can tell they don't.
> Nor are they documented if they do.

For specifying one-ended ranges, I find the current syntax OK-ish. It
would be reasonable to formulate a seperate TODO for supporting
things like date:2012-12-07

d

Re: [PATCH] TODO: date range queries - check

[Jameson Graef Rollins]

[-- Attachment #1: Type: text/plain, Size: 921 bytes --]

On Fri, Dec 07 2012, David Bremner <david@tethera.net> wrote:
> Jameson Graef Rollins <jrollins@finestructure.net> writes:
>> Has this TODO really been resolved?  Do searches like "since:2009-01-1"
>> or "until:2009-01-1" really now work?  As far as I can tell they don't.
>> Nor are they documented if they do.
>
> For specifying one-ended ranges, I find the current syntax OK-ish. It
> would be reasonable to formulate a seperate TODO for supporting
> things like date:2012-12-07

Yeah, I think this is reasonable.  I still think it would be nice to
have the requested functionality.  I'm probably having a visceral
reaction to the fact that I initially thought the new parser *did*
support these kinds of searches, and was really confused when I got
errors.  Austin's custom query parser from way back did support this, so
it is possible.  It doesn't hurt anything to continue to list it as a
desired feature.

jamie.

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: [PATCH] TODO: date range queries - check

[Jani Nikula]

On Fri, 07 Dec 2012, Jameson Graef Rollins <jrollins@finestructure.net> wrote:
> On Fri, Dec 07 2012, David Bremner <david@tethera.net> wrote:
>> Jameson Graef Rollins <jrollins@finestructure.net> writes:
>>> Has this TODO really been resolved?  Do searches like "since:2009-01-1"
>>> or "until:2009-01-1" really now work?  As far as I can tell they don't.
>>> Nor are they documented if they do.
>>
>> For specifying one-ended ranges, I find the current syntax OK-ish. It
>> would be reasonable to formulate a seperate TODO for supporting
>> things like date:2012-12-07
>
> Yeah, I think this is reasonable.  I still think it would be nice to
> have the requested functionality.  I'm probably having a visceral
> reaction to the fact that I initially thought the new parser *did*
> support these kinds of searches, and was really confused when I got
> errors.  Austin's custom query parser from way back did support this, so
> it is possible.  It doesn't hurt anything to continue to list it as a
> desired feature.

Fine by me. I was just trying to clean up the file a bit, that's
all. The only downside I can think of is potential new users stumbling
on this and thinking we still don't have date queries. *shrug*.

BR,
Jani.

Re: [PATCH] TODO: date range queries - check

[Jameson Graef Rollins]

[-- Attachment #1: Type: text/plain, Size: 493 bytes --]

On Fri, Dec 07 2012, Jani Nikula <jani@nikula.org> wrote:
> Fine by me. I was just trying to clean up the file a bit, that's
> all. The only downside I can think of is potential new users stumbling
> on this and thinking we still don't have date queries. *shrug*.

That's why it should probably just be modified, instead of removed.

I certainly hope that users aren't looking in the TODO for features that
are already implemented!  If anyone is doing that they've got other
problems.

jamie.

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: [PATCH] TODO: date range queries - check

[David Bremner]

Jameson Graef Rollins <jrollins@finestructure.net> writes:

> On Fri, Dec 07 2012, Jani Nikula <jani@nikula.org> wrote:
>> Fine by me. I was just trying to clean up the file a bit, that's
>> all. The only downside I can think of is potential new users stumbling
>> on this and thinking we still don't have date queries. *shrug*.
>
> That's why it should probably just be modified, instead of removed.
>

Patch? Concrete wording suggestion?

d

Re: [PATCH] TODO: date range queries - check

[Jameson Graef Rollins]

[-- Attachment #1: Type: text/plain, Size: 569 bytes --]

On Sat, Dec 08 2012, David Bremner <david@tethera.net> wrote:
> Patch? Concrete wording suggestion?

How about:

It would be nice to not require both endpoints to be specified in date
searches.  For example it would be nice to be able to say things like
"date:2009-01-1", to specify a search over a particular day, or
"since:'1 month' to specify an open-ended range, and have the other
endpoint be implicit.  To do any of this we're probably going to need to
break down an write our own parser for the query string rather than
using Xapian's QueryParser class.

jamie.

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: [PATCH] TODO: date range queries - check

[Jani Nikula]

On Sat, 08 Dec 2012, Jameson Graef Rollins <jrollins@finestructure.net> wrote:
> On Sat, Dec 08 2012, David Bremner <david@tethera.net> wrote:
>> Patch? Concrete wording suggestion?
>
> How about:
>
> It would be nice to not require both endpoints to be specified in date
> searches.  For example it would be nice to be able to say things like
> "date:2009-01-1", to specify a search over a particular day, or
> "since:'1 month' to specify an open-ended range, and have the other
> endpoint be implicit.  To do any of this we're probably going to need to
> break down an write our own parser for the query string rather than
> using Xapian's QueryParser class.

FWIW, this means, in practice:

Make date:<expr> an alias for date:<expr>..<expr>

Make since:<expr> an alias for date:<expr>..

Make until:<expr> an alias for date:..<expr>

We'll either need to write our own query parser or some new features to
upstream Xapian to do any of these.


BR,
Jani.

Re: [PATCH] TODO: date range queries - check

[Tomi Ollila]

On Sun, Dec 09 2012, Jani Nikula <jani@nikula.org> wrote:

> On Sat, 08 Dec 2012, Jameson Graef Rollins <jrollins@finestructure.net> wrote:
>> On Sat, Dec 08 2012, David Bremner <david@tethera.net> wrote:
>>> Patch? Concrete wording suggestion?
>>
>> How about:
>>
>> It would be nice to not require both endpoints to be specified in date
>> searches.  For example it would be nice to be able to say things like
>> "date:2009-01-1", to specify a search over a particular day, or
>> "since:'1 month' to specify an open-ended range, and have the other
>> endpoint be implicit.  To do any of this we're probably going to need to
>> break down an write our own parser for the query string rather than
>> using Xapian's QueryParser class.
>
> FWIW, this means, in practice:
>
> Make date:<expr> an alias for date:<expr>..<expr>
>
> Make since:<expr> an alias for date:<expr>..
>
> Make until:<expr> an alias for date:..<expr>


I personally don't desire since: and until: prefixes (and as not being
native english speaker maybe more so)

I'd like to also drop the 'date:' part (just for convenience, nothing
against using english word there ;) and use only e.g. 1d..  (and so on).


Actually I am (still) running my hacky wrapper which converts strings
like '1d..' found in command line to '<seconds-since-epoc>..' before passing
the command line to notmuch. I find it very convenient...

e.g. IMHO ..yesterday would be more convenient than since:yesterday

>
> We'll either need to write our own query parser or some new features to
> upstream Xapian to do any of these.

> BR,
> Jani.


FWIW

Tomi

Re: [PATCH] TODO: date range queries - check

[David Bremner]

I'm marking this as moreinfo since the patch as is doesn't seem to
satisfy people. Personally I think the key point is that
e.g. date:2012-12-10 should do something reasonable (presumably return
all messages on that day, pax questions about timezone); whether we need
new prefixes I'm less convinced.  However, as far as I understand it,
any solution to the "single date" problem would make the "syntactic
sugar prefixes" rather easy.

As a meta-point, I don't think we should spend too much time and effort
debating TODO; it isn't like this is some contract about deliverables,
or even a document that most of us look at much.

Re: [PATCH] TODO: date range queries - check

[Daniel Kahn Gillmor]

On 12/07/2012 07:19 AM, David Bremner wrote:
> For specifying one-ended ranges, I find the current syntax OK-ish. It
> would be reasonable to formulate a seperate TODO for supporting
> things like date:2012-12-07

Out of curiosity, how does this syntax interact with timezones?

If i send a mail in the wee hours of the 27th from the east coast of the
US (GMT-0500) and jrollins reads it from the west coast of the US
(GMT-0800) where it is still the 26th, should our notmuch queries behave
differently when searching for this message?

are we tracking the TZ of the Date: header in the database at all?  does
it make sense to take that into account for these queries?

	--dkg, posing questions he has no answers to

Re: [PATCH] TODO: date range queries - check

[Jani Nikula]

On Fri, 07 Dec 2012, Daniel Kahn Gillmor <dkg@fifthhorseman.net> wrote:
> On 12/07/2012 07:19 AM, David Bremner wrote:
>> For specifying one-ended ranges, I find the current syntax OK-ish. It
>> would be reasonable to formulate a seperate TODO for supporting
>> things like date:2012-12-07
>
> Out of curiosity, how does this syntax interact with timezones?

We don't have this particular syntax yet, but it would seem sensible to
expand that to the range expression date:2012-12-07..2012-12-07, which
is currently supported, and means all the messages from the beginning of
specified date to the end.

> If i send a mail in the wee hours of the 27th from the east coast of the
> US (GMT-0500) and jrollins reads it from the west coast of the US
> (GMT-0800) where it is still the 26th, should our notmuch queries behave
> differently when searching for this message?

I'll answer that question with another question: Do you think date
queries should be interpreted according to the timezone of the sender or
the recipient? I'm inclined to believe the local timezone of the system
the query is run on (typically the recipient's timezone) is the only
answer that makes sense, and my ever pessimist/realist gut feeling is
that any attempts to be more clever about the difference in timezones
than the user will be doomed. For you the message date is the 27th, and
for jrollins it's the 26th. But read on.

> are we tracking the TZ of the Date: header in the database at all?  does
> it make sense to take that into account for these queries?

We don't track it or take it into account. The Date: header is converted
to a time_t value using g_mime_utils_header_decode_date(), and stored
into the database as a timestamp. The date: queries are also converted
to timestamps, using local timezone *unless* a timezone is specified in
the query.

And this is the interesting part: You can specify the timezone in the
query. If jrollins talks about a message you sent him on the 26th, you
can, as you know his timezone, query
date:2012-12-26-0800..2012-12-26-0800 to get all messages on that day in
*his* timezone. Including the one you sent him on *your* 27th.

> 	--dkg, posing questions he has no answers to

At the end of the day, I don't have an answer as to whether or indeed
how the sender's timezone could be taken into account in the
queries. But I hope the above clears up how we handle dates and
timezones now, and perhaps leads you to an answer I don't have!


BR,
Jani.