how do the different frontends deal with displaying large queries?

how do the different frontends deal with displaying large queries?

[Gaute Hope]

Hi,

I am wondering how the different frontends deal with displaying large
queries? Do you just display everything at the time? Or do you display a
limited number and then show more on-demand?

In the on-demand case; how do you get to the same position? Keeping a
query object alive for long might make it invalid when the index is
changed.

Showing a limited number and restarting the query from the same position
might miss threads if something in the first batch has been removed.

Regards, Gaute

Re: how do the different frontends deal with displaying large queries?

[David Bremner]

Gaute Hope <eg@gaute.vetsj.com> writes:

> Hi,
>
> I am wondering how the different frontends deal with displaying large
> queries? Do you just display everything at the time? Or do you display a
> limited number and then show more on-demand?
>

I only know the emacs front end, and I suspect the answer won't help you
much. We start filling an emacs buffer asynchronously with the results,
and display the top of the buffer with the first results.  Results are
appended to the end of the buffer, typically much faster than the user
can scroll.

d

Re: how do the different frontends deal with displaying large queries?

[Gaute Hope]

[-- Attachment #1: Type: text/plain, Size: 759 bytes --]

That makes sense of course, that could work.

Thanks, Gaute

Den onsdag 24. februar 2016 skrev David Bremner <david@tethera.net>
følgende:

> Gaute Hope <eg@gaute.vetsj.com <javascript:;>> writes:
>
> > Hi,
> >
> > I am wondering how the different frontends deal with displaying large
> > queries? Do you just display everything at the time? Or do you display a
> > limited number and then show more on-demand?
> >
>
> I only know the emacs front end, and I suspect the answer won't help you
> much. We start filling an emacs buffer asynchronously with the results,
> and display the top of the buffer with the first results.  Results are
> appended to the end of the buffer, typically much faster than the user
> can scroll.
>
> d
>

[-- Attachment #2: Type: text/html, Size: 1103 bytes --]

Re: how do the different frontends deal with displaying large queries?

[Gaute Hope]

David Bremner writes on February 24, 2016 18:48:
> Gaute Hope <eg@gaute.vetsj.com> writes:
>
>> Hi,
>>
>> I am wondering how the different frontends deal with displaying large
>> queries? Do you just display everything at the time? Or do you display a
>> limited number and then show more on-demand?
>>
>
> I only know the emacs front end, and I suspect the answer won't help you
> much. We start filling an emacs buffer asynchronously with the results,
> and display the top of the buffer with the first results.  Results are
> appended to the end of the buffer, typically much faster than the user
> can scroll.

Hi,

I am testing out this way of loading threads, however, I run into an
issue which I have been bugging you with before:

While loading the threads, if I make enough modifications to the
database while the query is still loading (specifically 1: removing the
unread tag from a thread, and 2: adding the unread tag to the same thread)
apparently the query is invalidated in some way, and I get a hard crash
in `notmuch_tags_get ()`.

The thread in question has alreaday been loaded and is displayed.

Do you deal with this issue in some way?

[ code in question: https://github.com/gauteh/astroid/pull/95 ]

Regards, Gaute

Re: how do the different frontends deal with displaying large queries?

[David Bremner]

Gaute Hope <eg@gaute.vetsj.com> writes:

>
> While loading the threads, if I make enough modifications to the
> database while the query is still loading (specifically 1: removing the
> unread tag from a thread, and 2: adding the unread tag to the same thread)
> apparently the query is invalidated in some way, and I get a hard crash
> in `notmuch_tags_get ()`.
>
> The thread in question has alreaday been loaded and is displayed.
>
> Do you deal with this issue in some way?
>

I think we "deal with" it by not leaving the database open very
long. The query is dumped as s-expr by a seperate process. Or perhaps we
just aren't doing the same level of concurrent operations, I don't know.

d

Re: how do the different frontends deal with displaying large queries?

[Gaute Hope]

David Bremner writes on February 27, 2016 13:50:
> Gaute Hope <eg@gaute.vetsj.com> writes:
> 
>>
>> While loading the threads, if I make enough modifications to the
>> database while the query is still loading (specifically 1: removing the
>> unread tag from a thread, and 2: adding the unread tag to the same thread)
>> apparently the query is invalidated in some way, and I get a hard crash
>> in `notmuch_tags_get ()`.
>>
>> The thread in question has alreaday been loaded and is displayed.
>>
>> Do you deal with this issue in some way?
>>
> 
> I think we "deal with" it by not leaving the database open very
> long. The query is dumped as s-expr by a seperate process. Or perhaps we
> just aren't doing the same level of concurrent operations, I don't know.

Neither do I, I load the threads in the background. But for a long query
that takes a while. The database is only kept open for the duration it
takes to load the threads in the query.

How long is the db open? Would you notice if it crashed? I'm guessing
the abort is triggered by an Xapian::ModifiedDatabase somewhere.

I am only able to consistently reproduce the error if I change the tags
of a thread not yet loaded, a (somewhat messy) test case is available
here:

https://github.com/gauteh/astroid/blob/ti-stateless-query/test/test_notmuch_standalone.cc

If nothing else, it demonstrates the difficulty of handling these
exceptions in functions that don't yet catch them, since they could
happen along many of the internal steps.

Regards, Gaute