bug#37871: 27.0.50; Merge feature/gnus-select2

bug#37871: 27.0.50; Merge feature/gnus-select2

[Eric Abrahamsen]

Hi all, Andy Cohen's nnselect Gnus backend looks ready to merge -- he
wrote to emacs.devel but it occurs to me it might be better to start
here. His original message is here, and I've pasted the full text below
the link:

https://lists.gnu.org/archive/html/emacs-devel/2019-10/msg00689.html

------

After many months I have updated the gnus-select2 feature branch. This
incorporates a number of bug fixes as well as updating to the latest
master. I have been using this version (or something close to it) for
nearly a year with no problems. For those who use gnus please check it
out and if no obvious bugs appear I will figure out how to push this
into master.

This branch introduces a new "virtual" backend, nnselect, that allows an
arbitrary list of messages to be treated as a normal gnus group. The
messages may come from multiple servers and multiple backends. The
resulting group behaves in (almost) all respects like any real group:
messages may be threaded, marked, moved, deleted, copied, etc. nnselect
groups may be ephemeral or persistent.

An obvious use case for nnselect is to create a functional group out of
the messages produced by some kind of search. In this feature branch
searching (performed by nnir) simply returns a list of articles that is
passed to nnselect, which handles the group creation and message
manipulation. (This should be transparent to the user---that is,
searching should appear to function as before, except that the group
created from the search has more functionality).

Below is a first pass at some documentation that hasn't yet been pushed
into the gnus manual on the branch.

=nnselect= is a select method for virtual groups composed of messages
from other groups that may span multiple servers and backends. For the
most part these groups behave like any other group: messages may be
threaded, marked, moved, deleted, copied, etc.; groups may be ephemeral
or persistent; groups may be created via gnus-group-make-group or
browsed as foreign via gnus-group-browse-foreign-server.

The key to using an nnselect group is specifying the messages to
include. Each nnselect group has a group parameter =nnselect-specs=
which is an alist with two elements: a function =nnselect-function=; and
arguments =nnselect-args= to be passed to the function, if any.

The function =nnselect-function= must return a vector. Each element of
this vector is in turn a 3-element vector corresponding to one
message. The 3 elements are: the fully-qualified group name; the message
number; and a "score" that can be used for additional sorting. The
values for the score are arbitrary, and are not used directly by the
nnselect method---they may, for example, all be set to 100.

As an example we might have a group whose nnselect-specs are

 (nnselect-specs
  (nnselect-function . identity)
  (nnselect-args .
                 [["nnimap+work:mail" 595 100]
                  ["nnimap+home:sent" 223 100]
                ["nntp+news.gmane.org:gmane.emacs.gnus.general" 23666 100])))

Here the function is the identity and the argument is just a list of
messages.

Or the function might be a search query (using nnir):

 (nnselect-specs
  (nnselect-function . nnir-run-query)
  (nnselect-args
   (nnir-query-spec
    (query . "FLAGGED")
    (criteria . ""))
   (nnir-group-spec
    ("nnimap:home")
    ("nnimap:work"))))

This creates a group composed of all flagged messages on two imap
servers, "home" and "work".

And one last example. Here is a function that runs a search query to
find all message that have been received over the past =days=

(defun my-recent-email (days)
      (let ((query-spec
             (list
              (cons 'query
                    (format-time-string "SENTSINCE %d-%b-%Y"
                                        (time-subtract (current-time)
                                                       (days-to-time days))))
              (cons 'criteria "")))
            (group-spec  (list (list "nnimap:home") (list "nnimap:work"))))
        (nnir-run-query   (cons 'nnir-specs
                                (list (cons 'nnir-query-spec query-spec)
                                      (cons 'nnir-group-spec group-spec))))))

Then an nnselect-specs 

 (nnselect-specs
  (nnselect-function . my-recent-email)
  (nnselect-args . 7))

will provide a group composed of all messages on the home and work
servers received in the last 7 days.

** scanning

Refreshing the selection of an nnselect group by running the
nnselect-function may take a long time to complete. Consequently
nnselect groups are not refreshed by default when
=gnus-group-get-new-news= is invoked. In those cases where running the
function is not too time-consuming, a group parameter of nnselect-rescan
can be set to allow automatic refreshing. A refresh can always be
invoked manually through =gnus-group-get-new-news-this-group=.

bug#37871: 27.0.50; Merge feature/gnus-select2

[Lars Ingebrigtsen]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> Hi all, Andy Cohen's nnselect Gnus backend looks ready to merge -- he
> wrote to emacs.devel but it occurs to me it might be better to start
> here. His original message is here, and I've pasted the full text below
> the link:
>
> https://lists.gnu.org/archive/html/emacs-devel/2019-10/msg00689.html

Sounds nice.  Is there any impact on Gnus itself, or is it all
constrained to the new backend?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#37871: 27.0.50; Merge feature/gnus-select2

[Eric Abrahamsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> Hi all, Andy Cohen's nnselect Gnus backend looks ready to merge -- he
>> wrote to emacs.devel but it occurs to me it might be better to start
>> here. His original message is here, and I've pasted the full text below
>> the link:
>>
>> https://lists.gnu.org/archive/html/emacs-devel/2019-10/msg00689.html
>
> Sounds nice.  Is there any impact on Gnus itself, or is it all
> constrained to the new backend?

It shifts nnir over to use nnselect for holding search results, I think
that's all. It makes nnir much easier to use (in the hacking sense,
should be transparent for users). It also let me write the unified
gnus-search interface on top of it, which wasn't possible with the
current implementation of nnir.

bug#37871: 27.0.50; Merge feature/gnus-select2

[Lars Ingebrigtsen]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> It shifts nnir over to use nnselect for holding search results, I think
> that's all. It makes nnir much easier to use (in the hacking sense,
> should be transparent for users). It also let me write the unified
> gnus-search interface on top of it, which wasn't possible with the
> current implementation of nnir.

Sounds great.  Merge away.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#37871: 27.0.50; Merge feature/gnus-select2

[Eric Abrahamsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> It shifts nnir over to use nnselect for holding search results, I think
>> that's all. It makes nnir much easier to use (in the hacking sense,
>> should be transparent for users). It also let me write the unified
>> gnus-search interface on top of it, which wasn't possible with the
>> current implementation of nnir.
>
> Sounds great.  Merge away.  :-)

Andy, do you want to do the honors?

bug#37871: 27.0.50; Merge feature/gnus-select2

[Lars Ingebrigtsen]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

>> Sounds great.  Merge away.  :-)
>
> Andy, do you want to do the honors?

This was in October 2019, and nnselect.el doesn't seem to exist yet.
:-)  Somebody should get mergin'.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#37871: 27.0.50; Merge feature/gnus-select2

[Eric Abrahamsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>>> Sounds great.  Merge away.  :-)
>>
>> Andy, do you want to do the honors?
>
> This was in October 2019, and nnselect.el doesn't seem to exist yet.
> :-)  Somebody should get mergin'.

It's possible Andy was waiting for me to finish my *-retrieve-headers
patch, allowing backends to return headers directly, before he merged
nnselect, as that patch affects how that backend works. Either way, I
really need to take half a day and finish that patch (it's delicate, and
needs some consideration). And Andy should be the one to do the nnselect
merge.

bug#37871: 27.0.50; Merge feature/gnus-select2

[Eric Abrahamsen]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> Lars Ingebrigtsen <larsi@gnus.org> writes:
>
>> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>>
>>>> Sounds great.  Merge away.  :-)
>>>
>>> Andy, do you want to do the honors?
>>
>> This was in October 2019, and nnselect.el doesn't seem to exist yet.
>> :-)  Somebody should get mergin'.
>
> It's possible Andy was waiting for me to finish my *-retrieve-headers
> patch, allowing backends to return headers directly, before he merged
> nnselect, as that patch affects how that backend works. Either way, I
> really need to take half a day and finish that patch (it's delicate, and
> needs some consideration). And Andy should be the one to do the nnselect
> merge.

And it's in!

Lars, what do you think about bumping the Gnus version number for big
changes like this? If only to make bug triage easier. In retrospect I
wish I'd done this for the obarray->hashtable change.

bug#37871: 27.0.50; Merge feature/gnus-select2

[Lars Ingebrigtsen]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> Lars, what do you think about bumping the Gnus version number for big
> changes like this? If only to make bug triage easier. In retrospect I
> wish I'd done this for the obarray->hashtable change.

Hm...  I'm not sure this is a big enough reason to bump the version
number in itself, but it's sure been a while since it's been increased.
The main reason is, of course, that I haven't done the t-shirts for Ma
Gnus yet.  :-/

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#37871: 27.0.50; Merge feature/gnus-select2

[Eric Abrahamsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> Lars, what do you think about bumping the Gnus version number for big
>> changes like this? If only to make bug triage easier. In retrospect I
>> wish I'd done this for the obarray->hashtable change.
>
> Hm...  I'm not sure this is a big enough reason to bump the version
> number in itself, but it's sure been a while since it's been increased.
> The main reason is, of course, that I haven't done the t-shirts for Ma
> Gnus yet.  :-/

Well I've got this gnus-search package I'd like to put in on top of
nnselect, which is much more user-visible, so maybe then.

T-shirts should be the priority, though :)

bug#37871: 27.0.50; Merge feature/gnus-select2

[Basil L. Contovounesios]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> And it's in!

Thanks to everyone who worked on this!

I already pushed some minor code/doc cleanups to more closely follow
Emacs conventions in:

Fix formatting of recent Gnus nnselect changes
d7197f9d99 2020-09-06 16:06:57 +0100
https://git.savannah.gnu.org/cgit/emacs.git/commit/?id=d7197f9d99ca6aa326d38e64ca9eb1d13e18d664

But I have some remaining comments.

First, the merge introduced the following build warning:

  In end of data:
  org/ol-gnus.el:271:1: Warning: the function ‘nnir-article-group’ is not known
      to be defined.

Org should probably be updated to use the new names of things.

Second, several changes in gnus-msg.el have resulted in misindented code
of the form:

  (unwind-protect
      (progn
        (one-single-form))
    ;; No unwind forms!
    )

If the unwind-protect+progn wrappers are no longer needed, they should
be removed and the code reindented.

More worryingly, some of the changes that let-bind gnus-newsgroup-name
are preceded by the following commentary:

  ;; We can't `let' gnus-newsgroup-name here, since that leads
  ;; to local variables leaking.

If the leaking is still an issue, then the nnselect changes should be
reconsidered; otherwise these stale comments should be removed.

Third, the bodies of gnus-fetch-headers and nnheader-parse-nov are
misindented.  (BTW, please respect the setting of indent-tabs-mode in
Emacs' top-level dir-locals-file, and especially don't use tab
characters for aligning comments to the right of code.)

Finally, the merge makes several backward-incompatible changes to user
options which should be rectified, lest we break existing user
configurations.  The renamed user options gnus-refer-thread-use-nnir and
nnir-retrieve-headers-override-function should be defined as obsolete
varaliases for their new names gnus-refer-thread-use-search and
nnselect-retrieve-headers-override-function, respectively.  Also, their
:version tags should be bumped to 28.1.  The removed user option
nnir-summary-line-format should still be defined but made obsolete.

Thanks,

-- 
Basil

bug#37871: 27.0.50; Merge feature/gnus-select2

[Andrew Cohen]

Dear Basil

>>>>> "BLC" == Basil L Contovounesios <contovob@tcd.ie> writes:

[...]

    BLC> I already pushed some minor code/doc cleanups to more closely
    BLC> follow Emacs conventions in:

    BLC> Fix formatting of recent Gnus nnselect changes d7197f9d99
    BLC> 2020-09-06 16:06:57 +0100
    BLC> https://git.savannah.gnu.org/cgit/emacs.git/commit/?id=d7197f9d99ca6aa326d38e64ca9eb1d13e18d664

Thanks for all this!

    BLC> But I have some remaining comments.

    BLC> First, the merge introduced the following build warning:

    BLC>   In end of data: org/ol-gnus.el:271:1: Warning: the function
    BLC> ‘nnir-article-group’ is not known to be defined.

    BLC> Org should probably be updated to use the new names of things.

Clearly this needs to be changed. I wasn't expecting this internal
function to be used outside of gnus (and it probably shouldn't). 

    BLC> Second, several changes in gnus-msg.el have resulted in
    BLC> misindented code of the form:

Ugh. I spent hours trying to make sure the code was indented properly. I
must be doing something wrong.

    BLC>   (unwind-protect (progn (one-single-form)) ;; No unwind forms!
    BLC> )

I'm sorry I don't see exactly which unwind-protect you are referring to?
I don't think I changed any of the several unwind protects in
gnus-msg.el. Maybe someone else remembers why they are this way?

    BLC> If the unwind-protect+progn wrappers are no longer needed, they
    BLC> should be removed and the code reindented.

    BLC> More worryingly, some of the changes that let-bind
    BLC> gnus-newsgroup-name are preceded by the following commentary:

    BLC>   ;; We can't `let' gnus-newsgroup-name here, since that leads
    BLC> ;; to local variables leaking.

    BLC> If the leaking is still an issue, then the nnselect changes
    BLC> should be reconsidered; otherwise these stale comments should
    BLC> be removed.

I believe the leaking was a bug that has been dealt with, so the
comments should be removed. However I might have failed to fix it (I
have had a variety of people test it, but you never know).

    BLC> Third, the bodies of gnus-fetch-headers and nnheader-parse-nov
    BLC> are misindented.  (BTW, please respect the setting of
    BLC> indent-tabs-mode in Emacs' top-level dir-locals-file, and
    BLC> especially don't use tab characters for aligning comments to
    BLC> the right of code.)

See my above comment :(  All settings are at default, so I must be
making some other stupid mistake.

    BLC> Finally, the merge makes several backward-incompatible changes
    BLC> to user options which should be rectified, lest we break
    BLC> existing user configurations.  The renamed user options
    BLC> gnus-refer-thread-use-nnir and
    BLC> nnir-retrieve-headers-override-function should be defined as
    BLC> obsolete varaliases for their new names
    BLC> gnus-refer-thread-use-search and
    BLC> nnselect-retrieve-headers-override-function, respectively.
    BLC> Also, their :version tags should be bumped to 28.1.  The
    BLC> removed user option nnir-summary-line-format should still be
    BLC> defined but made obsolete.

Right. Thanks for noting these.

    BLC> Thanks,

    BLC> -- Basil

bug#37871: 27.0.50; Merge feature/gnus-select2

[Eric Abrahamsen]

This feature is now in and fixed up, so I'm closing this report.