Turning Gnus groups into real objects

Turning Gnus groups into real objects

[Eric Abrahamsen]

[The following is some musings on changing the implementation of
mail/news groups in Gnus, and can be ignored if that doesn't interest
you. I'll send in actual code as bug reports, later.]

I've started thinking about the best way to go about turning Gnus groups
into real objects, and wanted to put some ideas down here.

Currently, Gnus groups are represented using lists -- these lists are
referred to as "infos". There's also the concept of an "entry", which is
basically an unread-message count (an integer) appended to the front of
an info. The term "group" usually just means the string name of the
group.

I'd like to turn groups into real objects using EIEIO. Advantages
include making the code more legible -- removing the distinction between
group/info/entry, and dropping a lot of boilerplate code. It would also
allow very easy adjustments to different kinds of groups: ie a "draft
group" class, an "archive group" class, with their own different
behaviors. The nnimap and nnmaildir backends already extend groups with
extra information, and having actual classes would make that less
awkward.

Thinking about the least-disruptive way of getting there, it seemed like
a good first step to fix up the accessor functions used to make changes
to group data.

Group infos are manipulated using a series of `gnus-info-*' getter
macros (starting line 2797 in gnus.el), with a corresponding series of
`gnus-info-set-*' setter macros, though the setters are used less often.
In fact, quite a bit of the Gnus code ignores the getters/setters
altogether, and manipulates the info lists directly (with code like
"(setcar (nthcdr 3 info)...)", etc).

So I thought a good first step would be to regularize the code so that
it only uses the accessors to touch group data, making it easier to
later change the underlying implementation. But this is going to involve
a fair amount of personal preference.

First of all, I'd love to get rid of the group/info/entry distinction,
and only have "groups" (plus "group-name", if we have to refer to the
string name). So the accessors would be named "gnus-group-*" instead of
"gnus-info-*".

A further personal preference is using a getter function, and then
pairing that with `setf' to set the value (as opposed to having a
distinct setter function). I don't have a very strong argument for this,
I just find it conceptually cleaner.

When defining classes, a slot can have a :reader tag and a :writer tag,
or an :accessor tag that does both. So for example the current code
would look like:

(defclass gnus-group ()
  ((marks
    :type list
    :initform nil
    :writer gnus-info-set-marks
    :reader gnus-info-marks)
   ...))

Whereas my mild preference would be for:

(defclass gnus-group ()
  ((marks
    :type list
    :initform nil
    :accessor gnus-info-marks)
   ...))

And using `setf' with that, to set. The existing `gnus-info-set-*'
macros aren't used all that much, and it wouldn't be too much work to
switch them out for `setf's.

But it would also be great to drop the info nomenclature altogether, in
which case (adding backwards compatibility) all the mark handling code
would look like:

(defclass gnus-group ()
  ((marks
    :type list
    :initform nil
    :accessor gnus-group-marks)
   ...))

(define-obsolete-function-alias 'gnus-info-marks 'gnus-group-marks "27.1")

(defmacro gnus-info-set-marks (group marks &optional _extend)
  (setf (gnus-group-marks info) marks))

(make-obsolete 'gnus-info-set-marks
	       "use `setf' with `gnus-group-marks' instead" "27.1")

This is already fairly long, I'll leave it at that. I'm curious if
anyone's got any opinions about this.

Thanks,
Eric

Re: Turning Gnus groups into real objects

[Andy Moreton]

On Wed 17 Jul 2019, Eric Abrahamsen wrote:

> [The following is some musings on changing the implementation of
> mail/news groups in Gnus, and can be ignored if that doesn't interest
> you. I'll send in actual code as bug reports, later.]

This all sounds useful and sensible. It should make it easier to
maintain gnus and to extend it to support new backends etc. I assume
that the idea is to change the runtime representation, while leaving
the serialized state in .gnusrc.eld unchanged.

Please write tests when doing this refactoring work, so that it is
easier to detect when changes cause compatibility problems.

Thanks for working on this,

    AndyM

Re: Turning Gnus groups into real objects

[Stefan Monnier]

> I'd like to turn groups into real objects

I think that can make a fair bit of sense.

> using EIEIO.

But I have to warn you that EIEIO objects are pretty costly, i.e. slow
to construct (which goes through `make-instance`) and slow to access
their slots (which goes through `slot-value`).  Depending on your code's
structure it might not matter too much, or it might render the code
unacceptably slow.

If creation of those objects and/or access to their slots is expected to
be frequent enough to matter, you might be better off using
`cl-defstruct` instead, which is much more lightweight and will likely
serve you just as well here.


        Stefan

Re: Turning Gnus groups into real objects

[Eric Abrahamsen]

Andy Moreton <andrewjmoreton@gmail.com> writes:

> On Wed 17 Jul 2019, Eric Abrahamsen wrote:
>
>> [The following is some musings on changing the implementation of
>> mail/news groups in Gnus, and can be ignored if that doesn't interest
>> you. I'll send in actual code as bug reports, later.]
>
> This all sounds useful and sensible. It should make it easier to
> maintain gnus and to extend it to support new backends etc. I assume
> that the idea is to change the runtime representation, while leaving
> the serialized state in .gnusrc.eld unchanged.

Yup, the group serialization won't change, so the files should remain
compatible. In fact the "real" useful change will be turning the
servers into objects, but that will take a lot of work, and these other
changes are sort of meant to "clear out the undergrowth" first, as it
were.

> Please write tests when doing this refactoring work, so that it is
> easier to detect when changes cause compatibility problems.

Will do -- another (eventual) goal of these changes is to separate out
data structures from representation, so the tests could create a dummy
server and groups and test them without needing a whole installation in
place.

Re: Turning Gnus groups into real objects

[Eric Abrahamsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> I'd like to turn groups into real objects
>
> I think that can make a fair bit of sense.
>
>> using EIEIO.
>
> But I have to warn you that EIEIO objects are pretty costly, i.e. slow
> to construct (which goes through `make-instance`) and slow to access
> their slots (which goes through `slot-value`).  Depending on your code's
> structure it might not matter too much, or it might render the code
> unacceptably slow.
>
> If creation of those objects and/or access to their slots is expected to
> be frequent enough to matter, you might be better off using
> `cl-defstruct` instead, which is much more lightweight and will likely
> serve you just as well here.

Very good to know! I was originally wavering between eieio objects and
structs, and didn't have a very compelling reason to end up with
objects, except I thought we could "probably use" the extra
functionality at some point. But speed is most definitely a concern, and
totally outweighs any hypothetical future usefulness of objects. I'll go
with structs.

Thanks,
Eric

Re: Turning Gnus groups into real objects

[Lars Ingebrigtsen]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> Group infos are manipulated using a series of `gnus-info-*' getter
> macros (starting line 2797 in gnus.el), with a corresponding series of
> `gnus-info-set-*' setter macros, though the setters are used less often.
> In fact, quite a bit of the Gnus code ignores the getters/setters
> altogether, and manipulates the info lists directly (with code like
> "(setcar (nthcdr 3 info)...)", etc).

Fixing up that (and using setf instead of info-set-*) would be nice.

> So I thought a good first step would be to regularize the code so that
> it only uses the accessors to touch group data, making it easier to
> later change the underlying implementation. But this is going to involve
> a fair amount of personal preference.
>
> First of all, I'd love to get rid of the group/info/entry distinction,
> and only have "groups" (plus "group-name", if we have to refer to the
> string name). So the accessors would be named "gnus-group-*" instead of
> "gnus-info-*".

The "group" thing is already a very overloaded concept.  "gnus-group-"
function refer to the gnus-group-mode, for the most part.  I think it
makes more sense to just keep the "info" concept.  I mean, a "group
object" could equally well refer to the state of the info in a summary
buffer, for instance.

> When defining classes, a slot can have a :reader tag and a :writer tag,
> or an :accessor tag that does both. So for example the current code
> would look like:
>
> (defclass gnus-group ()
>   ((marks
>     :type list
>     :initform nil
>     :writer gnus-info-set-marks
>     :reader gnus-info-marks)
>    ...))

As for using cl-defstructs for the group infos -- I don't think that's
realistic.  The most important reason is that (as you've noted) a lot of
the code don't use the setters and getters.  That can be fixed in-tree,
but people all around the world have code out-of-tree that accesses
those things.  The design of those data structures have basically not
changed, only been extended, since around 1987.

The other reason is that people do have tens of thousands of groups,
especially the few that are still using NNTP servers.  Getting from the
serialised format to the in-memory Gnus structures (but the info and the
group entries) uses (probably) fastest path possible.  This conversion
into another data type will probably make startup significantly slower
for these users.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Turning Gnus groups into real objects

[Eric Abrahamsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> Group infos are manipulated using a series of `gnus-info-*' getter
>> macros (starting line 2797 in gnus.el), with a corresponding series of
>> `gnus-info-set-*' setter macros, though the setters are used less often.
>> In fact, quite a bit of the Gnus code ignores the getters/setters
>> altogether, and manipulates the info lists directly (with code like
>> "(setcar (nthcdr 3 info)...)", etc).
>
> Fixing up that (and using setf instead of info-set-*) would be nice.

Okay, I'll continue that work.

>> So I thought a good first step would be to regularize the code so that
>> it only uses the accessors to touch group data, making it easier to
>> later change the underlying implementation. But this is going to involve
>> a fair amount of personal preference.
>>
>> First of all, I'd love to get rid of the group/info/entry distinction,
>> and only have "groups" (plus "group-name", if we have to refer to the
>> string name). So the accessors would be named "gnus-group-*" instead of
>> "gnus-info-*".
>
> The "group" thing is already a very overloaded concept.  "gnus-group-"
> function refer to the gnus-group-mode, for the most part.  I think it
> makes more sense to just keep the "info" concept.  I mean, a "group
> object" could equally well refer to the state of the info in a summary
> buffer, for instance.

I'm totally fine with that, but if we're not able to change the
underlying data structures, it's mostly a moot point -- we're still
going to need group vs info vs entry.

>> When defining classes, a slot can have a :reader tag and a :writer tag,
>> or an :accessor tag that does both. So for example the current code
>> would look like:
>>
>> (defclass gnus-group ()
>>   ((marks
>>     :type list
>>     :initform nil
>>     :writer gnus-info-set-marks
>>     :reader gnus-info-marks)
>>    ...))
>
> As for using cl-defstructs for the group infos -- I don't think that's
> realistic.  The most important reason is that (as you've noted) a lot of
> the code don't use the setters and getters.  That can be fixed in-tree,
> but people all around the world have code out-of-tree that accesses
> those things.  The design of those data structures have basically not
> changed, only been extended, since around 1987.
>
> The other reason is that people do have tens of thousands of groups,
> especially the few that are still using NNTP servers.  Getting from the
> serialised format to the in-memory Gnus structures (but the info and the
> group entries) uses (probably) fastest path possible.  This conversion
> into another data type will probably make startup significantly slower
> for these users.

Well... bummer. I guess I still feel that any additional slowdown in
startup could be offset by speedups elsewhere, but if we want to keep
that level of backward compatibility, there's not much else that can be
done.

My original long-term plan was to replace the nnoo machinery with Emacs'
built-in object stuff: defvoo would become object slots, and deffoo
would become generic functions. Would this be a bad idea for the same
reason? That would also be a bummer, but at the same time save me a lot
of work :)

Thanks,
Eric

Re: Turning Gnus groups into real objects

[Lars Ingebrigtsen]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> My original long-term plan was to replace the nnoo machinery with Emacs'
> built-in object stuff: defvoo would become object slots, and deffoo
> would become generic functions. Would this be a bad idea for the same
> reason? That would also be a bummer, but at the same time save me a lot
> of work :)

The reason the voo stuff is like it is is for backwards-compatibility
with out-of-tree backends.  If we want to keep that, then rewriting to
use a proper object system would require two pairs of interface
functions.

But I think rewriting that stuff is a more self-contained operation
because the nn* interface at least is defined as an interface (which the
group/info structures are not).  So I think that's a much better thing
to be rewritten.

I.e., first define what the interface should be, then rewrite the
backends, one by one, to use that new and better interface.  People who
have customisations to the code will find that their customisations
don't work in the new versions, but that's OK, I think.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Turning Gnus groups into real objects

[Eric Abrahamsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> My original long-term plan was to replace the nnoo machinery with Emacs'
>> built-in object stuff: defvoo would become object slots, and deffoo
>> would become generic functions. Would this be a bad idea for the same
>> reason? That would also be a bummer, but at the same time save me a lot
>> of work :)
>
> The reason the voo stuff is like it is is for backwards-compatibility
> with out-of-tree backends.  If we want to keep that, then rewriting to
> use a proper object system would require two pairs of interface
> functions.

I guess it seems unlikely that a lot of external code is messing with
server internals, but I suppose it's possible that some Gnus astronauts
have written entire new backends for themselves. But maintaining two
pairs of interface functions would be a nightmare, and likely negate
most of the potential benefit of using generic functions.

> But I think rewriting that stuff is a more self-contained operation
> because the nn* interface at least is defined as an interface (which the
> group/info structures are not).  So I think that's a much better thing
> to be rewritten.
>
> I.e., first define what the interface should be, then rewrite the
> backends, one by one, to use that new and better interface.  People who
> have customisations to the code will find that their customisations
> don't work in the new versions, but that's OK, I think.

Okay, cool. Just for clarity, we'd end up with something looking like:

(defclass gnus-server ()
  ((address :type string)
  ...))

(defclass gnus-nnimap (gnus-server)
  ((capabilities
    :type string)
   (newlinep
    :type boolean)
   ...))

(cl-defmethod gnus-request-list ((server gnus-nnimap))
  (when (nnimap-change-group nil server)
    (with-current-buffer nntp-server-buffer
      (erase-buffer)
      (let ((groups
	     (with-current-buffer (nnimap-buffer)
	       (nnimap-get-groups)))
	    sequences responses)
	...))))

I'd aim at a few general principles:

1. Move more code into the methods, in particular the base methods.
2. Where possible, give servers their own process buffer (for if we ever
   want to make Gnus threaded).
3. Let all servers keep track of a list of "their" groups.
4. Wherever possible, pass servers as arguments instead of guessing them
   from the environment.
5. Keep a "real" list of servers, equivalent to `(cons gnus-select-method
   (append gnus-secondary-select-methods gnus-server-alist))', so we
   don't have to keep looking for them.

I can try to start off with the absolute smallest change possible, but
something tells me it won't work to switch the backends over gradually,
ie to maintain a hybrid system for a while. It will probably need to
happen all in one go (though again, I can minimize changes in the
beginning).

Anyway, all this is still quite a ways down the road. Thanks for looking
at this.

Eric

Re: Turning Gnus groups into real objects

[Stefan Monnier]

> (defclass gnus-server ()
>   ((address :type string)
>   ...))
>
> (defclass gnus-nnimap (gnus-server)
>   ((capabilities
>     :type string)
>    (newlinep
>     :type boolean)
>    ...))

Aka

    (cl-defstruct (gnus-server)
      (address nil :type string))

    (cl-defstruct (gnus-nnimap
                   (:include gnus-server))
      (capabilities nil :type string)
      (newlinep     nil :type boolean))

> (cl-defmethod gnus-request-list ((server gnus-nnimap))

BTW, I was wondering whether we could preserve backward compatiblity with
defvoo and defffoo.  But I guess for `defvoo`, we'd need the equivalent
of `define-symbol-macro` whereas we currently only have `cl-symbol-macrolet`.


        Stefan

Re: Turning Gnus groups into real objects

[Eric Abrahamsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> (defclass gnus-server ()
>>   ((address :type string)
>>   ...))
>>
>> (defclass gnus-nnimap (gnus-server)
>>   ((capabilities
>>     :type string)
>>    (newlinep
>>     :type boolean)
>>    ...))
>
> Aka
>
>     (cl-defstruct (gnus-server)
>       (address nil :type string))
>
>     (cl-defstruct (gnus-nnimap
>                    (:include gnus-server))
>       (capabilities nil :type string)
>       (newlinep     nil :type boolean))
>
>> (cl-defmethod gnus-request-list ((server gnus-nnimap))

Aw... I was also considering a bit of (tastefully designed) multiple
inheritance.

> BTW, I was wondering whether we could preserve backward compatiblity with
> defvoo and defffoo.  But I guess for `defvoo`, we'd need the equivalent
> of `define-symbol-macro` whereas we currently only have `cl-symbol-macrolet`.

That would be an interesting thing to have for many reasons. But here it
seems like it would mainly be useful to support backends defined outside
of Gnus, in which case we'd also have to cover `nnoo-declare' and
friends, and... yuck.

Re: Turning Gnus groups into real objects

[Lars Ingebrigtsen]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> I guess it seems unlikely that a lot of external code is messing with
> server internals, but I suppose it's possible that some Gnus astronauts
> have written entire new backends for themselves.

Yes, there's a bunch of out-of-tree backends out there.

> But maintaining two pairs of interface functions would be a nightmare,
> and likely negate most of the potential benefit of using generic
> functions.

I don't see why.  You'd just leave the gnus-int-* functions alone (but
deprecated) and carry on.

> Okay, cool. Just for clarity, we'd end up with something looking like:
>
> (defclass gnus-server ()
>   ((address :type string)
>   ...))
>
> (defclass gnus-nnimap (gnus-server)
>   ((capabilities
>     :type string)
>    (newlinep
>     :type boolean)
>    ...))
>
> (cl-defmethod gnus-request-list ((server gnus-nnimap))
>   (when (nnimap-change-group nil server)
>     (with-current-buffer nntp-server-buffer
>       (erase-buffer)
>       (let ((groups
> 	     (with-current-buffer (nnimap-buffer)
> 	       (nnimap-get-groups)))
> 	    sequences responses)
> 	...))))

Oh, I see -- want the objects to be in Gnus, not just in the backends?
Yes, then I can see how that'd be more complicated to have the two
interfaces.

But I don't there's that much value in carrying around the backend
objects inside Gnus.  I think the current list-based server objects
could still live on the Gnus side.  And I think moving away from those
would lead to, basically, breaking half the Gnus users' .gnus.el files.

> I'd aim at a few general principles:
>
> 1. Move more code into the methods, in particular the base methods.

I'm not sure what you mean here...

> 2. Where possible, give servers their own process buffer (for if we ever
>    want to make Gnus threaded).

Don't they have that already?

> 3. Let all servers keep track of a list of "their" groups.

Keep track in what way?

> 4. Wherever possible, pass servers as arguments instead of guessing them
>    from the environment.

Sure.

> 5. Keep a "real" list of servers, equivalent to `(cons gnus-select-method
>    (append gnus-secondary-select-methods gnus-server-alist))', so we
>    don't have to keep looking for them.

I'm not sure what you refer to -- there's a bunch of code that allows
you to refer to servers based on the name as well as the actual backend
definition (that has grown for historical reasons)...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Turning Gnus groups into real objects

[Eric Abrahamsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>

[...]

> Oh, I see -- want the objects to be in Gnus, not just in the backends?
> Yes, then I can see how that'd be more complicated to have the two
> interfaces.
>
> But I don't there's that much value in carrying around the backend
> objects inside Gnus. I think the current list-based server objects
> could still live on the Gnus side. And I think moving away from those
> would lead to, basically, breaking half the Gnus users'.gnus.el files.

Okay, maybe this whole thing would end up being too intrusive, then. The
point would be to change calling conventions of the server functions, to
increase code re-use and composability. I suppose it might be possible
to do a bare minimum of replacing nnoo with structs, but then we
wouldn't be able to replace deffoo with generics. And I don't see how we
could use structs inside the backend file, but lists in the rest of the
code.

>> I'd aim at a few general principles:
>>
>> 1. Move more code into the methods, in particular the base methods.
>
> I'm not sure what you mean here...

See below

>> 2. Where possible, give servers their own process buffer (for if we ever
>> want to make Gnus threaded).
>
> Don't they have that already?

Near as I can tell, all the backends dump their remote responses into
`nntp-server-buffer'. That works fine because they each dump and read in
turn, and erase the buffer when they start. With threading, the
responses get dumped in more or less random order, so they end up
reading each other's responses. I ran into this when I was working on
gnus-search.el, and got excited about searching multiple IMAP servers
concurrently, using threads.

>> 3. Let all servers keep track of a list of "their" groups.
>
> Keep track in what way?
>
>> 4. Wherever possible, pass servers as arguments instead of guessing them
>>    from the environment.
>
> Sure.
>
>> 5. Keep a "real" list of servers, equivalent to `(cons gnus-select-method
>>    (append gnus-secondary-select-methods gnus-server-alist))', so we
>>    don't have to keep looking for them.
>
> I'm not sure what you refer to -- there's a bunch of code that allows
> you to refer to servers based on the name as well as the actual backend
> definition (that has grown for historical reasons)...

Okay, one theoretical example that combines all of the above (and will
probably illustrate why this whole project is a non-starter):

Right now, every time you hit "g" in the Group buffer, Gnus cycles over
all groups, collects those eligible for refresh, rebuilds the relevant
list of servers for those groups, and then asks each server to update
its particular subset of groups.

What I think could be much cleaner (and I'm not saying this is
necessarily a great idea, it's just illustrative) is that
`gnus-get-unread-articles' would look like:

(defun gnus-get-unread-articles (level)
  (dolist (server (gnus-all-servers))
    (gnus-server-update level)))

Where `gnus-all-servers' just returns a list of all the server objects.
`gnus-server-update' would be a generic function looking like:

(cl-defgeneric gnus-server-update ((server gnus-server)
				   level)
  (let ((groups (seq-filter (lambda (g)
			      (>= level (gnus-info-level g)))
			    (gnus-server-groups server))))
    (when groups
      update groups...)))

This is what I mean by "move code into base methods" (and
`gnus-server-groups' is the "keeping track of their groups" part). This
base method applies to all servers, but different server classes would
have the opportunity to augment it using :before :after and :around
methods, or override it completely.

Again, this particular case is just for illustration of the principle.

Anyway, I'm guessing all this would simply be too intrusive. So if we
wanted to preserve compatibility with backends defined out-of-tree, we
could a) redefine nnoo-declare/defvoo/deffoo to create ad-hoc structs
(probably not), or b) adjust gnus-check-backend-function and
gnus-get-function to check if the server is a list or a struct and
dispatch to different kinds of functions. But doing it this way would
mean having to keep nnoo.el, getting none of the benefits of generic
functions, and adding complexity and confusion.

I can't think of any other way of preserving compatibility. Probably
this just isn't feasible without breaking support.

Eric

Re: Turning Gnus groups into real objects

[Barry Fishman]

This seems like its going to be a full rewrite, where some users are forced to
have to do a lot of work just to keep their code working.  Would it be a
bit easier if you just forked the project and used a new package prefix.
Then the code could get debugged as a separate mail system, and people
could transition to it at their convenience.

It would certainly make finding one's code that needs to be changed
easier, and since the data file are kept the same, one can avoid loosing
any data even if they try both systems (not concurrently) during a
transition.

--
Barry Fishman

Re: Turning Gnus groups into real objects

[Stefan Monnier]

> (probably not), or b) adjust gnus-check-backend-function and
> gnus-get-function to check if the server is a list or a struct and
> dispatch to different kinds of functions.

FWIW, cl-defmethod can also dispatch based on whether it's a struct or
list, but I'm not sure so the "old-style" functionality could maybe be
implemented as a kind of "compatibility backend of type `cons`"?


        Stefan

Re: Turning Gnus groups into real objects

[Lars Ingebrigtsen]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

>>> 2. Where possible, give servers their own process buffer (for if we ever
>>> want to make Gnus threaded).
>>
>> Don't they have that already?
>
> Near as I can tell, all the backends dump their remote responses into
> `nntp-server-buffer'. That works fine because they each dump and read in
> turn, and erase the buffer when they start. With threading, the
> responses get dumped in more or less random order, so they end up
> reading each other's responses. I ran into this when I was working on
> gnus-search.el, and got excited about searching multiple IMAP servers
> concurrently, using threads.

Oh, that server buffer.  Yeah, most backends have their own per-server
buffer for communicating with the, er, server, which is then parsed and
then dumped into nntp-server-buffer in the format Gnus expects.

I'd have expected a new backend interface not to use
nntp-server-buffer -- or any buffer -- for communication with Gnus, but
just return articles as a list of objects.  It'd be more efficient.

> (cl-defgeneric gnus-server-update ((server gnus-server)
> 				   level)
>   (let ((groups (seq-filter (lambda (g)
> 			      (>= level (gnus-info-level g)))
> 			    (gnus-server-groups server))))
>     (when groups
>       update groups...)))
>
> This is what I mean by "move code into base methods" (and
> `gnus-server-groups' is the "keeping track of their groups" part). This
> base method applies to all servers, but different server classes would
> have the opportunity to augment it using :before :after and :around
> methods, or override it completely.

I think that this sounds like code duplication, doesn't it?  And while
IMAP does have an in-backend sense of readedness etc, most of the other
backends don't...

> Anyway, I'm guessing all this would simply be too intrusive. So if we
> wanted to preserve compatibility with backends defined out-of-tree, we
> could a) redefine nnoo-declare/defvoo/deffoo to create ad-hoc structs
> (probably not), or b) adjust gnus-check-backend-function and
> gnus-get-function to check if the server is a list or a struct and
> dispatch to different kinds of functions. But doing it this way would
> mean having to keep nnoo.el, getting none of the benefits of generic
> functions, and adding complexity and confusion.

It would mean keeping nnoo.el, but it'd be deprecated and would
eventually go away.

I don't really see much of a complication here.  You call functions like
`gnus-open-server' (that takes a method), and it'd look at whether the
backend is new-style or old-style and call the backends according to the
new or old conventions.  (And the new-style is, of course, with the
backend state in a struct instead of spread out over a bunch of
variables.)

There's a limited number of interface functions...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Turning Gnus groups into real objects

[Eric Abrahamsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>>>> 2. Where possible, give servers their own process buffer (for if we ever
>>>> want to make Gnus threaded).
>>>
>>> Don't they have that already?
>>
>> Near as I can tell, all the backends dump their remote responses into
>> `nntp-server-buffer'. That works fine because they each dump and read in
>> turn, and erase the buffer when they start. With threading, the
>> responses get dumped in more or less random order, so they end up
>> reading each other's responses. I ran into this when I was working on
>> gnus-search.el, and got excited about searching multiple IMAP servers
>> concurrently, using threads.
>
> Oh, that server buffer.

Sorry, that was worded wrong.

> Yeah, most backends have their own per-server buffer for communicating
> with the, er, server, which is then parsed and then dumped into
> nntp-server-buffer in the format Gnus expects.
>
> I'd have expected a new backend interface not to use
> nntp-server-buffer -- or any buffer -- for communication with Gnus, but
> just return articles as a list of objects.  It'd be more efficient.

Yes! That was definitely something I had in mind. We shouldn't have to
take server output and munge it into some other server output and then
parse it.

>> (cl-defgeneric gnus-server-update ((server gnus-server)
>> 				   level)
>>   (let ((groups (seq-filter (lambda (g)
>> 			      (>= level (gnus-info-level g)))
>> 			    (gnus-server-groups server))))
>>     (when groups
>>       update groups...)))
>>
>> This is what I mean by "move code into base methods" (and
>> `gnus-server-groups' is the "keeping track of their groups" part). This
>> base method applies to all servers, but different server classes would
>> have the opportunity to augment it using :before :after and :around
>> methods, or override it completely.
>
> I think that this sounds like code duplication, doesn't it?  And while
> IMAP does have an in-backend sense of readedness etc, most of the other
> backends don't...

Actually, done right, it should result in *less* code duplication. Code
common to all servers goes in base methods that specialize on
`gnus-server' -- the same code runs for all servers. Code specific to a
class of backends or a single backend goes in a method that augments or
is composed with that base method somehow.

Anyhow, this is moot if we're not changing the interface. At least,
mostly moot...

>> Anyway, I'm guessing all this would simply be too intrusive. So if we
>> wanted to preserve compatibility with backends defined out-of-tree, we
>> could a) redefine nnoo-declare/defvoo/deffoo to create ad-hoc structs
>> (probably not), or b) adjust gnus-check-backend-function and
>> gnus-get-function to check if the server is a list or a struct and
>> dispatch to different kinds of functions. But doing it this way would
>> mean having to keep nnoo.el, getting none of the benefits of generic
>> functions, and adding complexity and confusion.
>
> It would mean keeping nnoo.el, but it'd be deprecated and would
> eventually go away.
>
> I don't really see much of a complication here.  You call functions like
> `gnus-open-server' (that takes a method), and it'd look at whether the
> backend is new-style or old-style and call the backends according to the
> new or old conventions.  (And the new-style is, of course, with the
> backend state in a struct instead of spread out over a bunch of
> variables.)

Right, that's what I meant by adjusting check-backend-function and
gnus-get-function. Or maybe Stefan's right that there's a way to do it
all with generic functions.

Anyway, it sounds like this might be feasible. I understand we're
keeping support for out-of-tree backends and not changing the backend
interface. It will take me a little while to get to this, but I'll be
back when I've got some proof-of-concept code to look at.

Thanks,
Eric

Re: Turning Gnus groups into real objects

[Eric Abrahamsen]

On 07/21/19 09:40 AM, Stefan Monnier wrote:
>> (probably not), or b) adjust gnus-check-backend-function and
>> gnus-get-function to check if the server is a list or a struct and
>> dispatch to different kinds of functions.
>
> FWIW, cl-defmethod can also dispatch based on whether it's a struct or
> list, but I'm not sure so the "old-style" functionality could maybe be
> implemented as a kind of "compatibility backend of type `cons`"?

If I'm understanding you correctly, you're saying we could "do
something" to the old style backends so that they're more definitely
identifiable as a Gnus server, and not just any old list?

It's possible to write new specializers for methods, right? If we could
use that to positively identify a old-style backend, I would love to
move the whole machinery into generic functions.

Re: Turning Gnus groups into real objects

[Eric Abrahamsen]

Barry Fishman <barry@ecubist.org> writes:

> This seems like its going to be a full rewrite, where some users are forced to
> have to do a lot of work just to keep their code working.  Would it be a
> bit easier if you just forked the project and used a new package prefix.
> Then the code could get debugged as a separate mail system, and people
> could transition to it at their convenience.
>
> It would certainly make finding one's code that needs to be changed
> easier, and since the data file are kept the same, one can avoid loosing
> any data even if they try both systems (not concurrently) during a
> transition.

Ugh, I'm very wary of ending up in a "you break it, you buy it"
situation. In terms of what we're talking about in this thread, anyway,
backwards compatibility should be maintained, not only for users but
also for hackers. I suppose it's possible that at some point in the
future we could consider taking a deep breath and breaking
compatibility -- and in that case I think you're right that a temporary
fork might be a good idea -- but I'm in no hurry to get there.