gnus-server-to-method crash on virtual server name in gnus-secondary-select-methods

gnus-server-to-method crash on virtual server name in gnus-secondary-select-methods

[Greg Klanderman]

Hi, hopefully this is the correct list for reporting Gnus problems
these days; I see there is also info-gnus-english@ but it doesn't look
terribly active.

Not subscribed so please include me in replies.

I'm a long time Gnus user since early/mid 90's, and was fairly active
reporting problems back then (still listed in 'Contributors'!).

I have recently transitioned my email from nnfolder to nnmaildir as I
ran up against 1Gb group file size limit (still on XEmacs 21.4 and
Gnus 5.10.8).

[As an aside, it took me some time to realize that I could not just
incrementally transition my largest groups; it would appear that you
can only have new mail delivered to a single server (get-new-mail=t)
when splitting with Gnus.  If that is still the case with current
development version, it would be really nice to explain that
somewhere in the manual.  I did look at the current manual online

https://www.gnu.org/software/emacs/manual/html_node/gnus/index.html

and didn't see this covered.  I had expected that nnmail-split-fancy
would be able to just deliver to multiple backends/servers, either
using the explicit "backend+name:group" form, or that it would look up
the server based on the group name and deliver properly.  But it seems
that there is a loop over servers in gnus-secondary-select-methods,
and whichever appears first and allows getting new mail gets all the
new mail.]

Anyway, in the process of this transition, I found that when I put my
new nnmaildir server, named "mail" before '(nnfolder "") in
gnus-secondary-select-methods, it crashes on startup in
gnus-server-to-method.  Since that code looks the same in emacs 21.7 /
Gnus 5.13 it seemed warranted to report the issue.

In detail: I still have some legacy nnfolder groups (old groups that
are no longer delivered to).  On startup, gnus-read-active-file loops
over gnus-secondary-select-methods, and when the nnfolder appears
second, Gnus has not yet opened the nnfolder server when
gnus-read-active-file-1 is (first) called with the nnmaildir and loops
over infos, calling gnus-find-method-for-group, in turn calling
gnus-server-to-method for one of the nnfolder groups
(gnus-server-to-method is called with "nnfolder:").

In that function, this code (unchanged in 5.13 / Debian testing emacs-gtk):

    ;; It could be a named method, search all servers
    (let ((servers gnus-secondary-select-methods))
      (while (and servers
                  (not (equal server (format "%s:%s" (caar servers)
                                             (cadar servers)))))
        (pop servers))
      (car servers))

is not handling the fact that gnus-secondary-select-methods may
contain virtual server name strings (only handling select methods,
assuming it can take car/cdr of elements).

For now I can just swap the order in gnus-secondary-select-methods so
that the nnfolder is opened first but someday I'll transition to GNU
emacs and newer Gnus so hopefully you can address this.

[As a second aside, can you explain this code in
gnus-summary-move-article (logically unchanged in current 5.13):

    ;; See whether the article is to be put in the cache.
    (let* ((expirable (gnus-group-auto-expirable-p to-group))
           (marks (if expirable
                      gnus-article-mark-lists
                    (delete '(expirable . expire)
                            (copy-sequence
                             gnus-article-mark-lists))))
       ..)..)

It was unexpected that when I moved (B m) my mail to the new nnmaildir
backend groups, articles marked expirable lost that mark, leaving them
merely read.  Is the condition in the 'if' there maybe reversed?  I
could maybe imagine that when moving to an auto-expirable group you
might want to clear expirable on a move, assuming it will eventually
be marked expirable again, automatically.

IMO moving articles should preserve expirable like any other mark.

I'd also love it if nnmail-expiry-wait were measured relative to when
I mark an article expirable, is there any way to get that behavior in
newer Gnus?]


Thank you for so many decades of great work on Gnus!

Greg

Re: gnus-server-to-method crash on virtual server name in gnus-secondary-select-methods

[Eric Abrahamsen]

Greg Klanderman <gak@klanderman.net> writes:

> Hi, hopefully this is the correct list for reporting Gnus problems
> these days; I see there is also info-gnus-english@ but it doesn't look
> terribly active.
>
> Not subscribed so please include me in replies.
>
> I'm a long time Gnus user since early/mid 90's, and was fairly active
> reporting problems back then (still listed in 'Contributors'!).

Welcome back!

> I have recently transitioned my email from nnfolder to nnmaildir as I
> ran up against 1Gb group file size limit (still on XEmacs 21.4 and
> Gnus 5.10.8).
>

[...]

> Anyway, in the process of this transition, I found that when I put my
> new nnmaildir server, named "mail" before '(nnfolder "") in
> gnus-secondary-select-methods, it crashes on startup in
> gnus-server-to-method. Since that code looks the same in emacs 21.7 /
> Gnus 5.13 it seemed warranted to report the issue.
>
> In detail: I still have some legacy nnfolder groups (old groups that
> are no longer delivered to). On startup, gnus-read-active-file loops
> over gnus-secondary-select-methods, and when the nnfolder appears
> second, Gnus has not yet opened the nnfolder server when
> gnus-read-active-file-1 is (first) called with the nnmaildir and loops
> over infos, calling gnus-find-method-for-group, in turn calling
> gnus-server-to-method for one of the nnfolder groups
> (gnus-server-to-method is called with "nnfolder:").
>
> In that function, this code (unchanged in 5.13 / Debian testing emacs-gtk):
>
>;; It could be a named method, search all servers
>     (let ((servers gnus-secondary-select-methods))
>       (while (and servers
>                   (not (equal server (format "%s:%s" (caar servers)
>                                              (cadar servers)))))
>         (pop servers))
>       (car servers))
>
> is not handling the fact that gnus-secondary-select-methods may
> contain virtual server name strings (only handling select methods,
> assuming it can take car/cdr of elements).

Okay, I'm not 100% parsing the above. I think I understand the
fundamental issue: on startup, when Gnus is looping over all infos and
categorizing them by select method, it barfs somehow when checking if an
info belongs to your nnfolder server, because it somehow can't look up
the server. Because... it's a string name, not a (list) method? I'm not
sure. Can you show your complete config for
`gnus-secondary-select-methods'? I don't think you're supposed to be
able to define a server as a plain string.

I have left your other asides aside! While this is a fine place to raise
general Gnus questions/issues (the gnus.general group would be another
option), if you have a very specific problem or feature request you'd
like to make, I would recommend just opening a bug report. Lars is
watching those pretty closely.

Also, upgrade your Emacs :)

Eric

Re: gnus-server-to-method crash on virtual server name in gnus-secondary-select-methods

[Greg Klanderman]

Hi Eric,

Thank you for your reply and sorry for the delay getting back to you..

I ended up spending a lot of time looking into nnmaildir issues the
last few weeks after switching to it, and have now decided that nnml
is probably the better choice despite some warnings below here

https://www.gnu.org/software/emacs/manual/html_node/gnus/Choosing-a-Mail-Back-End.html

and will probably be trying that next.

>>>>> On January 8, 2021 Eric Abrahamsen <eric@ericabrahamsen.net> wrote:

> Welcome back!

thank you!

>> In that function, this code (unchanged in 5.13 / Debian testing emacs-gtk):
>> 
>> ;; It could be a named method, search all servers
>> (let ((servers gnus-secondary-select-methods))
>> (while (and servers
>> (not (equal server (format "%s:%s" (caar servers)
>> (cadar servers)))))
>> (pop servers))
>> (car servers))
>> 
>> is not handling the fact that gnus-secondary-select-methods may
>> contain virtual server name strings (only handling select methods,
>> assuming it can take car/cdr of elements).

> Okay, I'm not 100% parsing the above. I think I understand the
> fundamental issue: on startup, when Gnus is looping over all infos and
> categorizing them by select method, it barfs somehow when checking if an
> info belongs to your nnfolder server, because it somehow can't look up
> the server. Because... it's a string name, not a (list) method?

Almost; because the nnmaildir is a string name.

> I'm not sure. Can you show your complete config for
> `gnus-secondary-select-methods'?

Yes, sorry I should have been more clear; the bug occurs when

gnus-secondary-select-methods is '("mail" (nnfolder ""))

and "mail" is defined in the server buffer as

(nnmaildir "mail"
           (directory "~/Gnus/mail")
           (get-new-mail t))

When the order is swapped, the bug is avoided.

With "mail" appearing first, when looking up an info belonging to the
nnfolder, it hits the code cited, and barfs on caar and cadar of
servers (the car of servers is "mail").

> I don't think you're supposed to be able to define a server as a
> plain string.

According to

https://www.gnu.org/software/emacs/manual/html_node/gnus/Servers-and-Methods.html

not only can you use virtual server names (strings) "wherever you
would normally use a select method", it explicitly states you can do
so in 'gnus-secondary-select-method' (presumably a typo that the final
's' is missing).

So based on that alone, any code which loops over
gnus-secondary-select-methods taking car/cdr of elements is highly
suspicious.  The additional information I gave just demonstrated one
way to reach this problematic logic.

> I have left your other asides aside!

No problem.. probably best to post those separately once I find the
right forum.

> While this is a fine place to raise general Gnus questions/issues
> (the gnus.general group would be another option),

I think you mean 'gmane.emacs.gnus.general'?  It is hard to tell what
information on

https://www.gnu.org/software/emacs/manual/html_node/gnus/Gnus-Development.html

and 

https://www.gnus.org/resources.html

is up-to-date; some links are clearly dead.

Is that newsgroup still bi-directionally gatewayed with ding@gnus.org?

Is there any way to browse archives on the web?

I hadn't known the whole history of gmane or even that it was Lars'
project, sorry to hear of all the trouble that caused him, though I
got side-tracked for some time reading a bunch of his blog and was
reminded how enjoyable his writing is..

> if you have a very specific problem or feature request you'd
> like to make, I would recommend just opening a bug report.
> Lars is watching those pretty closely.

The only reference I could find to reporting bugs is bugs@gnus.org; is
that current?  Is there really no web-based bug tracking system?

> Also, upgrade your Emacs :)

Yeah I know!  And there are a huge number of new features, etc..
Unfortunately in my earlier much less time constrained life 20+ years
ago, I built up a huge amount of .emacs customizations (40+ .el files,
16k+ lines), which will take some time to sort thru and convert the
stuff I'm still using, as well as configure a lot of new default
behavior in GNU emacs.  But xemacs is generally working fine for me.
I do have a very minimal config for GNU emacs (which I need to use for
slime for work), but to tell you the truth, especially now that I've
been WFH 10+ months over an X-forwarding ssh connection, GNU emacs is
so much slower, it is nearly unusable while xemacs is nearly as good
as being in the office (we cannot have code on work laptop).

Greg

Re: gnus-server-to-method crash on virtual server name in gnus-secondary-select-methods

[Eric Abrahamsen]

On 01/18/21 13:07 PM, Greg Klanderman wrote:
> Hi Eric,
>
> Thank you for your reply and sorry for the delay getting back to you..
>
> I ended up spending a lot of time looking into nnmaildir issues the
> last few weeks after switching to it, and have now decided that nnml
> is probably the better choice despite some warnings below here
>
> https://www.gnu.org/software/emacs/manual/html_node/gnus/Choosing-a-Mail-Back-End.html
>
> and will probably be trying that next.
>
>>>>>> On January 8, 2021 Eric Abrahamsen <eric@ericabrahamsen.net> wrote:
>
>> Welcome back!
>
> thank you!
>
>>> In that function, this code (unchanged in 5.13 / Debian testing emacs-gtk):
>>> 
>>> ;; It could be a named method, search all servers
>>> (let ((servers gnus-secondary-select-methods))
>>> (while (and servers
>>> (not (equal server (format "%s:%s" (caar servers)
>>> (cadar servers)))))
>>> (pop servers))
>>> (car servers))
>>> 
>>> is not handling the fact that gnus-secondary-select-methods may
>>> contain virtual server name strings (only handling select methods,
>>> assuming it can take car/cdr of elements).
>
>> Okay, I'm not 100% parsing the above. I think I understand the
>> fundamental issue: on startup, when Gnus is looping over all infos and
>> categorizing them by select method, it barfs somehow when checking if an
>> info belongs to your nnfolder server, because it somehow can't look up
>> the server. Because... it's a string name, not a (list) method?
>
> Almost; because the nnmaildir is a string name.
>
>> I'm not sure. Can you show your complete config for
>> `gnus-secondary-select-methods'?
>
> Yes, sorry I should have been more clear; the bug occurs when
>
> gnus-secondary-select-methods is '("mail" (nnfolder ""))
>
> and "mail" is defined in the server buffer as
>
> (nnmaildir "mail"
>            (directory "~/Gnus/mail")
>            (get-new-mail t))
>
> When the order is swapped, the bug is avoided.
>
> With "mail" appearing first, when looking up an info belonging to the
> nnfolder, it hits the code cited, and barfs on caar and cadar of
> servers (the car of servers is "mail").
>
>> I don't think you're supposed to be able to define a server as a
>> plain string.
>
> According to
>
> https://www.gnu.org/software/emacs/manual/html_node/gnus/Servers-and-Methods.html
>
> not only can you use virtual server names (strings) "wherever you
> would normally use a select method", it explicitly states you can do
> so in 'gnus-secondary-select-method' (presumably a typo that the final
> 's' is missing).
>
> So based on that alone, any code which loops over
> gnus-secondary-select-methods taking car/cdr of elements is highly
> suspicious.  The additional information I gave just demonstrated one
> way to reach this problematic logic.

I still can't believe this is the way it's supposed to be used. The code
snippet you posted (where the error actually arises) has been that way
since The Dawn of Time. I also don't see why you _would_ define a server
via the *Server* buffer, and then put it again in
`gnus-secondary-select-methods'. Generally you define servers either in
one place or the other -- if you've done it with "a" in the *Server*
buffer, there's no need for it to appear in your config files, too.

>> I have left your other asides aside!
>
> No problem.. probably best to post those separately once I find the
> right forum.
>
>> While this is a fine place to raise general Gnus questions/issues
>> (the gnus.general group would be another option),
>
> I think you mean 'gmane.emacs.gnus.general'?  It is hard to tell what
> information on
>
> https://www.gnu.org/software/emacs/manual/html_node/gnus/Gnus-Development.html
>
> and 
>
> https://www.gnus.org/resources.html
>
> is up-to-date; some links are clearly dead.
>
> Is that newsgroup still bi-directionally gatewayed with ding@gnus.org?

Yes, sorry, I did mean gmane.emacs.gnus.general, and yes that's still
gatewayed to ding@gnus.org. So far as I know there's just
gmane.emacs.gnus.general and gmane.emacs.gnus.user. I don't think
there's any real difference anymore (probably "general" used to be more
for development?) but you do get different people responding in the
different groups.

> Is there any way to browse archives on the web?

There used to be, but I think no more, since gmane moved servers.

> I hadn't known the whole history of gmane or even that it was Lars'
> project, sorry to hear of all the trouble that caused him, though I
> got side-tracked for some time reading a bunch of his blog and was
> reminded how enjoyable his writing is..
>
>> if you have a very specific problem or feature request you'd
>> like to make, I would recommend just opening a bug report.
>> Lars is watching those pretty closely.
>
> The only reference I could find to reporting bugs is bugs@gnus.org; is
> that current?  Is there really no web-based bug tracking system?

Generally we report bugs from within Emacs, using M-x report-emacs-bug.
You can see them online here:

https://debbugs.gnu.org/cgi/pkgreport.cgi?package=emacs

Yours,
Eric

Re: gnus-server-to-method crash on virtual server name in gnus-secondary-select-methods

[Greg Klanderman]

Hi Eric,

>>>>> On January 21, 2021 Eric Abrahamsen <eric@ericabrahamsen.net> wrote:

>> According to
>> 
>> https://www.gnu.org/software/emacs/manual/html_node/gnus/Servers-and-Methods.html
>> 
>> not only can you use virtual server names (strings) "wherever you
>> would normally use a select method", it explicitly states you can do
>> so in 'gnus-secondary-select-method' (presumably a typo that the final
>> 's' is missing).
>> 
>> So based on that alone, any code which loops over
>> gnus-secondary-select-methods taking car/cdr of elements is highly
>> suspicious.  The additional information I gave just demonstrated one
>> way to reach this problematic logic.

> I still can't believe this is the way it's supposed to be used. The code
> snippet you posted (where the error actually arises) has been that way
> since The Dawn of Time. I also don't see why you _would_ define a server
> via the *Server* buffer, and then put it again in
> `gnus-secondary-select-methods'. Generally you define servers either in
> one place or the other -- if you've done it with "a" in the *Server*
> buffer, there's no need for it to appear in your config files, too.

Ahh I had not realized that I only needed it in one place, but it
makes total sense.  Somehow I took the bit of manual I linked above to
mean I should create the server in the server buffer, then put its
name in gnus-secondary-select-methods.

So maybe the real bug is just the documentation?

>>> I have left your other asides aside!
>> 
>> No problem.. probably best to post those separately once I find the
>> right forum.
>> 
>>> While this is a fine place to raise general Gnus questions/issues
>>> (the gnus.general group would be another option),
>> 
>> I think you mean 'gmane.emacs.gnus.general'?  It is hard to tell what
>> information on
>> 
>> https://www.gnu.org/software/emacs/manual/html_node/gnus/Gnus-Development.html
>> 
>> and
>> 
>> https://www.gnus.org/resources.html
>> 
>> is up-to-date; some links are clearly dead.
>> 
>> Is that newsgroup still bi-directionally gatewayed with ding@gnus.org?

> Yes, sorry, I did mean gmane.emacs.gnus.general, and yes that's still
> gatewayed to ding@gnus.org. So far as I know there's just
> gmane.emacs.gnus.general and gmane.emacs.gnus.user. I don't think
> there's any real difference anymore (probably "general" used to be more
> for development?) but you do get different people responding in the
> different groups.

OK great, is there a gatewayed email list for gmane.emacs.gnus.user as
well?

>> Is there any way to browse archives on the web?

>> The only reference I could find to reporting bugs is bugs@gnus.org; is
>> that current?  Is there really no web-based bug tracking system?

> Generally we report bugs from within Emacs, using M-x report-emacs-bug.
> You can see them online here:

> https://debbugs.gnu.org/cgi/pkgreport.cgi?package=emacs

Great, thank you.  I'm a bit wary of M-x report-emacs-bug from work as
I suspect it would run afoul of our security policy.

I spent quite a bit of time last week converting some more low-hanging
fruit of my xemacs config, and turning off new behaviors that
disagreed with me, so I've got Emacs in a bit more tolerable state for
editing.

My biggest concern now with fully transitioning, other than a lot more
time, is how slow it is over ssh forwarded X11.  As I said xemacs is
perfectly snappy, but Emacs is taking sometimes 30-60+ sec just to
create a new frame.

I turned off tooltips which seemed to be causing much of the latency
issues, then it seemed that toolbars & menubars were the issue because
after dragging another window over an Emacs frame, everything would
redraw immediately but the menubars and toolbars, which could again
take 30-60+ seconds with Emacs being essentially frozen to input.
Switching gtk to lucid (Debian testing) did not make appreciable
difference.

I've now noticed that the problem only occurs when a frame of
an Emacs is dragged over another frame of the same Emacs, so I suspect
some problem with the event handling loop.  I will submit a bug
report; this is perfectly reproducible with emacs -Q after ssh'ing
from my work laptop (on home network) to my work desktop (in office
30mi away) and then back to my personal home desktop, even with
tooltips/tool bars/menu bars/scroll bars off.

And then I'll get back to Gnus..

thank you,
Greg

Re: gnus-server-to-method crash on virtual server name in gnus-secondary-select-methods

[Eric Abrahamsen]

Greg Klanderman <gak@klanderman.net> writes:

> Hi Eric,
>
>>>>>> On January 21, 2021 Eric Abrahamsen <eric@ericabrahamsen.net> wrote:
>
>>> According to
>>> 
>>> https://www.gnu.org/software/emacs/manual/html_node/gnus/Servers-and-Methods.html
>>> 
>>> not only can you use virtual server names (strings) "wherever you
>>> would normally use a select method", it explicitly states you can do
>>> so in 'gnus-secondary-select-method' (presumably a typo that the final
>>> 's' is missing).
>>> 
>>> So based on that alone, any code which loops over
>>> gnus-secondary-select-methods taking car/cdr of elements is highly
>>> suspicious.  The additional information I gave just demonstrated one
>>> way to reach this problematic logic.
>
>> I still can't believe this is the way it's supposed to be used. The code
>> snippet you posted (where the error actually arises) has been that way
>> since The Dawn of Time. I also don't see why you _would_ define a server
>> via the *Server* buffer, and then put it again in
>> `gnus-secondary-select-methods'. Generally you define servers either in
>> one place or the other -- if you've done it with "a" in the *Server*
>> buffer, there's no need for it to appear in your config files, too.
>
> Ahh I had not realized that I only needed it in one place, but it
> makes total sense.  Somehow I took the bit of manual I linked above to
> mean I should create the server in the server buffer, then put its
> name in gnus-secondary-select-methods.
>
> So maybe the real bug is just the documentation?

I think so, I'm not aware of any code that is prepared to find a plain
string inside `gnus-secondary-select-methods'. Maybe I'll open a bug for
this just to make sure.

>>>> I have left your other asides aside!
>>> 
>>> No problem.. probably best to post those separately once I find the
>>> right forum.
>>> 
>>>> While this is a fine place to raise general Gnus questions/issues
>>>> (the gnus.general group would be another option),
>>> 
>>> I think you mean 'gmane.emacs.gnus.general'?  It is hard to tell what
>>> information on
>>> 
>>> https://www.gnu.org/software/emacs/manual/html_node/gnus/Gnus-Development.html
>>> 
>>> and
>>> 
>>> https://www.gnus.org/resources.html
>>> 
>>> is up-to-date; some links are clearly dead.
>>> 
>>> Is that newsgroup still bi-directionally gatewayed with ding@gnus.org?
>
>> Yes, sorry, I did mean gmane.emacs.gnus.general, and yes that's still
>> gatewayed to ding@gnus.org. So far as I know there's just
>> gmane.emacs.gnus.general and gmane.emacs.gnus.user. I don't think
>> there's any real difference anymore (probably "general" used to be more
>> for development?) but you do get different people responding in the
>> different groups.
>
> OK great, is there a gatewayed email list for gmane.emacs.gnus.user as
> well?

I don't think so, no.

>>> Is there any way to browse archives on the web?
>
>>> The only reference I could find to reporting bugs is bugs@gnus.org; is
>>> that current?  Is there really no web-based bug tracking system?
>
>> Generally we report bugs from within Emacs, using M-x report-emacs-bug.
>> You can see them online here:
>
>> https://debbugs.gnu.org/cgi/pkgreport.cgi?package=emacs
>
> Great, thank you.  I'm a bit wary of M-x report-emacs-bug from work as
> I suspect it would run afoul of our security policy.
>
> I spent quite a bit of time last week converting some more low-hanging
> fruit of my xemacs config, and turning off new behaviors that
> disagreed with me, so I've got Emacs in a bit more tolerable state for
> editing.
>
> My biggest concern now with fully transitioning, other than a lot more
> time, is how slow it is over ssh forwarded X11.  As I said xemacs is
> perfectly snappy, but Emacs is taking sometimes 30-60+ sec just to
> create a new frame.
>
> I turned off tooltips which seemed to be causing much of the latency
> issues, then it seemed that toolbars & menubars were the issue because
> after dragging another window over an Emacs frame, everything would
> redraw immediately but the menubars and toolbars, which could again
> take 30-60+ seconds with Emacs being essentially frozen to input.
> Switching gtk to lucid (Debian testing) did not make appreciable
> difference.
>
> I've now noticed that the problem only occurs when a frame of
> an Emacs is dragged over another frame of the same Emacs, so I suspect
> some problem with the event handling loop.  I will submit a bug
> report; this is perfectly reproducible with emacs -Q after ssh'ing
> from my work laptop (on home network) to my work desktop (in office
> 30mi away) and then back to my personal home desktop, even with
> tooltips/tool bars/menu bars/scroll bars off.
>
> And then I'll get back to Gnus..

We'll be here :)

Re: gnus-server-to-method crash on virtual server name in gnus-secondary-select-methods

[Greg Klanderman]

>>>>> On January 25, 2021 Eric Abrahamsen <eric@ericabrahamsen.net> wrote:

>> So maybe the real bug is just the documentation?

> I think so, I'm not aware of any code that is prepared to find a plain
> string inside `gnus-secondary-select-methods'. Maybe I'll open a bug for
> this just to make sure.

I can open one as well when I get back around to it.

>> And then I'll get back to Gnus..

> We'll be here :)

sounds good, thank you!

Greg

Re: gnus-server-to-method crash on virtual server name in gnus-secondary-select-methods

[Robert Pluim]

[-- Attachment #1: Type: text/plain, Size: 1554 bytes --]

>>>>> On Mon, 25 Jan 2021 12:51:04 -0500, Greg Klanderman <gak@klanderman.net> said:

    Greg> My biggest concern now with fully transitioning, other than a lot more
    Greg> time, is how slow it is over ssh forwarded X11.  As I said xemacs is
    Greg> perfectly snappy, but Emacs is taking sometimes 30-60+ sec just to
    Greg> create a new frame.

Does the attached patch improve frame creation?

    Greg> I turned off tooltips which seemed to be causing much of the latency
    Greg> issues, then it seemed that toolbars & menubars were the issue because
    Greg> after dragging another window over an Emacs frame, everything would
    Greg> redraw immediately but the menubars and toolbars, which could again
    Greg> take 30-60+ seconds with Emacs being essentially frozen to input.
    Greg> Switching gtk to lucid (Debian testing) did not make appreciable
    Greg> difference.

    Greg> I've now noticed that the problem only occurs when a frame of
    Greg> an Emacs is dragged over another frame of the same Emacs, so I suspect
    Greg> some problem with the event handling loop.  I will submit a bug
    Greg> report; this is perfectly reproducible with emacs -Q after ssh'ing
    Greg> from my work laptop (on home network) to my work desktop (in office
    Greg> 30mi away) and then back to my personal home desktop, even with
    Greg> tooltips/tool bars/menu bars/scroll bars off.

Which X server are you using? I donʼt see this at all (but admittedly
the two machines in question are a meter apart)


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: check_x_display_less.patch --]
[-- Type: text/x-diff, Size: 593 bytes --]

diff --git a/src/xterm.c b/src/xterm.c
index 2e0407aff4..448e06eb8b 100644
--- a/src/xterm.c
+++ b/src/xterm.c
@@ -12709,8 +12709,6 @@ x_term_init (Lisp_Object display_name, char *xrm_option, char *resource_name)
       ++x_initialized;
     }
 
-  if (! x_display_ok (SSDATA (display_name)))
-    error ("Display %s can't be opened", SSDATA (display_name));
 
 #ifdef USE_GTK
   {
@@ -12838,6 +12836,7 @@ #define NUM_ARGV 10
   /* Detect failure.  */
   if (dpy == 0)
     {
+      error ("Display %s can't be opened", SSDATA (display_name));
       unblock_input ();
       return 0;
     }

slow X11 frame creation and refresh after occlusion (was: gnus-server-to-method crash on virtual server name in gnus-secondary-select-methods)

[Greg Klanderman]

Hi Robert,

Thank you for your response!


>>>>> On January 26, 2021 Robert Pluim <rpluim@gmail.com> wrote:

>>>>> On Mon, 25 Jan 2021 12:51:04 -0500, Greg Klanderman <gak@klanderman.net> said:
Greg> My biggest concern now with fully transitioning, other than a lot more
Greg> time, is how slow it is over ssh forwarded X11.  As I said xemacs is
Greg> perfectly snappy, but Emacs is taking sometimes 30-60+ sec just to
Greg> create a new frame.

> Does the attached patch improve frame creation?

Let me get back to you.. I did clone the git repository last week but
have not gotten around to building yet.. I did just look and find the
building guide so hopefully get to that in the next couple days..

But IIUC the patch removes one of two XOpenDisplay() calls, so at best
would halve the latency in opening a new frame, so I can't see it
completely solving the problem.

Also, shouldn't the opening of the display only take place for the
first frame on that display?

Greg> I turned off tooltips which seemed to be causing much of the latency
Greg> issues, then it seemed that toolbars & menubars were the issue because
Greg> after dragging another window over an Emacs frame, everything would
Greg> redraw immediately but the menubars and toolbars, which could again
Greg> take 30-60+ seconds with Emacs being essentially frozen to input.
Greg> Switching gtk to lucid (Debian testing) did not make appreciable
Greg> difference.

Greg> I've now noticed that the problem only occurs when a frame of
Greg> an Emacs is dragged over another frame of the same Emacs, so I suspect
Greg> some problem with the event handling loop.  I will submit a bug
Greg> report; this is perfectly reproducible with emacs -Q after ssh'ing
Greg> from my work laptop (on home network) to my work desktop (in office
Greg> 30mi away) and then back to my personal home desktop, even with
Greg> tooltips/tool bars/menu bars/scroll bars off.

> Which X server are you using? I don?t see this at all (but admittedly
> the two machines in question are a meter apart)

Yeah I do not see it between my work laptop and home desktop plugged
into the same switch (I am working fully remote, from home).

Displaying from my work desktop (in office 30 mi away) to work laptop
on my home network it is very obvious, and even more so displaying
from my home desktop via work desktop and back onto my work laptop
(again, on my home network).

My home desktop is running Debian testing, and the work laptop and
desktop are running Debian testing with work's patches applied.

Generally I am running in "old-school" mode, without a desktop
environment, using fvwm2 2.6.9 + a few local patches, started via
startx from a terminal login.

I will check if I see the issue using the default desktop environment
(I think cinnamon is standard on the work laptop).. and get back to
you on that as well.

X server is whatever is default on Debian testing, let me know if
there is some command you want me to run to determine any specific
details..

thanks,
Greg


> diff --git a/src/xterm.c b/src/xterm.c
> index 2e0407aff4..448e06eb8b 100644
> --- a/src/xterm.c
> +++ b/src/xterm.c
> @@ -12709,8 +12709,6 @@ x_term_init (Lisp_Object display_name, char *xrm_option, char *resource_name)
>        ++x_initialized;
>      }

> -  if (! x_display_ok (SSDATA (display_name)))
> -    error ("Display %s can't be opened", SSDATA (display_name));

>  #ifdef USE_GTK
>    {
> @@ -12838,6 +12836,7 @@ #define NUM_ARGV 10
>    /* Detect failure.  */
>    if (dpy == 0)
>      {
> +      error ("Display %s can't be opened", SSDATA (display_name));
>        unblock_input ();
>        return 0;
>      }

Re: slow X11 frame creation and refresh after occlusion

[Robert Pluim]

>>>>> On Tue, 26 Jan 2021 14:09:03 -0500, Greg Klanderman <gak@klanderman.net> said:
    >> Does the attached patch improve frame creation?

    Greg> Let me get back to you.. I did clone the git repository last week but
    Greg> have not gotten around to building yet.. I did just look and find the
    Greg> building guide so hopefully get to that in the next couple days..

    Greg> But IIUC the patch removes one of two XOpenDisplay() calls, so at best
    Greg> would halve the latency in opening a new frame, so I can't see it
    Greg> completely solving the problem.

    Greg> Also, shouldn't the opening of the display only take place for the
    Greg> first frame on that display?

Hmm, I think youʼre right.

    >> Which X server are you using? I don?t see this at all (but admittedly
    >> the two machines in question are a meter apart)

    Greg> I will check if I see the issue using the default desktop environment
    Greg> (I think cinnamon is standard on the work laptop).. and get back to
    Greg> you on that as well.

    Greg> X server is whatever is default on Debian testing, let me know if
    Greg> there is some command you want me to run to determine any specific
    Greg> details..

Other things to check:

- which font backend are you using? (frame-parameter nil
  'font-backend) will tell you
- which version of emacs? emacs-master has a change to reduce the
  number of fonts it probes, which is always a killer over a forwarded
  X connection

Robert

Re: slow X11 frame creation and refresh after occlusion

[Greg Klanderman]

Thanks Robert, no time yet for much investigation on my side, but I
can answer your followup questions quickly..

>>>>> On January 27, 2021 Robert Pluim <rpluim@gmail.com> wrote:

> Other things to check:

> - which font backend are you using? (frame-parameter nil
>   'font-backend) will tell you

(frame-parameter nil 'font-backend)
(xfthb x)

any suggestions for settings I can try?

> - which version of emacs? emacs-master has a change to reduce the
>   number of fonts it probes, which is always a killer over a forwarded
>   X connection

didn't find it searching 'git log' a bit can you point me to it?

I assume not in this one:

(emacs-version)
"GNU Emacs 27.1 (build 1, x86_64-pc-linux-gnu, X toolkit, Xaw3d scroll bars)
 of 2020-12-21, modified by Debian"

system-configuration-options
"--build x86_64-linux-gnu
--prefix=/usr
--sharedstatedir=/var/lib
--libexecdir=/usr/lib
--localstatedir=/var/lib
--infodir=/usr/share/info
--mandir=/usr/share/man
--enable-libsystemd
--with-pop=yes
--enable-locallisppath=/etc/emacs:/usr/local/share/emacs/27.1/site-lisp:/usr/local/share/emacs/site-lisp:/usr/share/emacs/27.1/site-lisp:/usr/share/emacs/site-lisp
--with-sound=alsa
--without-gconf
--with-mailutils
--build x86_64-linux-gnu
--prefix=/usr
--sharedstatedir=/var/lib
--libexecdir=/usr/lib
--localstatedir=/var/lib
--infodir=/usr/share/info
--mandir=/usr/share/man
--enable-libsystemd
--with-pop=yes
--enable-locallisppath=/etc/emacs:/usr/local/share/emacs/27.1/site-lisp:/usr/local/share/emacs/site-lisp:/usr/share/emacs/27.1/site-lisp:/usr/share/emacs/site-lisp
--with-sound=alsa
--without-gconf
--with-mailutils
--with-x=yes
--with-x-toolkit=lucid
--with-toolkit-scroll-bars
--without-gsettings
'CFLAGS=-g -O2 -fdebug-prefix-map=/build/emacs-uC3JdZ/emacs-27.1+1=. -fstack-protector-strong -Wformat -Werror=format-security -Wall'
'CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=2'
LDFLAGS=-Wl,-z,relro"

yikes, typing M-x replace-string RET the first time takes 30 sec
before prompting "Replace string: " in the minibuffer!  Again over
ssh-forwarded X11 (but only one hop, from work to home).

Greg

Re: slow X11 frame creation and refresh after occlusion

[Robert Pluim]

>>>>> On Sat, 30 Jan 2021 14:32:28 -0500, Greg Klanderman <gak@klanderman.net> said:

    Greg> Thanks Robert, no time yet for much investigation on my side, but I
    Greg> can answer your followup questions quickly..

>>>>> On January 27, 2021 Robert Pluim <rpluim@gmail.com> wrote:

    >> Other things to check:

    >> - which font backend are you using? (frame-parameter nil
    >> 'font-backend) will tell you

    Greg> (frame-parameter nil 'font-backend)
    Greg> (xfthb x)

    Greg> any suggestions for settings I can try?

Configure '--with-cairo'. Iʼm hoping that will be more efficient in
terms of loading fonts.

    >> - which version of emacs? emacs-master has a change to reduce the
    >> number of fonts it probes, which is always a killer over a forwarded
    >> X connection

    Greg> didn't find it searching 'git log' a bit can you point me to it?

    Greg> I assume not in this one:

    Greg> (emacs-version)
    Greg> "GNU Emacs 27.1 (build 1, x86_64-pc-linux-gnu, X toolkit, Xaw3d scroll bars)
    Greg>  of 2020-12-21, modified by Debian"

Definitely not in there, itʼs master-only:

$ git log --author rpluim@gmail.com --grep font

    commit 20d13e424fb2e7dcc5e6ea1848bca4376d22bab1
    Author: Robert Pluim <rpluim@gmail.com>
    Date:   Fri Sep 11 10:29:24 2020 +0200

        Stop querying for fonts as soon as a match is found

Re: slow X11 frame creation and refresh after occlusion

[Greg Klanderman]

>>>>> On February 1, 2021 Robert Pluim <rpluim@gmail.com> wrote:

Greg> (frame-parameter nil 'font-backend)
Greg> (xfthb x)

Greg> any suggestions for settings I can try?

> Configure '--with-cairo'. I'm hoping that will be more efficient in
> terms of loading fonts.

OK I will have a look.  Is there any way to determine if font loading
is causing significant delay?  And would that be an issue on
subsequent to the first frame on a display?

Greg> (emacs-version)
Greg> "GNU Emacs 27.1 (build 1, x86_64-pc-linux-gnu, X toolkit, Xaw3d scroll bars)
Greg> of 2020-12-21, modified by Debian"

> Definitely not in there, it's master-only:

> $ git log --author rpluim@gmail.com --grep font

>     commit 20d13e424fb2e7dcc5e6ea1848bca4376d22bab1
>     Author: Robert Pluim <rpluim@gmail.com>
>     Date:   Fri Sep 11 10:29:24 2020 +0200

>         Stop querying for fonts as soon as a match is found

OK, another good pointer.

Did you have any ideas on my other email, and the finding that the
30-60 sec of lossage I am seeing when dragging one emacs frame over
another seems related to not running a full blown desktop environment?

Have to see if I can trace the xevents and find anything interesting..

thank you,
Greg

Re: slow X11 frame creation and refresh after occlusion

[Robert Pluim]

>>>>> On Wed, 03 Feb 2021 16:52:15 -0500, Greg Klanderman <gak@klanderman.net> said:

>>>>> On February 1, 2021 Robert Pluim <rpluim@gmail.com> wrote:
    Greg> (frame-parameter nil 'font-backend)
    Greg> (xfthb x)

    Greg> any suggestions for settings I can try?

    >> Configure '--with-cairo'. I'm hoping that will be more efficient in
    >> terms of loading fonts.

    Greg> OK I will have a look.  Is there any way to determine if font loading
    Greg> is causing significant delay?  And would that be an issue on
    Greg> subsequent to the first frame on a display?

Emacs does a bunch of font-related stuff every time you create a new
graphical frame. You could try running emacs under 'perf' to see if it
gives any insight.

    Greg> (emacs-version)
    Greg> "GNU Emacs 27.1 (build 1, x86_64-pc-linux-gnu, X toolkit, Xaw3d scroll bars)
    Greg> of 2020-12-21, modified by Debian"

    >> Definitely not in there, it's master-only:

    >> $ git log --author rpluim@gmail.com --grep font

    >> commit 20d13e424fb2e7dcc5e6ea1848bca4376d22bab1
    >> Author: Robert Pluim <rpluim@gmail.com>
    >> Date:   Fri Sep 11 10:29:24 2020 +0200

    >> Stop querying for fonts as soon as a match is found

    Greg> OK, another good pointer.

    Greg> Did you have any ideas on my other email, and the finding that the
    Greg> 30-60 sec of lossage I am seeing when dragging one emacs frame over
    Greg> another seems related to not running a full blown desktop environment?

That one I have no idea about.

Robert

Re: slow X11 frame creation and refresh after occlusion

[Greg Klanderman]

>>>>> On February 4, 2021 Robert Pluim <rpluim@gmail.com> wrote:

>>>>> On Wed, 03 Feb 2021 16:52:15 -0500, Greg Klanderman <gak@klanderman.net> said:
>>>>> On February 1, 2021 Robert Pluim <rpluim@gmail.com> wrote:
Greg> (frame-parameter nil 'font-backend)
Greg> (xfthb x)

Greg> any suggestions for settings I can try?

>>> Configure '--with-cairo'. I'm hoping that will be more efficient in
>>> terms of loading fonts.

Greg> OK I will have a look.  Is there any way to determine if font loading
Greg> is causing significant delay?  And would that be an issue on
Greg> subsequent to the first frame on a display?

> Emacs does a bunch of font-related stuff every time you create a new
> graphical frame. You could try running emacs under 'perf' to see if it
> gives any insight.

OK I built from git, --with-cairo (interestingly configure does not
fail if you specify --with-cairo but don't have the libraries
installed) and am able to reproduce the issues I was seeing.

It looks like xterm.c has most of the X event handling, so maybe I
will try to put in some debug prints to see what's going on, or try
one of the X event tracing programs, or 'perf' next..

thanks,
Greg

Re: slow X11 frame creation and refresh after occlusion

[Robert Pluim]

>>>>> On Thu, 04 Feb 2021 16:14:48 -0500, Greg Klanderman <gak@klanderman.net> said:

>>>>> On February 4, 2021 Robert Pluim <rpluim@gmail.com> wrote:
>>>>> On Wed, 03 Feb 2021 16:52:15 -0500, Greg Klanderman <gak@klanderman.net> said:
>>>>> On February 1, 2021 Robert Pluim <rpluim@gmail.com> wrote:
    Greg> (frame-parameter nil 'font-backend)
    Greg> (xfthb x)

    Greg> any suggestions for settings I can try?

    >>>> Configure '--with-cairo'. I'm hoping that will be more efficient in
    >>>> terms of loading fonts.

    Greg> OK I will have a look.  Is there any way to determine if font loading
    Greg> is causing significant delay?  And would that be an issue on
    Greg> subsequent to the first frame on a display?

    >> Emacs does a bunch of font-related stuff every time you create a new
    >> graphical frame. You could try running emacs under 'perf' to see if it
    >> gives any insight.

    Greg> OK I built from git, --with-cairo (interestingly configure does not
    Greg> fail if you specify --with-cairo but don't have the libraries
    Greg> installed) and am able to reproduce the issues I was seeing.

Itʼs an option, which means cairo is...optional :-)

(I canʼt think offhand of an emacs configure option that causes
configure to fail if itʼs not satisfied. Someone will now point one
out).

    Greg> It looks like xterm.c has most of the X event handling, so maybe I
    Greg> will try to put in some debug prints to see what's going on, or try
    Greg> one of the X event tracing programs, or 'perf' next..

Good luck. You might want to experiment with
    emacs -xrm "emacs.synchronous: true"
That would make tracing stuff predictably easier.

Robert

Re: slow X11 frame creation and refresh after occlusion

[Greg Klanderman]

>>>>> On February 5, 2021 Robert Pluim <rpluim@gmail.com> wrote:

>>>>> On Thu, 04 Feb 2021 16:14:48 -0500, Greg Klanderman <gak@klanderman.net> said:
>>>>> On February 4, 2021 Robert Pluim <rpluim@gmail.com> wrote:
>>>>> On Wed, 03 Feb 2021 16:52:15 -0500, Greg Klanderman <gak@klanderman.net> said:
>>>>> On February 1, 2021 Robert Pluim <rpluim@gmail.com> wrote:
Greg> (frame-parameter nil 'font-backend)
Greg> (xfthb x)

Greg> any suggestions for settings I can try?

>>>>> Configure '--with-cairo'. I'm hoping that will be more efficient in
>>>>> terms of loading fonts.

Greg> OK I will have a look.  Is there any way to determine if font loading
Greg> is causing significant delay?  And would that be an issue on
Greg> subsequent to the first frame on a display?

>>> Emacs does a bunch of font-related stuff every time you create a new
>>> graphical frame. You could try running emacs under 'perf' to see if it
>>> gives any insight.

Greg> OK I built from git, --with-cairo (interestingly configure does not
Greg> fail if you specify --with-cairo but don't have the libraries
Greg> installed) and am able to reproduce the issues I was seeing.

> It?s an option, which means cairo is...optional :-)

Oh, I thought that when you explicitly had a --with-FOO option, it
made it required...

> (I can?t think offhand of an emacs configure option that causes
> configure to fail if it?s not satisfied. Someone will now point one
> out).

I had one that was behaving like that, --with-sound=alsa I think.
I realize that's a little different than a boolean option though.

Greg> It looks like xterm.c has most of the X event handling, so maybe I
Greg> will try to put in some debug prints to see what's going on, or try
Greg> one of the X event tracing programs, or 'perf' next..

> Good luck. You might want to experiment with
>     emacs -xrm "emacs.synchronous: true"
> That would make tracing stuff predictably easier.

OK I will try that resource.

So far I managed to build xtruss and get it working after recursively
debugging a crash where it didn't detect an error opening DISPLAY :10
(which an ssh session already had open).  Thankfully strace quickly
showed the problem and I was able to patch it.

So I have two X event traces for emacs, one under fvwn where it has
the problem and one under cinnamon DE that does not, but have not
gotten far into trying to analyze the differences.  After collecting
the cinnamon DE trace, my laptop wedged and I had to reboot it, it
sometimes does that when VT switching.

But maybe I should re-do with synchronous=true first..

Anyway just based on size there is a significant difference:

% wc -l /tmp/xtruss.emacs.*
  45442 /tmp/xtruss.emacs.bad
   7625 /tmp/xtruss.emacs.good
  53067 total

Both consist of running emacs, moving the frame aside (so second frame
will not be on top), then C-x 5 2, waiting for emacs to be responsive,
then dragging the new frame over the first then leaving it off to the
side, and waiting for emacs to be responsive to input again.  The
final step taking 30-60 sec in the bad case.

Will update if I find anything.

Greg

Re: slow X11 frame creation and refresh after occlusion

[Greg Klanderman]

>>>>> On January 27, 2021 Robert Pluim <rpluim@gmail.com> wrote:

>>> Which X server are you using? I don?t see this at all (but admittedly
>>> the two machines in question are a meter apart)

Greg> I will check if I see the issue using the default desktop environment
Greg> (I think cinnamon is standard on the work laptop).. and get back to
Greg> you on that as well.

Hmmm, well I just checked with the standard gdm3 / Xorg server /
cinnamon DE environment and emacs works perfectly.. creating a new
frame is still a little slow, but nothing like I was seeing, and
dragging frames of the same emacs over each other does not even result
in any perceptible redisplay latency.  I really don't have any idea
what an application like emacs would "see" differently without a
DE... any ideas how to figure out what's going on?

thanks,
Greg