bug#71452: 30.0.50; GNUS hangs on IMAP/s reads starting at commit bbc1803

bug#71452: 30.0.50; GNUS hangs on IMAP/s reads starting at commit bbc1803

[epg]

1. Configure GNUS like so:

(setq gnus-select-method '(nnnil "")
      gnus-secondary-select-methods
      '(
        ;; machine imap.fastmail.com login epg@gillespies.us
        (nnimap "imap.fastmail.com"
                (nnimap-stream ssl)
                (nnimap-record-commands t)
                )
        )
      )

2. Start GNUS, open some folders and messages.

It's not clear to me exactly what's going on; sometimes the hang
happens upon opening a folder, sometimes opening a message.

git bisect shows that commit
bbc18031aff6f22a1f2b63355f18f294fbdeb797 introduced the bug.



In GNU Emacs 30.0.50 (build 3, x86_64-pc-linux-gnu, X toolkit,
 cairo version 1.18.0) of 2024-06-08 built on dogato
Repository revision: 4468216f17aca811c02b49d18c5bee4fb9fd6943
Repository branch: master
Windowing system distributor 'The X.Org Foundation', version 11.0.12101012
System Description: openSUSE Tumbleweed

Configured using:
 'configure -C --prefix=/opt/emacs-30.0.50.2.4468216f17a
 --disable-silent-rules --with-x-toolkit=athena --without-sqlite3
 --without-lcms2 --without-libsystemd --without-imagemagick
 --with-json --with-tree-sitter --without-xft --without-libotf
 --without-toolkit-scroll-bars --without-xaw3d --without-gpm
 --without-dbus --without-gconf --without-gsettings
 --without-selinux --without-compress-install --with-x'

Configured features:
CAIRO FREETYPE GIF GLIB GMP GNUTLS HARFBUZZ JPEG LIBXML2 MODULES
NOTIFY INOTIFY PDUMPER PNG RSVG SECCOMP SOUND THREADS TIFF
TREE_SITTER WEBP X11 XDBE XIM XINPUT2 XPM LUCID ZLIB

Important settings:
  value of $LC_CTYPE: en_US.UTF-8
  value of $XMODIFIERS: @im=local
  locale-coding-system: utf-8-unix

Major mode: Group

Minor modes in effect:
  gnus-undo-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  minibuffer-regexp-mode: t
  buffer-read-only: t
  line-number-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t

Load-path shadows:
None found.

Features:
(shadow emacsbug mule-util cl-extra sort gnus-cite shr-color
color mm-archive mail-extr textsec uni-scripts idna-mapping
ucs-normalize uni-confusable textsec-check gnus-async gnus-bcklg
qp gnus-ml disp-table nndraft nnmh utf-7 network-stream nsm
nnfolder nnnil gnus-agent gnus-srvr gnus-score score-mode
nnvirtual gnus-msg gnus-art mm-uu mml2015 mm-view mml-smime smime
gnutls dig nntp gnus-cache gnus-sum gnus-group gnus-undo
gnus-start gnus-dbus dbus gnus-cloud nnimap nnmail mail-source
utf7 nnoo parse-time iso8601 gnus-spec gnus-int gnus-range
gnus-win gnus nnheader range elfeed-show elfeed-search wid-edit
bookmark message sendmail yank-media dired dired-loaddefs rfc822
mml mml-sec epa epg rfc6068 epg-config gnus-util mm-decode
mm-bodies mm-encode mail-parse rfc2231 rfc2047 rfc2045 mm-util
ietf-drums mail-prsvr mailabbrev mail-utils gmm-utils mailheader
shr pixel-fill kinsoku puny svg dom elfeed-csv elfeed elfeed-curl
elfeed-log elfeed-db elfeed-lib time-date avl-tree generator
url-queue url-file browse-url xml-query xml rust-utils rust-mode
derived rust-rustfmt rust-playpen rust-compile rust-cargo eglot
external-completion jsonrpc xref flymake thingatpt project compat
diff ert pp ewoc debug backtrace help-mode find-func filenotify
warnings compile text-property-search imenu ob-sql ob ob-tangle
ol rx org-src ob-ref ob-lob ob-table ob-exp ob-comint comint
ansi-osc ansi-color ring ob-core org-cycle org-fold org-fold-core
ob-eval org-keys oc org-compat org-macs format-spec pcase vc-git
diff-mode track-changes easy-mmode vc-dispatcher url-cache url
url-proxy url-privacy url-expand url-methods url-history
url-cookie generate-lisp-file url-domsuf mailcap url-util
url-parse auth-source cl-seq eieio eieio-core cl-macs icons
cl-loaddefs cl-lib password-cache json subr-x map byte-opt gv
bytecomp byte-compile url-vars rmc iso-transl tooltip cconv eldoc
paren electric uniquify ediff-hook vc-hooks lisp-float-type
elisp-mode mwheel term/x-win x-win term/common-win x-dnd
touch-screen tool-bar dnd fontset image regexp-opt fringe
tabulated-list replace newcomment text-mode lisp-mode prog-mode
register page tab-bar menu-bar rfn-eshadow isearch easymenu timer
select scroll-bar mouse jit-lock font-lock syntax font-core
term/tty-colors frame minibuffer nadvice seq simple cl-generic
indonesian philippine cham georgian utf-8-lang misc-lang
vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms
cp51932 hebrew greek romanian slovak czech european ethiopic
indian cyrillic chinese composite emoji-zwj charscript charprop
case-table epa-hook jka-cmpr-hook help abbrev obarray oclosure
cl-preloaded button loaddefs theme-loaddefs faces cus-face
macroexp files window text-properties overlay sha1 md5 base64
format env code-pages mule custom widget keymap
hashtable-print-readable backquote threads inotify
dynamic-setting font-render-setting cairo x-toolkit xinput2 x
multi-tty move-toolbar make-network-process emacs)

Memory information:
((conses 16 356352 45636) (symbols 48 22166 7) (strings 32 71942 3853)
 (string-bytes 1 1769693) (vectors 16 60022) (vector-slots 8 675472 56298)
 (floats 8 406 13148) (intervals 56 451 94) (buffers 992 21))

bug#71452: 30.0.50; GNUS hangs on IMAP/s reads starting at commit bbc1803

[Eli Zaretskii]

> From: epg@pretzelnet.org
> Date: Sun, 09 Jun 2024 11:36:30 -0500
> 
> 
> 1. Configure GNUS like so:
> 
> (setq gnus-select-method '(nnnil "")
>       gnus-secondary-select-methods
>       '(
>         ;; machine imap.fastmail.com login epg@gillespies.us
>         (nnimap "imap.fastmail.com"
>                 (nnimap-stream ssl)
>                 (nnimap-record-commands t)
>                 )
>         )
>       )
> 
> 2. Start GNUS, open some folders and messages.
> 
> It's not clear to me exactly what's going on; sometimes the hang
> happens upon opening a folder, sometimes opening a message.
> 
> git bisect shows that commit
> bbc18031aff6f22a1f2b63355f18f294fbdeb797 introduced the bug.

Commit 1f407e286c was supposed to fix that.  Did you try updating from
Git?

bug#71452: 30.0.50; GNUS hangs on IMAP/s reads starting at commit bbc1803

[epg]

Eli Zaretskii <eliz@gnu.org> writes:

> Commit 1f407e286c was supposed to fix that.  Did you try updating from
> Git?

I'm sorry I wasn't clear:  the problem is present in master as of
0aed6f5f0d9319dc616424e2c7df80580a04c739.  I was reporting the
first commit to show the bug, not the latest.

Thanks!

-- 
Eric Gillespie  <*>  epg@pretzelnet.org

bug#71452: 30.0.50; GNUS hangs on IMAP/s reads starting at commit bbc1803

[Eli Zaretskii]

> From: epg@pretzelnet.org
> Cc: 71452@debbugs.gnu.org
> Date: Sun, 09 Jun 2024 13:56:28 -0500
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Commit 1f407e286c was supposed to fix that.  Did you try updating from
> > Git?
> 
> I'm sorry I wasn't clear:  the problem is present in master as of
> 0aed6f5f0d9319dc616424e2c7df80580a04c739.  I was reporting the
> first commit to show the bug, not the latest.

Does it help to set read-process-output-fast to a nil value?

bug#71452: 30.0.50; GNUS hangs on IMAP/s reads starting at commit bbc1803

[epg]

Eli Zaretskii <eliz@gnu.org> writes:

> Does it help to set read-process-output-fast to a nil value?

As far as I can tell, yes that does help!  I opened quite a few
folders and messages, way more than it took to cause a hang
earlier.  No hang!

Tested with 'emacs -Q' built from commit
0aed6f5f0d9319dc616424e2c7df80580a04c739 and with the following
configuration:

(setq
      read-process-output-fast nil
      gnus-select-method '(nnnil "")
      gnus-secondary-select-methods
      '(
        (nnimap "imap.fastmail.com"
                (nnimap-stream ssl)
                (nnimap-record-commands t)
                 ;; TODO expiry?!  (nnmail-expiry-target "nnimap+gmail:[Gmail]/Trash")
                )
        )
      )

Thanks!

-- 
Eric Gillespie  <*>  epg@pretzelnet.org

bug#71452: 30.0.50; GNUS hangs on IMAP/s reads starting at commit bbc1803

[Stefan Kangas]

epg@pretzelnet.org writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>> Does it help to set read-process-output-fast to a nil value?
>
> As far as I can tell, yes that does help!  I opened quite a few
> folders and messages, way more than it took to cause a hang
> earlier.  No hang!
>
> Tested with 'emacs -Q' built from commit
> 0aed6f5f0d9319dc616424e2c7df80580a04c739 and with the following
> configuration:
>
> (setq
>       read-process-output-fast nil
>       gnus-select-method '(nnnil "")
>       gnus-secondary-select-methods
>       '(
>         (nnimap "imap.fastmail.com"
>                 (nnimap-stream ssl)
>                 (nnimap-record-commands t)
>                  ;; TODO expiry?!  (nnmail-expiry-target "nnimap+gmail:[Gmail]/Trash")
>                 )
>         )
>       )
>
> Thanks!

Copying in Dmitry.

Not sure if it's relevant, but note also my recent commit
a1a4c6dccbe7aeeac97635df54797ebf6c0ce789.

bug#71452: 30.0.50; GNUS hangs on IMAP/s reads starting at commit bbc1803

[Dmitry Gutov]

On 09/06/2024 23:43, Stefan Kangas wrote:
> epg@pretzelnet.org  writes:
> 
>> Eli Zaretskii<eliz@gnu.org>  writes:
>>
>>> Does it help to set read-process-output-fast to a nil value?
>> As far as I can tell, yes that does help!  I opened quite a few
>> folders and messages, way more than it took to cause a hang
>> earlier.  No hang!
>>
>> Tested with 'emacs -Q' built from commit
>> 0aed6f5f0d9319dc616424e2c7df80580a04c739 and with the following
>> configuration:
>>
>> (setq
>>        read-process-output-fast nil
>>        gnus-select-method '(nnnil "")
>>        gnus-secondary-select-methods
>>        '(
>>          (nnimap "imap.fastmail.com"
>>                  (nnimap-stream ssl)
>>                  (nnimap-record-commands t)
>>                   ;; TODO expiry?!  (nnmail-expiry-target "nnimap+gmail:[Gmail]/Trash")
>>                  )
>>          )
>>        )
>>
>> Thanks!
> Copying in Dmitry.
> 
> Not sure if it's relevant, but note also my recent commit
> a1a4c6dccbe7aeeac97635df54797ebf6c0ce789.

Thank you.

OP, could you retest with the latest master and read-process-output-fast 
set to its default value (t)?

bug#71452: 30.0.50; GNUS hangs on IMAP/s reads starting at commit bbc1803

[epg]

Dmitry Gutov <dmitry@gutov.dev> writes:

> OP, could you retest with the latest master and read-process-output-fast set
> to its default value (t)?
>

As of 12d44fe6420e84eab8f750f9a0f8cd73c3e70bb2, still hangs with
read-process-output-fast t, works with it nil.

Thanks!

-- 
Eric Gillespie  <*>  epg@pretzelnet.org

bug#71452: 30.0.50; GNUS hangs on IMAP/s reads starting at commit bbc1803

[Dmitry Gutov]

[-- Attachment #1: Type: text/plain, Size: 1741 bytes --]

On 10/06/2024 00:30, epg@pretzelnet.org wrote:
> Dmitry Gutov<dmitry@gutov.dev>  writes:
> 
>> OP, could you retest with the latest master and read-process-output-fast set
>> to its default value (t)?
>>
> As of 12d44fe6420e84eab8f750f9a0f8cd73c3e70bb2, still hangs with
> read-process-output-fast t, works with it nil.

Okay, thank you. I can reproduce it too.

FWIW, it hangs here:

Lisp Backtrace:
"accept-process-output" (0xf19ff6a8)
"nnheader-accept-process-output" (0xf19ff610)
"nnimap-wait-for-response" (0xf19ff5a8)
"nnimap-retrieve-headers" (0xf19ff530)
"gnus-retrieve-headers" (0xf19ff4b8)
"gnus-cache-retrieve-headers" (0xf19ff450)
"gnus-retrieve-headers" (0xf19ff3d0)
"gnus-fetch-headers" (0xf19ff360)
"gnus-select-newsgroup" (0xf19ff2c0)
"gnus-summary-read-group-1" (0xf19ff228)
"gnus-summary-read-group" (0xf19ff188)
"gnus-group-read-group" (0xf19ff100)
--Type <RET> for more, q to quit, c to continue without paging--
"gnus-group-select-group" (0xffffdb10)
"funcall-interactively" (0xffffdb08)
"call-interactively" (0xf19ff070)
"command-execute" (0xffffddf8)

Commands can be different, but nnimap-wait-for-response is where it 
stops. And it's not edebug-able: instrumenting the function makes the 
bug go away.

Print-debugging seems to say that there might be something wrong with 
what (forward-line -1) does: output has the ^M chars, and apparently the 
new logic makes it skip over too many characters.

The only thing relevant I found is this bit, which for now I simply 
copied over from read_and_dispose_of_process_output. But the latter 
calls it after "decoding to string" yet before inserting the string into 
the buffer. Can we do this after the decode_coding_c_string call where 
dst_object is the target buffer?

[-- Attachment #2: Vlast_coding_system_used.diff --]
[-- Type: text/x-patch, Size: 2290 bytes --]

diff --git a/src/process.c b/src/process.c
index 9670be64279..420b4533cc6 100644
--- a/src/process.c
+++ b/src/process.c
@@ -6372,6 +6372,5 @@
     {			/* We have to decode the input.  */
       Lisp_Object curbuf;
-      int carryover = 0;
       specpdl_ref count1 = SPECPDL_INDEX ();
 
       XSETBUFFER (curbuf, current_buffer);
@@ -6387,12 +6389,45 @@
 
       TEMP_SET_PT_BOTH (PT + process_coding->produced_char,
 			PT_BYTE + process_coding->produced);
       signal_after_change (PT - process_coding->produced_char,
 			   0, process_coding->produced_char);
-      carryover = process_coding->carryover_bytes;
-      if (carryover > 0)
-	memcpy (buf, process_coding->carryover,
-		process_coding->carryover_bytes);
+
+      Vlast_coding_system_used = CODING_ID_NAME (process_coding->id);
+      /* A new coding system might be found.  */
+      if (!EQ (p->decode_coding_system, Vlast_coding_system_used))
+	{
+	  pset_decode_coding_system (p, Vlast_coding_system_used);
+
+	  /* Don't call setup_coding_system for
+	     proc_decode_coding_system[channel] here.  It is done in
+	     detect_coding called via decode_coding above.  */
+
+	  /* If a coding system for encoding is not yet decided, we set
+	     it as the same as coding-system for decoding.
+
+	     But, before doing that we must check if
+	     proc_encode_coding_system[p->outfd] surely points to a
+	     valid memory because p->outfd will be changed once EOF is
+	     sent to the process.  */
+	  eassert (p->outfd < FD_SETSIZE);
+	  if (NILP (p->encode_coding_system) && p->outfd >= 0
+	      && proc_encode_coding_system[p->outfd])
+	    {
+	      pset_encode_coding_system
+		(p, coding_inherit_eol_type (Vlast_coding_system_used, Qnil));
+	      setup_coding_system (p->encode_coding_system,
+				   proc_encode_coding_system[p->outfd]);
+	    }
+	}
+
+      if (process_coding->carryover_bytes > 0)
+	{
+	  if (SCHARS (p->decoding_buf) < process_coding->carryover_bytes)
+	    pset_decoding_buf (p, make_uninit_string (process_coding->carryover_bytes));
+	  memcpy (SDATA (p->decoding_buf), process_coding->carryover,
+		  process_coding->carryover_bytes);
+	  p->decoding_carryover = process_coding->carryover_bytes;
+	}
     }
 
   read_process_output_after_insert (p, &old_read_only, old_begv, old_zv,

bug#71452: 30.0.50; GNUS hangs on IMAP/s reads starting at commit bbc1803

[Dmitry Gutov]

On 10/06/2024 07:09, Dmitry Gutov wrote:
> Print-debugging seems to say that there might be something wrong with 
> what (forward-line -1) does: output has the ^M chars, and apparently the 
> new logic makes it skip over too many characters.

All right, this was the newline cache.

I've pushed a fix, commit 5a576069fcd. This fixes Gnus in my testing.

Also heads up: the variable to disable optimization had been renamed to 
'fast-read-process-output'. Just in case you or somebody else needs to 
disable the fast path for this or some newer reason.

bug#71452: 30.0.50; GNUS hangs on IMAP/s reads starting at commit bbc1803

[epg]

Dmitry Gutov <dmitry@gutov.dev> writes:

> I've pushed a fix, commit 5a576069fcd. This fixes Gnus in my testing.

Seems to work for me too.

Thanks!

-- 
Eric Gillespie  <*>  epg@pretzelnet.org

bug#71452: 30.0.50; GNUS hangs on IMAP/s reads starting at commit bbc1803

[Eric Gillespie]

That seems to have fixed it. Thanks!

bug#71452: 30.0.50; GNUS hangs on IMAP/s reads starting at commit bbc1803

[Eli Zaretskii]

> Date: Wed, 12 Jun 2024 09:22:46 -0500
> From: "Eric Gillespie" <epg@pretzelnet.org>
> Cc: "Eli Zaretskii" <eliz@gnu.org>, 71452@debbugs.gnu.org,
>  "Stefan Kangas" <stefankangas@gmail.com>
> 
> That seems to have fixed it. Thanks!

Thanks, closing.