bug#49682: 27.2.50; accept-process-output within accept-process-output hangs emacs

bug#49682: 27.2.50; accept-process-output within accept-process-output hangs emacs

[Rajeev N via Bug reports for GNU Emacs, the Swiss army knife of text editors]

(let () (run-with-timer 0 nil #'url-retrieve "https://www.gnu.org/" #'ignore) (diary-mail-entries))

 hangs emacs when smtpmail library is being used for sending mail.  My guess is accept-process-output in url library is called from within accept-process-output in smtpmail library.


In GNU Emacs 27.2.50 (build 5, x86_64-pc-linux-gnu, GTK+ Version 3.24.24, cairo version 1.16.0)
 of 2021-07-14 built on hp
Repository revision: 7ac411ae2ce91572a2bdb8eaa1ee6ceccf162e35
Repository branch: emacs-27
Windowing system distributor 'The X.Org Foundation', version 11.0.12011000
System Description: Debian GNU/Linux 11 (bullseye)

Recent messages:
Running in foreground: git --no-pager difftool --exit-code --no-prompt -x diff -ub HEAD  -- workspace/src/el/mail.el
Done (status=1): git --no-pager difftool --exit-code --no-prompt -x diff -ub HEAD  -- workspace/src/el/mail.el
Running in foreground: git --no-pager cat-file blob 6b74e7f4b1948f1f9495b9c8d1e1a02d12daa090:workspace/src/el/mail.el .
Done (status=0): git --no-pager cat-file blob 6b74e7f4b1948f1f9495b9c8d1e1a02d12daa090:workspace/src/el/mail.el .
Finding changes in /home/rajeev/workspace/src/el/mail.el...done
Mark saved where search started
Mark set
next-line: End of buffer
Hiding all blocks...done
Mark activated

Configured using:
 'configure --with-mailutils --with-cairo
 --prefix=/home/rajeev/tmp/build/em/o/emacs-27'

Configured features:
XPM JPEG TIFF GIF PNG RSVG CAIRO SOUND GPM DBUS GSETTINGS GLIB NOTIFY
INOTIFY ACL LIBSELINUX GNUTLS LIBXML2 FREETYPE HARFBUZZ M17N_FLT
LIBOTF ZLIB TOOLKIT_SCROLL_BARS GTK3 X11 XDBE XIM MODULES THREADS
LIBSYSTEMD JSON PDUMPER LCMS2 GMP

Important settings:
  value of $LC_TIME: en_GB.UTF-8
  value of $LANG: en_US.UTF-8
  value of $XMODIFIERS: @im=ibus
  locale-coding-system: utf-8-unix

Major mode: Lisp Interaction

Minor modes in effect:
  reveal-mode: t
  global-so-long-mode: t
  global-auto-revert-mode: t
  shell-dirtrack-mode: t
  midnight-mode: t
  display-time-mode: t
  tooltip-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  transient-mark-mode: t
  hs-minor-mode: t

Load-path shadows:
/home/rajeev/.config/emacs/elpa/map-3.0/map hides /home/rajeev/tmp/build/em/o/emacs-27/share/emacs/27.2.50/lisp/emacs-lisp/map

Features:
(shadow emacsbug cal-coptic cal-julian url-about url-dav url-dired
dired-aux misearch multi-isearch eieio-opt speedbar sb-image ezimage
dframe debug smtpmail help-fns radix-tree cl-print org-duration
mule-util js view sort gnus-cite gnus-bcklg utf-7 nnml gnus-async
gnus-ml face-remap smerge-mode diff cc-mode cc-fonts cc-guess cc-menus
cc-cmds cc-styles cc-align cc-engine cc-vars cc-defs vc-bzr vc-src
vc-sccs vc-svn vc-cvs vc-rcs vc-dir ewoc vc vc-dispatcher mm-archive
network-stream url-cache edmacro kmacro server cursor-sensor
time-stamp bbdb-gnus nnfolder pcase rx xt-mouse which-func imenu
timeclock spam spam-stat gnus-uu yenc semantic/util-modes
semantic/util semantic semantic/tag semantic/lex semantic/fw
mode-local cedet org-indent reveal mailalias bbdb-message mail-extr
ol-eww eww mm-url thingatpt url-queue ol-rmail ol-mhe ol-irc ol-info
ol-gnus nnir ol-docview doc-view jka-compr image-mode exif ol-bibtex
bibtex ol-bbdb ol-w3m icomplete so-long cl-extra autorevert filenotify
bbdb-anniv tramp tramp-loaddefs trampver tramp-integration files-x
tramp-compat shell ls-lisp printing ps-print ps-print-loaddefs ps-def
lpr web-server web-server-status-codes el/web midnight el/cron
backtrace help-mode qp el/wthr el/av el/hass el/fin el/remote
el/script type-break cal-iso org-id lunar solar cal-dst holidays
hol-loaddefs el/calc olc el/loc term disp-table ehelp dirtrack
hideshow dbus parsec gnutls gnus-delay gnus-draft gnus-agent gnus-srvr
gnus-score score-mode nnvirtual nntp gnus-cache gnus-msg nndraft nnmh
gnus-icalendar org-capture gnus-art mm-uu mml2015 mm-view mml-smime
smime dig icalendar sieve sieve-mode sieve-manage sasl sasl-anonymous
sasl-login sasl-plain sendmail time ox-odt rng-loc rng-uri rng-parse
rng-match rng-dt rng-util rng-pttrn nxml-parse nxml-ns nxml-enc xmltok
nxml-util ox-latex ox-icalendar ox-html table ox-ascii ox-publish ox
org-element avl-tree generator org-agenda org-refile org-crypt org ob
ob-tangle ob-ref ob-lob ob-table ob-exp org-macro org-footnote org-src
ob-comint org-pcomplete pcomplete comint ansi-color ring org-list
org-faces org-entities noutline outline org-version ob-emacs-lisp
ob-core ob-eval org-table ol org-keys org-compat advice org-macs
org-loaddefs find-func gnus-sum shr svg xml dom gnus-group gnus-undo
gnus-start gnus-cloud nnimap nnmail mail-source utf7 netrc nnoo
parse-time iso8601 gnus-spec gnus-int gnus-range message dired
dired-loaddefs format-spec rfc822 mml mml-sec epa derived mm-decode
mm-bodies mm-encode gmm-utils mailheader gnus-win gnus pp vc-git
diff-mode easy-mmode cus-edit cus-start cus-load nnheader gnus-util
rmail rmail-loaddefs text-property-search time-date mail-utils
wid-edit el/org el/doc el/mail oauth2 url-http url url-proxy
url-privacy url-expand url-methods url-history mailcap url-auth
mail-parse rfc2231 rfc2047 rfc2045 mm-util ietf-drums mail-prsvr
url-cookie url-domsuf url-util url-gw nsm rmc puny plstore epg
epg-config el/tools el/shell el/webdr el/xmpp el/diary timer-list
bbdb-mua el/timer el/util bbdb-com crm mailabbrev bbdb bbdb-site
timezone el/bbdb appt diary-lib diary-loaddefs cal-menu calendar
cal-loaddefs el/init wombat-theme info package easymenu browse-url
url-handlers url-parse auth-source cl-seq eieio eieio-core cl-macs
eieio-loaddefs password-cache json subr-x map url-vars seq byte-opt gv
bytecomp byte-compile cconv cl-loaddefs cl-lib tooltip eldoc electric
uniquify ediff-hook vc-hooks lisp-float-type mwheel term/x-win x-win
term/common-win x-dnd tool-bar dnd fontset image regexp-opt fringe
tabulated-list replace newcomment text-mode elisp-mode lisp-mode
prog-mode register page tab-bar menu-bar rfn-eshadow isearch timer
select scroll-bar mouse jit-lock font-lock syntax facemenu font-core
term/tty-colors frame minibuffer cl-generic cham georgian utf-8-lang
misc-lang vietnamese tibetan thai tai-viet lao korean japanese
eucjp-ms cp51932 hebrew greek romanian slovak czech european ethiopic
indian cyrillic chinese composite charscript charprop case-table
epa-hook jka-cmpr-hook help simple abbrev obarray cl-preloaded nadvice
loaddefs button faces cus-face macroexp files text-properties overlay
sha1 md5 base64 format env code-pages mule custom widget
hashtable-print-readable backquote threads dbusbind inotify lcms2
dynamic-setting system-font-setting font-render-setting cairo
move-toolbar gtk x-toolkit x multi-tty make-network-process emacs)

Memory information:
((conses 16 564438 119230)
 (symbols 48 40013 57)
 (strings 32 155671 22362)
 (string-bytes 1 5292815)
 (vectors 16 65816)
 (vector-slots 8 792333 55620)
 (floats 8 1282 1751)
 (intervals 56 14139 549)
 (buffers 1000 137))

bug#49682: 27.2.50; accept-process-output within accept-process-output hangs emacs

[Lars Ingebrigtsen]

Rajeev N via "Bug reports for GNU Emacs, the Swiss army knife of text
editors" <bug-gnu-emacs@gnu.org> writes:

> [Quoted text removed due to X-No-Archive]

Can you try the same recipe in the current development version of Emacs?
There's been some changes to try to guarantee forward progress when
there's several processes waiting for output.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#49682: 27.2.50; accept-process-output within accept-process-output hangs emacs

[Rajeev N via Bug reports for GNU Emacs, the Swiss army knife of text editors]

[-- Attachment #1: Type: text/html, Size: 344 bytes --]

bug#49682: 27.2.50; accept-process-output within accept-process-output hangs emacs

[Lars Ingebrigtsen]

Rajeev N <rajeev.jnk@sivalik.com> writes:

> [Quoted text removed due to X-No-Archive]

Thanks for checking.

The test case is:

(let () (run-with-timer 0 nil #'url-retrieve "https://www.gnu.org/" #'ignore) (diary-mail-entries))

This does not hang for me on the current trunk, unfortunately, but
perhaps there's a timing issue involved?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#49682: 27.2.50; accept-process-output within accept-process-output hangs emacs

[Lars Ingebrigtsen]

(Please keep the debbugs address in the CCs -- otherwise the mail won't
reach the bug tracker.)

Rajeev N <rajeev.jnk@sivalik.com> writes:

> Can you please confirm that it does not hang for you even when
> smtpmail elisp library is being used to send mail.

Yes, I use smtpmail.

bug#49682: 27.2.50; accept-process-output within accept-process-output hangs emacs

[Eli Zaretskii]

> Date: Wed, 21 Jul 2021 10:58:22 -0400
> From:  Rajeev N via "Bug reports for GNU Emacs,
>  the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>
> 
> (let () (run-with-timer 0 nil #'url-retrieve "https://www.gnu.org/" #'ignore) (diary-mail-entries))

Don't do that, because it will hurt.  Mixing timers with subprocesses
is asking for trouble.

bug#49682: 27.2.50; accept-process-output within accept-process-output hangs emacs

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> (let () (run-with-timer 0 nil #'url-retrieve "https://www.gnu.org/"
>> #'ignore) (diary-mail-entries))
>
> Don't do that, because it will hurt.  Mixing timers with subprocesses
> is asking for trouble.

That's not my experience, really -- url-queue.el is based around firing
off `url-retrieve' from timers, and I've never seen any problems.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#49682: 27.2.50; accept-process-output within accept-process-output hangs emacs

[Rajeev N via Bug reports for GNU Emacs, the Swiss army knife of text editors]

[-- Attachment #1: Type: text/html, Size: 884 bytes --]

bug#49682: 27.2.50; accept-process-output within accept-process-output hangs emacs

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Rajeev N <rajeev.jnk@sivalik.com>,  49682@debbugs.gnu.org
> Date: Wed, 21 Jul 2021 23:32:16 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> (let () (run-with-timer 0 nil #'url-retrieve "https://www.gnu.org/"
> >> #'ignore) (diary-mail-entries))
> >
> > Don't do that, because it will hurt.  Mixing timers with subprocesses
> > is asking for trouble.
> 
> That's not my experience, really -- url-queue.el is based around firing
> off `url-retrieve' from timers, and I've never seen any problems.

Sheer luck, IMO.

I don't really understand the need to run subprocesses from timers,
since subprocess support already includes async operation and built-in
time-outs.

bug#49682: 27.2.50; accept-process-output within accept-process-output hangs emacs

[Rajeev N via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Hi Lars,

Can you please check the following recipe, with smtpmail package without queuing, it hangs emacs for me, you may have to try couple of times.

(let ()
  (run-with-timer 0 nil #'url-retrieve "https://www.gnu.org/" #'ignore)
  (make-process :name "echo" :command '("echo")
   :sentinel (lambda (_p _e) (run-with-timer 0 nil #'diary-mail-entries))))

Thanks.

bug#49682: 27.2.50; accept-process-output within accept-process-output hangs emacs

[Dmitry Gutov]

On 22.07.2021 08:47, Eli Zaretskii wrote:
>> From: Lars Ingebrigtsen <larsi@gnus.org>
>> Cc: Rajeev N <rajeev.jnk@sivalik.com>,  49682@debbugs.gnu.org
>> Date: Wed, 21 Jul 2021 23:32:16 +0200
>>
>> Eli Zaretskii <eliz@gnu.org> writes:
>>
>>>> (let () (run-with-timer 0 nil #'url-retrieve "https://www.gnu.org/"
>>>> #'ignore) (diary-mail-entries))
>>>
>>> Don't do that, because it will hurt.  Mixing timers with subprocesses
>>> is asking for trouble.
>>
>> That's not my experience, really -- url-queue.el is based around firing
>> off `url-retrieve' from timers, and I've never seen any problems.
> 
> Sheer luck, IMO.
> 
> I don't really understand the need to run subprocesses from timers,
> since subprocess support already includes async operation and built-in
> time-outs.

This is a fairly common configuration. Consider a minimal IDE setup:

1. Code completion popup which is shown when Emacs is idle for XX ms.
2. Eldoc hints which are shown at YY ms after the user stops typing.

When both are implemented using HTTP calls, you get "timers with 
subprocesses".

In fact, I have a configuration like that (based on url.el) and get an 
occasional strong freeze (one that requires 3 C-g's to get out of) about 
once every 1-2 weeks. Haven't found a solid repro so far, so if the 
reporter has something stable-ish, I'll be following this report intently.

bug#49682: 27.2.50; accept-process-output within accept-process-output hangs emacs

[Lars Ingebrigtsen]

Rajeev N <rajeev.jnk@sivalik.com> writes:

> [Quoted text removed due to X-No-Archive]

I've tried it a bunch of times, but it never hangs for me.  It does
message this, though:

Sending via mail...
Blocking call to accept-process-output with quit inhibited!! [4 times]
Sending email  
Blocking call to accept-process-output with quit inhibited!!
Sending email done
Blocking call to accept-process-output with quit inhibited!!


-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#49682: 27.2.50; accept-process-output within accept-process-output hangs emacs

[Rajeev N via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Thank for checking.  It always hangs for me in the 1st or 2nd try.  Here is what I get when I send a usr2 signal to emacs process.

Debugger entered--Lisp error: (quit)
  accept-process-output(#<process smtpmail> 0.05)
  network-stream-get-response(#<process smtpmail> 1 "^[0-9]+ .*\15\n")
  network-stream-open-starttls("smtpmail" #<buffer *trace of SMTP session to smtp.gmail.com*> "smtp.gmail.com" 587 (:type starttls :return-list t :warn-unless-encrypted nil :capability-command "EHLO hp\15\n" :end-of-command "^[0-9]+ .*\15\n" :success "^2.*\n" :always-query-capabilities t :starttls-function #f(compiled-function (capabilities) #<bytecode 0x1565a5a06db9>) :client-certificate t :use-starttls-if-possible t))
  open-network-stream("smtpmail" #<buffer *trace of SMTP session to smtp.gmail.com*> "smtp.gmail.com" 587 :type starttls :return-list t :warn-unless-encrypted nil :capability-command "EHLO hp\15\n" :end-of-command "^[0-9]+ .*\15\n" :success "^2.*\n" :always-query-capabilities t :starttls-function #f(compiled-function (capabilities) #<bytecode 0x1565a5a06db9>) :client-certificate t :use-starttls-if-possible t)
  smtpmail-via-smtp(("rajeev.jnk@sivalik.com") #<buffer  smtpmail temp>)
  smtpmail-send-it()
  message-smtpmail-send-it()
  message-multi-smtp-send-mail()
  gnus-agent-send-mail()
  message--send-mail-maybe-partially()
  message-send-mail(nil)
  message-send-via-mail(nil)
  message-send(nil)
  message-send-and-exit(nil)
  funcall-interactively(message-send-and-exit nil)
  call-interactively(message-send-and-exit)
  diary-mail-entries()
  apply(diary-mail-entries nil)
  timer-event-handler([t 24827 60854 498512 nil diary-mail-entries nil nil 704000])

bug#49682: 27.2.50; accept-process-output within accept-process-output hangs emacs

[Dmitry Gutov]

On 23.07.2021 20:00, Rajeev N via Bug reports for GNU Emacs, the Swiss 
army knife of text editors wrote:
> (let ()
>    (run-with-timer 0 nil #'url-retrieve"https://www.gnu.org/"  #'ignore)
>    (make-process :name "echo" :command '("echo")
>     :sentinel (lambda (_p _e) (run-with-timer 0 nil #'diary-mail-entries))))

Any chance you have an example of the problem that doesn't require 
SMTP/Mail/Gnus to be configured in Emacs?

bug#49682: 27.2.50; accept-process-output within accept-process-output hangs emacs

[Rajeev N via Bug reports for GNU Emacs, the Swiss army knife of text editors]

For me the following also hangs emacs.  The problem seems to me that (accept-process-output stream 0.05) in network-stream-get-response does not return even though it has a timeout of .05.

(let ()
  (run-with-timer 0 nil #'url-retrieve "https://www.gnu.org/" #'ignore)
  (make-process :name "echo" :command '("echo")
		:sentinel (lambda (_p _e)
			    (with-current-buffer (generate-new-buffer " *debug")
			      (run-with-timer 0 nil #'open-network-stream
					      "testing" (current-buffer) "smtp.gmail.com" 587
					      :type 'starttls
					      :return-list t
					      :warn-unless-encrypted t
					      :capability-command "EHLO www.gnu.org\r\n"
					      :end-of-command "^[0-9]+ .*\r\n"
					      :success "^2.*\n"
					      :always-query-capabilities t
					      :starttls-function
					      (lambda (capabilities)
						(and (string-match "[ -]STARTTLS" capabilities)
						     "STARTTLS\r\n"))
					      :client-certificate t
					      :use-starttls-if-possible t)))))


On Mon, 26 Jul 2021 at  9:58 pm +0300, Dmitry Gutov wrote:

#+begin_quote
 On 23.07.2021 20:00, Rajeev N via Bug reports for GNU Emacs, the Swiss
 army knife of text editors wrote:

> (let ()
>    (run-with-timer 0 nil #'url-retrieve"https://www.gnu.org/"  #'ignore)
>    (make-process :name "echo" :command '("echo")
>     :sentinel (lambda (_p _e) (run-with-timer 0 nil #'diary-mail-entries))))

 Any chance you have an example of the problem that doesn't require
 SMTP/Mail/Gnus to be configured in Emacs?
#+end_quote

bug#49682: 27.2.50; accept-process-output within accept-process-output hangs emacs

[Dmitry Gutov]

On 26.07.2021 22:54, Rajeev N via Bug reports for GNU Emacs, the Swiss 
army knife of text editors wrote:
> For me the following also hangs emacs.  The problem seems to me that (accept-process-output stream 0.05) in network-stream-get-response does not return even though it has a timeout of .05.
> 
> (let ()
>    (run-with-timer 0 nil #'url-retrieve"https://www.gnu.org/"  #'ignore)
>    (make-process :name "echo" :command '("echo")
> 		:sentinel (lambda (_p _e)
> 			    (with-current-buffer (generate-new-buffer " *debug")
> 			      (run-with-timer 0 nil #'open-network-stream
> 					      "testing" (current-buffer) "smtp.gmail.com" 587
> 					      :type 'starttls
> 					      :return-list t
> 					      :warn-unless-encrypted t
> 					      :capability-command "EHLOwww.gnu.org\r\n"
> 					      :end-of-command "^[0-9]+ .*\r\n"
> 					      :success "^2.*\n"
> 					      :always-query-capabilities t
> 					      :starttls-function
> 					      (lambda (capabilities)
> 						(and (string-match "[ -]STARTTLS" capabilities)
> 						     "STARTTLS\r\n"))
> 					      :client-certificate t
> 					      :use-starttls-if-possible t)))))

Hmm, that doesn't freeze for me.

Even though if I follow 'C-x C-e' with 'C-p' right away, it will wait 
for like 0.5s before moving the cursor to the previous line.

I have tried it in a number of versions of Emacs, so far no luck.

Any chance you're not starting with 'emacs -Q' and your personal config 
affects this?

bug#49682: 27.2.50; accept-process-output within accept-process-output hangs emacs

[Rajeev N via Bug reports for GNU Emacs, the Swiss army knife of text editors]

I am able to reproduce the hang on head of master and emacs-27 branches using emacs -Q,  it does not happen every time, sometime it takes a few tries.  The following is what is shown by strace in a loop:

pselect6(20, [3 5 13 17 19], NULL, NULL, {tv_sec=0, tv_nsec=0}, {NULL, 8}) = 1 (in [19], left {tv_sec=0, tv_nsec=0})
rt_sigprocmask(SIG_BLOCK, [INT], [], 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, [INT], [], 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
pselect6(20, [3 5 6 8 13 15 17 19], [], NULL, {tv_sec=0, tv_nsec=0}, {NULL, 8}) = 1 (in [19], left {tv_sec=0, tv_nsec=0})
rt_sigprocmask(SIG_BLOCK, [INT], [], 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, [INT], [], 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
pselect6(20, [3 5 13 17 19], NULL, NULL, {tv_sec=0, tv_nsec=0}, {NULL, 8}) = 1 (in [19], left {tv_sec=0, tv_nsec=0})
rt_sigprocmask(SIG_BLOCK, [INT], [], 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, [INT], [], 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
pselect6(20, [3 5 6 8 13 15 17 19], [], NULL, {tv_sec=0, tv_nsec=0}, {NULL, 8}) = 1 (in [19], left {tv_sec=0, tv_nsec=0})
rt_sigprocmask(SIG_BLOCK, [INT], [], 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0

bug#49682: 27.2.50; accept-process-output within accept-process-output hangs emacs

[Dmitry Gutov]

On 26.07.2021 23:55, Rajeev N wrote:
> I am able to reproduce the hang on head of master and emacs-27 branches using emacs -Q,  it does not happen every time, sometime it takes a few tries.  The following is what is shown by strace in a loop:
> 
> pselect6(20, [3 5 13 17 19], NULL, NULL, {tv_sec=0, tv_nsec=0}, {NULL, 8}) = 1 (in [19], left {tv_sec=0, tv_nsec=0})
> rt_sigprocmask(SIG_BLOCK, [INT], [], 8) = 0
> rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
> rt_sigprocmask(SIG_BLOCK, [INT], [], 8) = 0
> rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
> pselect6(20, [3 5 6 8 13 15 17 19], [], NULL, {tv_sec=0, tv_nsec=0}, {NULL, 8}) = 1 (in [19], left {tv_sec=0, tv_nsec=0})
> rt_sigprocmask(SIG_BLOCK, [INT], [], 8) = 0
> rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
> rt_sigprocmask(SIG_BLOCK, [INT], [], 8) = 0
> rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
> pselect6(20, [3 5 13 17 19], NULL, NULL, {tv_sec=0, tv_nsec=0}, {NULL, 8}) = 1 (in [19], left {tv_sec=0, tv_nsec=0})
> rt_sigprocmask(SIG_BLOCK, [INT], [], 8) = 0
> rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
> rt_sigprocmask(SIG_BLOCK, [INT], [], 8) = 0
> rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
> pselect6(20, [3 5 6 8 13 15 17 19], [], NULL, {tv_sec=0, tv_nsec=0}, {NULL, 8}) = 1 (in [19], left {tv_sec=0, tv_nsec=0})
> rt_sigprocmask(SIG_BLOCK, [INT], [], 8) = 0
> rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0

Maybe Lars or Eli can interpret this log.

bug#49682: 27.2.50; accept-process-output within accept-process-output hangs emacs

[Rajeev N via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Another input that may help resolve the issue:

In the pselect loop- file descriptor 19 is show in CLOSE_WAIT state by lsof.

lsof:

emacs   66152 rajeev   19u     IPv4            2567759      0t0      TCP hp:37994->wildebeest.gnu.org:https (CLOSE_WAIT)

strace:

 pselect6(20, [3 5 6 8 13 15 17 19], [], NULL, {tv_sec=0, tv_nsec=0}, {NULL, 8}) = 1 (in [19], left {tv_sec=0, tv_nsec=0})
 rt_sigprocmask(SIG_BLOCK, [INT], [], 8) = 0
 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
 rt_sigprocmask(SIG_BLOCK, [INT], [], 8) = 0
 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
 pselect6(20, [3 5 13 17 19], NULL, NULL, {tv_sec=0, tv_nsec=0}, {NULL, 8}) = 1 (in [19], left {tv_sec=0, tv_nsec=0})

bug#49682: 27.2.50; accept-process-output within accept-process-output hangs emacs

[Eli Zaretskii]

> Disposition-Notification-To: rajeev.jnk@sivalik.com
> From: Rajeev N <rajeev.jnk@sivalik.com>
> Cc: Lars Ingebrigtsen <larsi@gnus.org>,  49682@debbugs.gnu.org,  Eli
>  Zaretskii <eliz@gnu.org>
> Date: Tue, 27 Jul 2021 13:22:54 -0400
> 
> Another input that may help resolve the issue:
> 
> In the pselect loop- file descriptor 19 is show in CLOSE_WAIT state by lsof.
> 
> lsof:
> 
> emacs   66152 rajeev   19u     IPv4            2567759      0t0      TCP hp:37994->wildebeest.gnu.org:https (CLOSE_WAIT)

If the remote host closed the connection, why isn't our sentinel being
called and the connection closed?

bug#49682: 27.2.50; accept-process-output within accept-process-output hangs emacs

[Rajeev N via Bug reports for GNU Emacs, the Swiss army knife of text editors]

It seems to be a bug-  a similar bug was fixed recently.

  http://git.savannah.gnu.org/cgit/emacs.git/commit/src/process.c?id=1773679af3241919a85d6174b1554070a63cca79

On Tue, 27 Jul 2021 at  8:35 PM +0300, Eli Zaretskii wrote:

#+begin_quote
> Disposition-Notification-To: rajeev.jnk@sivalik.com
> From: Rajeev N <rajeev.jnk@sivalik.com>
> Cc: Lars Ingebrigtsen <larsi@gnus.org>,  49682@debbugs.gnu.org,  Eli
>  Zaretskii <eliz@gnu.org>
> Date: Tue, 27 Jul 2021 13:22:54 -0400
> 
> Another input that may help resolve the issue:
> 
> In the pselect loop- file descriptor 19 is show in CLOSE_WAIT state by lsof.
> 
> lsof:
> 
> emacs   66152 rajeev   19u     IPv4            2567759      0t0      TCP hp:37994->wildebeest.gnu.org:https (CLOSE_WAIT)

 If the remote host closed the connection, why isn't our sentinel being
 called and the connection closed?
#+end_quote

bug#49682: 27.2.50; accept-process-output within accept-process-output hangs emacs

[Eli Zaretskii]

> Disposition-Notification-To: rajeev.jnk@sivalik.com
> From: Rajeev N <rajeev.jnk@sivalik.com>
> Cc: dgutov@yandex.ru,  larsi@gnus.org,  49682@debbugs.gnu.org
> Date: Tue, 27 Jul 2021 17:40:27 -0400
> 
> It seems to be a bug-  a similar bug was fixed recently.

It might be a bug, but we need a more detailed understanding of how
this happens.

I also very much wonder why many people are unable to reproduce this
problem on very similar if not identical systems.  Could this be a
kernel or a library bug specific to your system?

bug#49682: 27.2.50; accept-process-output within accept-process-output hangs emacs

[Rajeev N via Bug reports for GNU Emacs, the Swiss army knife of text editors]

[-- Attachment #1: Type: text/html, Size: 808 bytes --]

bug#49682: 27.2.50; accept-process-output within accept-process-output hangs emacs

[Robert Pluim]

>>>>> On Wed, 28 Jul 2021 19:12:58 +0300, Eli Zaretskii <eliz@gnu.org> said:

    >> Disposition-Notification-To: rajeev.jnk@sivalik.com
    >> From: Rajeev N <rajeev.jnk@sivalik.com>
    >> Cc: dgutov@yandex.ru,  larsi@gnus.org,  49682@debbugs.gnu.org
    >> Date: Tue, 27 Jul 2021 17:40:27 -0400
    >> 
    >> It seems to be a bug-  a similar bug was fixed recently.

    Eli> It might be a bug, but we need a more detailed understanding of how
    Eli> this happens.

    Eli> I also very much wonder why many people are unable to reproduce this
    Eli> problem on very similar if not identical systems.  Could this be a
    Eli> kernel or a library bug specific to your system?

I can reproduce it from 'emacs -Q -nw' on my Debian bullseye system,
but not from 'emacs -Q'

SIGUSR2 backtrace:

Debugger entered--Lisp error: (quit)
  accept-process-output(#<process testing> 0.05)
  network-stream-get-response(#<process testing> 250 "^[0-9]+ .*\15\n")
  network-stream-command(#<process testing> "EHLO www.gnu.org\15\n" "^[0-9]+ .*\15\n")
  network-stream-open-starttls("testing" #<buffer  *debug> "smtp.gmail.com" 587 (:type starttls :return-list t :warn-unless-encrypted t :capability-command "EHLO www.gnu.org\15\n" :end-of-command "^[0-9]+ .*\15\n" :success "^2.*\n" :always-query-capabilities t :starttls-function (lambda (capabilities) (and (string-match "[ -]STARTTLS" capabilities) "STARTTLS\15\n")) :client-certificate t :use-starttls-if-possible t))
  open-network-stream("testing" #<buffer  *debug> "smtp.gmail.com" 587 :type starttls :return-list t :warn-unless-encrypted t :capability-command "EHLO www.gnu.org\15\n" :end-of-command "^[0-9]+ .*\15\n" :success "^2.*\n" :always-query-capabilities t :starttls-function (lambda (capabilities) (and (string-match "[ -]STARTTLS" capabilities) "STARTTLS\15\n")) :client-certificate t :use-starttls-if-possible t)
  apply(open-network-stream ("testing" #<buffer  *debug> "smtp.gmail.com" 587 :type starttls :return-list t :warn-unless-encrypted t :capability-command "EHLO www.gnu.org\15\n" :end-of-command "^[0-9]+ .*\15\n" :success "^2.*\n" :always-query-capabilities t :starttls-function (lambda (capabilities) (and (string-match "[ -]STARTTLS" capabilities) "STARTTLS\15\n")) :client-certificate t :use-starttls-if-possible t))
  timer-event-handler([t 24845 26710 703698 nil open-network-stream ("testing" #<buffer  *debug> "smtp.gmail.com" 587 :type starttls :return-list t :warn-unless-encrypted t :capability-command "EHLO www.gnu.org\15\n" :end-of-command "^[0-9]+ .*\15\n" :success "^2.*\n" :always-query-capabilities t :starttls-function (lambda (capabilities) (and (string-match "[ -]STARTTLS" capabilities) "STARTTLS\15\n")) :client-certificate t :use-starttls-if-possible t) nil 536000])

Iʼd debug further, but I have other Emacs plans this weekend

Robert
-- 

bug#49682: 27.2.50; accept-process-output within accept-process-output hangs emacs

[LdBeth]

This bug seems happened many times to me (on Emacs 29.1 macOS, with
Mitsuharu's patches if that matters) whenever I open an multipart
email that contains HTML that has several image links through IMAP in
the Wanderlust email reader since I start to enable HTML images in the
email reader. (And for some reason C-g seems not able to interrupt the
hang)

The shr library uses url-queue to asynchronously download images
and Wanderlust also use `accept-process-output' call to handle IMAP.

For this particular case I used a dirty hack

(define-advice accept-process-output
    (:before (&rest _) sync-queue)
  (when (fboundp 'url-queue-check-progress)
    (funcall #'url-queue-check-progress)))

However, could we make `accept-process-output' atomic so
the timer won't interrupt it? Or there is no reliable
method yet to make any atomic operations in Emacs yet?

---
ldbeth

bug#49682: 27.2.50; accept-process-output within accept-process-output hangs emacs

[Eli Zaretskii]

> Date: Mon, 06 Nov 2023 11:14:18 -0600
> From: LdBeth <andpuke@foxmail.com>
> 
> This bug seems happened many times to me (on Emacs 29.1 macOS, with
> Mitsuharu's patches if that matters) whenever I open an multipart
> email that contains HTML that has several image links through IMAP in
> the Wanderlust email reader since I start to enable HTML images in the
> email reader. (And for some reason C-g seems not able to interrupt the
> hang)
> 
> The shr library uses url-queue to asynchronously download images
> and Wanderlust also use `accept-process-output' call to handle IMAP.
> 
> For this particular case I used a dirty hack
> 
> (define-advice accept-process-output
>     (:before (&rest _) sync-queue)
>   (when (fboundp 'url-queue-check-progress)
>     (funcall #'url-queue-check-progress)))
> 
> However, could we make `accept-process-output' atomic so
> the timer won't interrupt it? Or there is no reliable
> method yet to make any atomic operations in Emacs yet?

If a Lisp program wants to avoid timers during the call to
accept-process-output, could perhaps temporarily bind timer-list to
nil or something?

bug#49682: 27.2.50; accept-process-output within accept-process-output hangs emacs

[LdBeth]

>>>>> In <83a5rq3hna.fsf@gnu.org> 
>>>>>	Eli Zaretskii <eliz@gnu.org> wrote:

ldb> The shr library uses url-queue to asynchronously download images
ldb> and Wanderlust also use `accept-process-output' call to handle
ldb> IMAP.
ldb> 
ldb> For this particular case I used a dirty hack
ldb> 
ldb> (define-advice accept-process-output
ldb>     (:before (&rest _) sync-queue)
ldb>   (when (fboundp 'url-queue-check-progress)
ldb>     (funcall #'url-queue-check-progress)))
ldb> 
ldb> However, could we make `accept-process-output' atomic so the
ldb> timer won't interrupt it? Or there is no reliable method yet to
ldb> make any atomic operations in Emacs yet?

Eli> If a Lisp program wants to avoid timers during the call to
Eli> accept-process-output, could perhaps temporarily bind timer-list
Eli> to nil or something?

Thank you, let Wanderlust maintainers know and figure out
changes needed to fix the problem.

---
ldbeth

bug#49682: 27.2.50; accept-process-output within accept-process-output hangs emacs

[Michael Albinus]

LdBeth <andpuke@foxmail.com> writes:

Hi,

> Eli> If a Lisp program wants to avoid timers during the call to
> Eli> accept-process-output, could perhaps temporarily bind timer-list
> Eli> to nil or something?
>
> Thank you, let Wanderlust maintainers know and figure out
> changes needed to fix the problem.

FTR, you could check tramp.el in the Emacs git repository. There
is the macro with-tramp-suspended-timers, which is used in
tramp-accept-process-output. Just as example.

> ldbeth

Best regards, Michael.

bug#49682: 27.2.50; accept-process-output within accept-process-output hangs emacs

[Kazuhiro Ito]

> If a Lisp program wants to avoid timers during the call to
> accept-process-output, could perhaps temporarily bind timer-list to
> nil or something?

According to docstring, it seems that accept-process-output has option
not to run timers.

> (accept-process-output &optional PROCESS SECONDS MILLISEC
> JUST-THIS-ONE)
...
> If JUST-THIS-ONE is an integer, don’t run any timers either.

Can Lisp programs use the feature?

--
Kazuhiro Ito

bug#49682: 27.2.50; accept-process-output within accept-process-output hangs emacs

[Eli Zaretskii]

> Date: Thu, 09 Nov 2023 00:50:37 +0900
> From: Kazuhiro Ito <kzhr@d1.dion.ne.jp>
> Cc: 49682@debbugs.gnu.org
> 
> > If a Lisp program wants to avoid timers during the call to
> > accept-process-output, could perhaps temporarily bind timer-list to
> > nil or something?
> 
> According to docstring, it seems that accept-process-output has option
> not to run timers.
> 
> > (accept-process-output &optional PROCESS SECONDS MILLISEC
> > JUST-THIS-ONE)
> ...
> > If JUST-THIS-ONE is an integer, don’t run any timers either.
> 
> Can Lisp programs use the feature?

If they are okay with receiving output only from PROCESS, yes, they
can.