bug#40775: 28.0.50; dns-query sometimes infloops

bug#40775: 28.0.50; dns-query sometimes infloops

[Tassilo Horn]

[-- Attachment #1: Type: text/plain, Size: 2244 bytes --]


I use Gnus to read mail and news.  When selecting a message in Gnus, my
configuration will call gnus-treat-from-gravatar and
gnus-treat-mail-gravatar which try to fetch gravatar icons of the mail's
sender and recipients using Emacs' gravatar.el and insert them in the
mail header.

Since the recent changes to gravatar.el, the standard gravatar service
is libravatar now.  gravatar--service-libravatar does some DNS queries
to check if the domain of the mail address hosts the gravatar images
itself.

Sometimes, emacs gets stuck while doing the dns-query.  The emacs
process uses 100% of one core of my system, and this situation persists
for at least a minute (before I C-g, i.e., I don't know if it would ever
recover from such a situation).

This is the Lisp backtrace in one of such situations:

"Automatic GC" (0x0)
"timer-event-handler" (0xffff9e28)
"accept-process-output" (0xffffa4b0)
"dns-query" (0xffffa8f0)
"let*" (0xffffaa98)
"let" (0xffffabd8)
"while" (0xffffacb8)
"let" (0xffffadf8)
"catch" (0xffffaef8)
"let" (0xffffb038)
"if" (0xffffb108)
"progn" (0xffffb1c8)
"unwind-protect" (0xffffb298)
"let" (0xffffb3d8)
"gravatar--service-libravatar" (0xffffb548)
"funcall" (0xffffb540)
"format" (0xffffb658)
"gravatar-build-url" (0xffffb770)
"let" (0xffffb938)
"gravatar-retrieve" (0xffffbae8)
"gnus-gravatar-transform-address" (0xffffbea8)
"gnus-treat-mail-gravatar" (0xffffc180)
"th/gnus-article-prepared" (0xffffc3f0)
"run-hooks" (0xffffc668)
"apply" (0xffffc660)
"gnus-run-hooks" (0xffffc970)
"gnus-article-prepare" (0xffffce30)
"gnus-summary-display-article" (0xffffd200)
"gnus-summary-select-article" (0xffffd5d8)
"gnus-summary-scroll-up" (0xffffda80)
"funcall-interactively" (0xffffda78)
"call-interactively" (0xffffdd10)
"command-execute" (0xffffe0c8)
(gdb)

The non-emacs function th/gnus-article-prepared is just

(defun th/gnus-article-prepared ()
  (gnus-treat-from-gravatar)
  (gnus-treat-mail-gravatar))

Looking at dns-query, there's just the while loop with
accept-process-output which could cause an infloop.  However, with the
values of times and step, I'd say this cannot loop longer than about 5
seconds in the worst case...

The attached backtrace.txt.gz contains the full C and Lisp backtraces of
such an infloop.

[-- Attachment #2: backtrace.txt.gz --]
[-- Type: application/gzip, Size: 9299 bytes --]

[-- Attachment #3: Type: text/plain, Size: 8087 bytes --]



In GNU Emacs 28.0.50 (build 5, x86_64-pc-linux-gnu, GTK+ Version 3.24.17, cairo version 1.17.3)
 of 2020-04-21 built on thinkpad-t440p
Repository revision: f212daf0a96999b8c1e63217c22a86d6e1671687
Repository branch: master
Windowing system distributor 'The X.Org Foundation', version 11.0.12008000
System Description: Arch Linux

Recent messages:
Successfully sent message via fastmail account
20200422T190145.732> Opening nnml server on archive...
20200422T190145.733> Opening nnml server on archive...done
Wrote /home/horn/.gnus.d/News/archive/sent-mails/1914
20200422T190146.563> nnimap read 0k from imap.fastmail.com
Sending...done
command-execute: Buffer is read-only: #<buffer *Article nnimap+Fastmail:mailinglists/emacs-devel*>
20200422T190149.444> Exiting summary buffer and applying spam rules
20200422T190149.674> Expiring articles...
20200422T190149.907> Expiring articles...done

Configured using:
 'configure --with-modules 'CFLAGS=-O1 -g3 -ggdb''

Configured features:
XPM JPEG TIFF GIF PNG RSVG CAIRO SOUND GPM DBUS GSETTINGS GLIB NOTIFY
INOTIFY ACL GNUTLS LIBXML2 FREETYPE HARFBUZZ M17N_FLT LIBOTF ZLIB
TOOLKIT_SCROLL_BARS GTK3 X11 XDBE XIM MODULES THREADS LIBSYSTEMD JSON
PDUMPER LCMS2 GMP

Important settings:
  value of $LC_MONETARY: de_DE.utf8
  value of $LC_NUMERIC: de_DE.utf8
  value of $LC_TIME: de_DE.utf8
  value of $LANG: en_US.utf8
  value of $XMODIFIERS: @im=ibus
  locale-coding-system: utf-8-unix

Major mode: Group

Minor modes in effect:
  hl-line-mode: t
  cursor-sensor-mode: t
  gnus-topic-mode: t
  beacon-mode: t
  global-aggressive-indent-mode: t
  which-key-mode: t
  global-company-mode: t
  global-magit-file-mode: t
  magit-auto-revert-mode: t
  global-git-commit-mode: t
  async-bytecomp-package-mode: t
  global-undo-tree-mode: t
  override-global-mode: t
  minibuffer-depth-indicate-mode: t
  electric-pair-mode: t
  recentf-mode: t
  global-subword-mode: t
  subword-mode: t
  save-place-mode: t
  savehist-mode: t
  show-paren-mode: t
  gnus-undo-mode: t
  shell-dirtrack-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  buffer-read-only: t
  column-number-mode: t
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow emacsbug gnus-draft jka-compr mm-archive dired-aux gnus-dired
vc-mtn vc-hg expand-region subword-mode-expansions text-mode-expansions
the-org-mode-expansions web-mode-expansions er-basic-expansions
expand-region-core expand-region-custom hippie-exp qp mailalias rdictcc
sort gnus-cite gnus-async gnus-bcklg gnus-ml hl-line cursor-sensor
nndraft nnmh rot13 utf-7 nnml nnnil gnus-agent gnus-srvr gnus-score
score-mode nnvirtual gnus-cache gnus-demon nntp spam spam-stat gnus-uu
yenc gnus-msg gnus-gravatar mail-extr gnus-topic nnir gnus-registry
registry eieio-base th-private mule-util pulse profiler network-stream
gravatar url-cache misearch multi-isearch symbol-overlay cl-print vc-bzr
vc-src vc-sccs vc-svn vc-cvs vc-rcs eieio-opt speedbar ezimage dframe
dns help-fns radix-tree company-oddmuse company-keywords company-etags
etags fileloop generator company-gtags company-dabbrev-code
company-dabbrev company-files company-capf company-cmake company-xcode
company-clang company-semantic company-eclim company-template
company-bbdb paredit display-fill-column-indicator auto-package-update
finder-inf generic logview datetime extmap bindat fish-mode cargo
cargo-process rust-mode web-mode disp-table preview-latex auto-loads
tex-site eglot array jsonrpc ert pp debug backtrace xref flymake-proc
flymake compile beacon aggressive-indent rainbow-mode vc-git vc-dir ewoc
vc vc-dispatcher epa-file org ob ob-tangle ob-ref ob-lob ob-table ob-exp
org-macro org-footnote org-src ob-comint org-pcomplete org-list
org-faces org-entities org-version ob-emacs-lisp ob-core ob-eval
org-table ol org-keys org-compat org-macs org-loaddefs find-func
cal-menu calendar cal-loaddefs dired-x dired-collapse dired-hacks-utils
f s boxquote rect smtpmail-multi smtpmail sendmail ecomplete yasnippet
auto-dictionary flyspell ispell tramp-smb which-key
highlight-parentheses cl company-restclient know-your-http-well
http-status-codes http-relations http-methods http-headers company pcase
raven project restclient forge-list forge-commands forge-semi
forge-bitbucket buck forge-gogs gogs forge-gitea gtea forge-gitlab glab
forge-github ghub-graphql treepy gsexp ghub let-alist gnutls
forge-notify forge-revnote forge-pullreq forge-issue forge-topic
bug-reference forge-post markdown-mode color thingatpt noutline outline
forge-repo forge forge-core forge-db closql emacsql-sqlite emacsql
emacsql-compiler magit-submodule magit-obsolete magit-blame magit-stash
magit-reflog magit-bisect magit-push magit-pull magit-fetch magit-clone
magit-remote magit-commit magit-sequence magit-notes magit-worktree
magit-tag magit-merge magit-branch magit-reset magit-files magit-refs
magit-status magit magit-repos magit-apply magit-wip magit-log
which-func imenu magit-diff smerge-mode diff-mode magit-core
magit-autorevert autorevert filenotify magit-margin magit-transient
magit-process magit-mode git-commit transient magit-git magit-section
magit-utils crm log-edit pcvs-util add-log with-editor async-bytecomp
advice async server dash undo-tree diff visual-filename-abbrev debbugs
soap-client url-http url-auth url-gw nsm warnings rng-xsd rng-dt
rng-util xsd-regexp use-package-bind-key bind-key mb-depth
use-package-diminish windmove elec-pair rx recentf tree-widget cap-words
superword subword saveplace savehist paren smiley gnus-art mm-uu mml2015
mm-view mml-smime smime dig gnus-sum url url-proxy url-privacy
url-expand url-methods url-history mailcap shr url-cookie url-domsuf
url-util svg xml dom gnus-group gnus-undo gnus-start gnus-cloud nnimap
nnmail mail-source utf7 netrc nnoo gnus-spec gnus-int gnus-range message
rmc puny dired dired-loaddefs rfc822 mml mml-sec epa derived epg
epg-config mm-decode mm-bodies mm-encode mail-parse rfc2231 mailabbrev
gmm-utils mailheader gnus-win gnus wid-edit nnheader gnus-util rmail
rmail-loaddefs rfc2047 rfc2045 ietf-drums text-property-search mm-util
mail-prsvr mail-utils edmacro kmacro ligature easy-mmode dracula-theme
diminish cl-extra help-mode use-package-ensure use-package-core tramp
tramp-loaddefs trampver tramp-integration files-x tramp-compat shell
pcomplete comint ansi-color ring parse-time iso8601 time-date ls-lisp
format-spec info package easymenu browse-url url-handlers url-parse
auth-source cl-seq eieio eieio-core cl-macs eieio-loaddefs
password-cache json subr-x map url-vars seq byte-opt gv bytecomp
byte-compile cconv cl-loaddefs cl-lib tooltip eldoc electric uniquify
ediff-hook vc-hooks lisp-float-type mwheel term/x-win x-win
term/common-win x-dnd tool-bar dnd fontset image regexp-opt fringe
tabulated-list replace newcomment text-mode elisp-mode lisp-mode
prog-mode register page tab-bar menu-bar rfn-eshadow isearch timer
select scroll-bar mouse jit-lock font-lock syntax facemenu font-core
term/tty-colors frame minibuffer cl-generic cham georgian utf-8-lang
misc-lang vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms
cp51932 hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese composite charscript charprop case-table epa-hook
jka-cmpr-hook help simple abbrev obarray cl-preloaded nadvice loaddefs
button faces cus-face macroexp files text-properties overlay sha1 md5
base64 format env code-pages mule custom widget hashtable-print-readable
backquote threads dbusbind inotify lcms2 dynamic-setting
system-font-setting font-render-setting cairo move-toolbar gtk x-toolkit
x multi-tty make-network-process emacs)

Memory information:
((conses 16 716983 96334)
 (symbols 48 48873 46)
 (strings 32 237412 6465)
 (string-bytes 1 7560922)
 (vectors 16 153404)
 (vector-slots 8 3583548 117924)
 (floats 8 546 434)
 (intervals 56 9303 1320)
 (buffers 992 49))

bug#40775: 28.0.50; dns-query sometimes infloops

[Tassilo Horn]

Tassilo Horn <tsdh@gnu.org> writes:

> Since the recent changes to gravatar.el, the standard gravatar service
> is libravatar now.  gravatar--service-libravatar does some DNS queries
> to check if the domain of the mail address hosts the gravatar images
> itself.
>
> Sometimes, emacs gets stuck while doing the dns-query.  The emacs
> process uses 100% of one core of my system, and this situation
> persists for at least a minute (before I C-g, i.e., I don't know if it
> would ever recover from such a situation).

Ok, I think I have it.  It doesn't infloop, it just takes a lot of time
for DNS queries on some domains, and if the From, To and Cc of a message
contain many mail addresses, and for each one there are two dns-queries,
then it may add up to long delays (of several minutes).

For example, for me with emacs -Q

  (benchmark-run 1
    (dns-query "_avatars-sec._tcp.zjp.codes" 'SRV)
    (dns-query "_avatars._tcp.zjp.codes" 'SRV))

consistently takes 20 seconds, 10 seconds for each (unsuccessful) query.
Those queries will be performed for every foo@zjp.codes mail address in
a message.

In contrast, the calls for my own mail address

  (benchmark-run 1
    (dns-query "_avatars-sec._tcp.gnu.org" 'SRV)
    (dns-query "_avatars._tcp.gnu.org" 'SRV))

"only" take 5 seconds, 2.5 per query.

Bye,
Tassilo

bug#40775: 28.0.50; dns-query sometimes infloops

[Lars Ingebrigtsen]

Tassilo Horn <tsdh@gnu.org> writes:

>> Since the recent changes to gravatar.el, the standard gravatar service
>> is libravatar now.

(This bit was changed because of privacy concerns.)

> Ok, I think I have it.  It doesn't infloop, it just takes a lot of time
> for DNS queries on some domains, and if the From, To and Cc of a message
> contain many mail addresses, and for each one there are two dns-queries,
> then it may add up to long delays (of several minutes).
>
> For example, for me with emacs -Q
>
>   (benchmark-run 1
>     (dns-query "_avatars-sec._tcp.zjp.codes" 'SRV)
>     (dns-query "_avatars._tcp.zjp.codes" 'SRV))
>
> consistently takes 20 seconds, 10 seconds for each (unsuccessful) query.
> Those queries will be performed for every foo@zjp.codes mail address in
> a message.

These lookups should now be cached, so the delay should only happen the
first time, which should help some.  And all the lookups should be
asynchronous, I think, so there should be no hangs.  (If I remember
correctly.)

So I think everything in this bug report should work better now?  Are
you still seeing problems in this area (if you're still using
libravatar)?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#40775: 28.0.50; dns-query sometimes infloops

[Tassilo Horn]

References: <87eesfsaik.fsf@gnu.org> <87zhb2ywld.fsf@gnu.org> <87sg6l1req.fsf@gnus.org>
User-agent: mu4e 1.5.7; emacs 28.0.50
In-reply-to: <87sg6l1req.fsf@gnus.org>
Hi Lars,

>> For example, for me with emacs -Q
>>
>>   (benchmark-run 1
>>     (dns-query "_avatars-sec._tcp.zjp.codes" 'SRV)
>>     (dns-query "_avatars._tcp.zjp.codes" 'SRV))
>>
>> consistently takes 20 seconds, 10 seconds for each (unsuccessful)
>> query.  Those queries will be performed for every foo@zjp.codes mail
>> address in a message.
>
> These lookups should now be cached, so the delay should only happen
> the first time, which should help some.

The above benchmark is now twice as fast taking 10 seconds each time I
run it.

> And all the lookups should be asynchronous, I think, so there should
> be no hangs.  (If I remember correctly.)

Yes, seems like gravatar.el now does asynchronous lookups.

> So I think everything in this bug report should work better now?  Are
> you still seeing problems in this area (if you're still using
> libravatar)?

I'm using the default, so gravatar now.  I've now tried with

--8<---------------cut here---------------start------------->8---
(setq gravatar-service 'libravatar)
(gravatar-retrieve "foobar@zjp.codes" (lambda (g &rest args)
                                        (message "GRAVATAR: %S" g)))
--8<---------------cut here---------------end--------------->8---

which doesn't seem to block but pollutes the *Messages* buffer with
stuff like

--8<---------------cut here---------------start------------->8---
(fn STATUS)" nil t] 11 "

(fn URL)"] "r=g&d=404&s=32" format "%s/%s?%s" gravatar-hash] 7 "

(fn URL)"] "zjp.codes" ((("_avatars-sec" . "https") ("_avatars" . "http"))) (#2) inhibit-changing-match-data mapcar #[257 "A\"" [dns-get data] 4 "

(fn RECORD)"] dns-get answers #[257 "\"" [dns-get priority] 4 "
--8<---------------cut here---------------end--------------->8---

Not sure if it would do that when being used "normally", i.e., by Gnus
or some other client.

Anyway, in either case, it doesn't block anymore.  Feel free to close
the bug.

Bye,
Tassilo

bug#40775: 28.0.50; dns-query sometimes infloops

[Lars Ingebrigtsen]

Tassilo Horn <tsdh@gnu.org> writes:

> which doesn't seem to block but pollutes the *Messages* buffer with
> stuff like
>
> (fn STATUS)" nil t] 11 "
>
> (fn URL)"] "r=g&d=404&s=32" format "%s/%s?%s" gravatar-hash] 7 "
>
> (fn URL)"] "zjp.codes" ((("_avatars-sec" . "https") ("_avatars" . "http"))) (#2) inhibit-changing-match-data mapcar #[257 "A\"" [dns-get data] 4 "
>
> (fn RECORD)"] dns-get answers #[257 "\"" [dns-get priority] 4 "

I think it's just the async return value --
(progn (gravatar-retrieve "foobar@zjp.codes" 'ignore) nil) doesn't
message anything...

> Anyway, in either case, it doesn't block anymore.  Feel free to close
> the bug.

OK; done.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no