bug#20404: 25.0.50; Sometimes no fontification with jit-lock-defer-time

bug#20404: 25.0.50; Sometimes no fontification with jit-lock-defer-time

[Tassilo Horn]

I use the settings

  (setq jit-lock-defer-time 0.0312
        jit-lock-stealth-nice 0.117
        jit-lock-stealth-time 0.23)

to speed up scrolling in large buffers with complex font-lock rules.
That basically works nice.  However, quite often I'm faced with buffers
that are not fontified before I do something, e.g., move point.

This frequently happens when installing/upgrading packages with the
package manager.  That shows the buffer containing the byte-compliation
messages with fontified warnings and errors.  Every second time or so,
I'm faced with a completely unfontified buffer that gets fontified only
after selecting the window showing the buffer.

Another and even reproducible case is `report-emacs-bug'.  It goes like
this:

  1. emacs -Q
  2. eval my above settings in *scratch*
  3. M-x report-emacs-bug

The new message buffer is completely unfontified initially.  As soon as
you move point, font-lock kicks in and the header lines get their proper
fontification.  Or maybe it's the other way round, i.e., the buffer is
created, then redisplayed, then the fontification takes place but no
redisplay is performed thereafter.

Of course, what I had expected in both cases is that I get fontified
buffers almost immediately, e.g., after 0.0312 seconds.  (In the
byte-compile case I could imagine that it takes a bit longer if there
is so much output that redisplay isn't called for some longer period.)

Fun anecdote: I really have a pretty fast machine but still scrolling,
e.g., buffer.c, by pressing and holding `C-v' causes visible scrolling
for 1-2 seconds, and then the display freezes.  Emacs burns one core,
and the next visible change I can observe is that the end of the buffer
is displayed after the ~6 seconds where emacs seems frozen.  That lead
me to looking for solutions, and deferred JIT lock and stealth lock
seemed like a good idea (and they are!).

But I've been in contact with some other emacs user who didn't face
these display freezes on several machines with lower specs than mine.
Then we tried figuring out where's the difference.  I've tested several
different machines, emacs versions, toolkits, emacs -nw, optimized
builds and (almost) always got the display freezes.  He did the same and
didn't encounter the freezes.

What would you guess as likely culprit?

Solution to the riddle: V'ir pbasvtherq zl K jvaqbj flfgrz jvgu n irel
uvtu xrlobneq ercrng engr bs 58 Um.  Guhf, jura V cerff naq ubyq `P-i',
`fpebyy-hc-pbzznaq' vf pnyyrq 58 gvzrf cre frpbaq juvpu frrzf gb or fb
tbqqnz snfg gung sbag-ybpx trgf fb ohfl gung erqvfcynl unf ab punapr gb
xvpx va.  Fb Ryv, va pnfr lbh ernq guvf, urer vf nabgure pnfr jurer fbzr
sbeprq crevbqvp erqvfcynl pbhyq or avpr.



In GNU Emacs 25.0.50.12 (x86_64-unknown-linux-gnu, GTK+ Version 3.16.2)
 of 2015-04-21 on thinkpad-t440p
Windowing system distributor `The X.Org Foundation', version 11.0.11701000
System Description:	Arch Linux

Configured using:
 `configure --enable-checking=no --without-m17n-flt
 'CFLAGS=-march=native -O2''

Configured features:
XPM JPEG TIFF GIF PNG RSVG IMAGEMAGICK SOUND GPM DBUS GCONF GSETTINGS
NOTIFY ACL GNUTLS LIBXML2 FREETYPE LIBOTF XFT ZLIB

Important settings:
  value of $LC_MONETARY: de_DE.utf8
  value of $LC_NUMERIC: de_DE.utf8
  value of $LC_TIME: de_DE.utf8
  value of $LANG: en_US.utf8
  locale-coding-system: utf-8-unix

Major mode: LaTeX/PS

Minor modes in effect:
  rcirc-track-minor-mode: t
  auto-dictionary-mode: t
  flyspell-mode: t
  reftex-mode: t
  TeX-PDF-mode: t
  TeX-source-correlate-mode: t
  magit-auto-revert-mode: t
  diff-auto-refine-mode: t
  global-company-mode: t
  global-aggressive-indent-mode: t
  global-edit-server-edit-mode: t
  pdf-occur-global-minor-mode: t
  recentf-mode: t
  shell-dirtrack-mode: t
  outline-minor-mode: t
  th/sentence-hl-mode: t
  global-subword-mode: t
  subword-mode: t
  savehist-mode: t
  show-paren-mode: t
  ivy-mode: t
  minibuffer-depth-indicate-mode: t
  electric-pair-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  column-number-mode: t
  line-number-mode: t

Recent messages:
Wrote /home/horn/Repos/uni/ttc15-train-benchmark-funnyqt/doc/ttc-train-benchmark.tex
Type `C-c C-l' to display results of compilation.
LaTeX: successfully formatted {3} pages
Quit
Type `C-c C-l' to display results of compilation.
LaTeX: successfully formatted {3} pages
pdf-view-goto-page: No such page: 4
Auto-saving...done
Saving file /home/horn/Repos/uni/ttc15-train-benchmark-funnyqt/doc/ttc-train-benchmark.tex...
Wrote /home/horn/Repos/uni/ttc15-train-benchmark-funnyqt/doc/ttc-train-benchmark.tex

Load-path shadows:
~/Repos/el/auctex/lpath hides ~/Repos/el/gnus/lisp/lpath
~/Repos/el/gnus/lisp/md4 hides /home/horn/Repos/el/emacs/lisp/md4
~/Repos/el/gnus/lisp/color hides /home/horn/Repos/el/emacs/lisp/color
~/Repos/el/gnus/lisp/format-spec hides /home/horn/Repos/el/emacs/lisp/format-spec
~/Repos/el/gnus/lisp/password-cache hides /home/horn/Repos/el/emacs/lisp/password-cache
~/Repos/el/gnus/lisp/hex-util hides /home/horn/Repos/el/emacs/lisp/hex-util
~/Repos/el/gnus/lisp/dns-mode hides /home/horn/Repos/el/emacs/lisp/textmodes/dns-mode
~/Repos/el/gnus/lisp/dig hides /home/horn/Repos/el/emacs/lisp/net/dig
~/Repos/el/gnus/lisp/hmac-md5 hides /home/horn/Repos/el/emacs/lisp/net/hmac-md5
~/Repos/el/gnus/lisp/ntlm hides /home/horn/Repos/el/emacs/lisp/net/ntlm
~/Repos/el/gnus/lisp/hmac-def hides /home/horn/Repos/el/emacs/lisp/net/hmac-def
~/Repos/el/gnus/lisp/rfc2104 hides /home/horn/Repos/el/emacs/lisp/net/rfc2104
~/Repos/el/gnus/lisp/sasl-ntlm hides /home/horn/Repos/el/emacs/lisp/net/sasl-ntlm
~/Repos/el/gnus/lisp/sasl-cram hides /home/horn/Repos/el/emacs/lisp/net/sasl-cram
~/Repos/el/gnus/lisp/dns hides /home/horn/Repos/el/emacs/lisp/net/dns
~/Repos/el/gnus/lisp/sasl hides /home/horn/Repos/el/emacs/lisp/net/sasl
~/Repos/el/gnus/lisp/tls hides /home/horn/Repos/el/emacs/lisp/net/tls
~/Repos/el/gnus/lisp/sasl-scram-rfc hides /home/horn/Repos/el/emacs/lisp/net/sasl-scram-rfc
~/Repos/el/gnus/lisp/netrc hides /home/horn/Repos/el/emacs/lisp/net/netrc
~/Repos/el/gnus/lisp/sasl-digest hides /home/horn/Repos/el/emacs/lisp/net/sasl-digest
~/Repos/el/gnus/lisp/uudecode hides /home/horn/Repos/el/emacs/lisp/mail/uudecode
~/Repos/el/gnus/lisp/binhex hides /home/horn/Repos/el/emacs/lisp/mail/binhex
~/Repos/el/gnus/lisp/hashcash hides /home/horn/Repos/el/emacs/lisp/mail/hashcash
~/Repos/el/gnus/lisp/canlock hides /home/horn/Repos/el/emacs/lisp/gnus/canlock
~/Repos/el/gnus/lisp/nneething hides /home/horn/Repos/el/emacs/lisp/gnus/nneething
~/Repos/el/gnus/lisp/mm-encode hides /home/horn/Repos/el/emacs/lisp/gnus/mm-encode
~/Repos/el/gnus/lisp/mm-util hides /home/horn/Repos/el/emacs/lisp/gnus/mm-util
~/Repos/el/gnus/lisp/rfc2047 hides /home/horn/Repos/el/emacs/lisp/gnus/rfc2047
~/Repos/el/gnus/lisp/nnml hides /home/horn/Repos/el/emacs/lisp/gnus/nnml
~/Repos/el/gnus/lisp/gnus-cus hides /home/horn/Repos/el/emacs/lisp/gnus/gnus-cus
~/Repos/el/gnus/lisp/gnus-range hides /home/horn/Repos/el/emacs/lisp/gnus/gnus-range
~/Repos/el/gnus/lisp/gnus-int hides /home/horn/Repos/el/emacs/lisp/gnus/gnus-int
~/Repos/el/gnus/lisp/gnus-cloud hides /home/horn/Repos/el/emacs/lisp/gnus/gnus-cloud
~/Repos/el/gnus/lisp/spam-stat hides /home/horn/Repos/el/emacs/lisp/gnus/spam-stat
~/Repos/el/gnus/lisp/nnmh hides /home/horn/Repos/el/emacs/lisp/gnus/nnmh
~/Repos/el/gnus/lisp/gnus-mlspl hides /home/horn/Repos/el/emacs/lisp/gnus/gnus-mlspl
~/Repos/el/gnus/lisp/deuglify hides /home/horn/Repos/el/emacs/lisp/gnus/deuglify
~/Repos/el/gnus/lisp/gnus-gravatar hides /home/horn/Repos/el/emacs/lisp/gnus/gnus-gravatar
~/Repos/el/gnus/lisp/nngateway hides /home/horn/Repos/el/emacs/lisp/gnus/nngateway
~/Repos/el/gnus/lisp/ietf-drums hides /home/horn/Repos/el/emacs/lisp/gnus/ietf-drums
~/Repos/el/gnus/lisp/mail-parse hides /home/horn/Repos/el/emacs/lisp/gnus/mail-parse
~/Repos/el/gnus/lisp/gnus-salt hides /home/horn/Repos/el/emacs/lisp/gnus/gnus-salt
~/Repos/el/gnus/lisp/nnimap hides /home/horn/Repos/el/emacs/lisp/gnus/nnimap
~/Repos/el/gnus/lisp/gnus-draft hides /home/horn/Repos/el/emacs/lisp/gnus/gnus-draft
~/Repos/el/gnus/lisp/mail-source hides /home/horn/Repos/el/emacs/lisp/gnus/mail-source
~/Repos/el/gnus/lisp/messcompat hides /home/horn/Repos/el/emacs/lisp/gnus/messcompat
~/Repos/el/gnus/lisp/pop3 hides /home/horn/Repos/el/emacs/lisp/gnus/pop3
~/Repos/el/gnus/lisp/nnmaildir hides /home/horn/Repos/el/emacs/lisp/gnus/nnmaildir
~/Repos/el/gnus/lisp/nnheader hides /home/horn/Repos/el/emacs/lisp/gnus/nnheader
~/Repos/el/gnus/lisp/gnus-cite hides /home/horn/Repos/el/emacs/lisp/gnus/gnus-cite
~/Repos/el/gnus/lisp/nndiary hides /home/horn/Repos/el/emacs/lisp/gnus/nndiary
~/Repos/el/gnus/lisp/gnus-diary hides /home/horn/Repos/el/emacs/lisp/gnus/gnus-diary
~/Repos/el/gnus/lisp/nnfolder hides /home/horn/Repos/el/emacs/lisp/gnus/nnfolder
~/Repos/el/gnus/lisp/gnus-art hides /home/horn/Repos/el/emacs/lisp/gnus/gnus-art
~/Repos/el/gnus/lisp/gnus-demon hides /home/horn/Repos/el/emacs/lisp/gnus/gnus-demon
~/Repos/el/gnus/lisp/mml-sec hides /home/horn/Repos/el/emacs/lisp/gnus/mml-sec
~/Repos/el/gnus/lisp/nnir hides /home/horn/Repos/el/emacs/lisp/gnus/nnir
~/Repos/el/gnus/lisp/mm-partial hides /home/horn/Repos/el/emacs/lisp/gnus/mm-partial
~/Repos/el/gnus/lisp/gnus-registry hides /home/horn/Repos/el/emacs/lisp/gnus/gnus-registry
~/Repos/el/gnus/lisp/gnus-icalendar hides /home/horn/Repos/el/emacs/lisp/gnus/gnus-icalendar
~/Repos/el/gnus/lisp/compface hides /home/horn/Repos/el/emacs/lisp/gnus/compface
~/Repos/el/gnus/lisp/gnus-fun hides /home/horn/Repos/el/emacs/lisp/gnus/gnus-fun
~/Repos/el/gnus/lisp/gnus-start hides /home/horn/Repos/el/emacs/lisp/gnus/gnus-start
~/Repos/el/gnus/lisp/smiley hides /home/horn/Repos/el/emacs/lisp/gnus/smiley
~/Repos/el/gnus/lisp/gnus-picon hides /home/horn/Repos/el/emacs/lisp/gnus/gnus-picon
~/Repos/el/gnus/lisp/spam-report hides /home/horn/Repos/el/emacs/lisp/gnus/spam-report
~/Repos/el/gnus/lisp/nntp hides /home/horn/Repos/el/emacs/lisp/gnus/nntp
~/Repos/el/gnus/lisp/nnnil hides /home/horn/Repos/el/emacs/lisp/gnus/nnnil
~/Repos/el/gnus/lisp/nndir hides /home/horn/Repos/el/emacs/lisp/gnus/nndir
~/Repos/el/gnus/lisp/gnus-srvr hides /home/horn/Repos/el/emacs/lisp/gnus/gnus-srvr
~/Repos/el/gnus/lisp/smime hides /home/horn/Repos/el/emacs/lisp/gnus/smime
~/Repos/el/gnus/lisp/nnvirtual hides /home/horn/Repos/el/emacs/lisp/gnus/nnvirtual
~/Repos/el/gnus/lisp/gnus-notifications hides /home/horn/Repos/el/emacs/lisp/gnus/gnus-notifications
~/Repos/el/gnus/lisp/nnspool hides /home/horn/Repos/el/emacs/lisp/gnus/nnspool
~/Repos/el/gnus/lisp/gnus-group hides /home/horn/Repos/el/emacs/lisp/gnus/gnus-group
~/Repos/el/gnus/lisp/gnus-bcklg hides /home/horn/Repos/el/emacs/lisp/gnus/gnus-bcklg
~/Repos/el/gnus/lisp/gnus-util hides /home/horn/Repos/el/emacs/lisp/gnus/gnus-util
~/Repos/el/gnus/lisp/gnus-sieve hides /home/horn/Repos/el/emacs/lisp/gnus/gnus-sieve
~/Repos/el/gnus/lisp/nndraft hides /home/horn/Repos/el/emacs/lisp/gnus/nndraft
~/Repos/el/gnus/lisp/nnagent hides /home/horn/Repos/el/emacs/lisp/gnus/nnagent
~/Repos/el/gnus/lisp/gnus-spec hides /home/horn/Repos/el/emacs/lisp/gnus/gnus-spec
~/Repos/el/gnus/lisp/gnus-bookmark hides /home/horn/Repos/el/emacs/lisp/gnus/gnus-bookmark
~/Repos/el/gnus/lisp/mml1991 hides /home/horn/Repos/el/emacs/lisp/gnus/mml1991
~/Repos/el/gnus/lisp/rfc2231 hides /home/horn/Repos/el/emacs/lisp/gnus/rfc2231
~/Repos/el/gnus/lisp/yenc hides /home/horn/Repos/el/emacs/lisp/gnus/yenc
~/Repos/el/gnus/lisp/gnus-undo hides /home/horn/Repos/el/emacs/lisp/gnus/gnus-undo
~/Repos/el/gnus/lisp/ecomplete hides /home/horn/Repos/el/emacs/lisp/gnus/ecomplete
~/Repos/el/gnus/lisp/legacy-gnus-agent hides /home/horn/Repos/el/emacs/lisp/gnus/legacy-gnus-agent
~/Repos/el/gnus/lisp/utf7 hides /home/horn/Repos/el/emacs/lisp/gnus/utf7
~/Repos/el/gnus/lisp/rtree hides /home/horn/Repos/el/emacs/lisp/gnus/rtree
~/Repos/el/gnus/lisp/gnus-uu hides /home/horn/Repos/el/emacs/lisp/gnus/gnus-uu
~/Repos/el/gnus/lisp/gnus-ml hides /home/horn/Repos/el/emacs/lisp/gnus/gnus-ml
~/Repos/el/gnus/lisp/sieve hides /home/horn/Repos/el/emacs/lisp/gnus/sieve
~/Repos/el/gnus/lisp/gnus hides /home/horn/Repos/el/emacs/lisp/gnus/gnus
~/Repos/el/gnus/lisp/mml hides /home/horn/Repos/el/emacs/lisp/gnus/mml
~/Repos/el/gnus/lisp/message hides /home/horn/Repos/el/emacs/lisp/gnus/message
~/Repos/el/gnus/lisp/mml-smime hides /home/horn/Repos/el/emacs/lisp/gnus/mml-smime
~/Repos/el/gnus/lisp/gnus-eform hides /home/horn/Repos/el/emacs/lisp/gnus/gnus-eform
~/Repos/el/gnus/lisp/gnus-agent hides /home/horn/Repos/el/emacs/lisp/gnus/gnus-agent
~/Repos/el/gnus/lisp/gnus-logic hides /home/horn/Repos/el/emacs/lisp/gnus/gnus-logic
~/Repos/el/gnus/lisp/mm-extern hides /home/horn/Repos/el/emacs/lisp/gnus/mm-extern
~/Repos/el/gnus/lisp/nndoc hides /home/horn/Repos/el/emacs/lisp/gnus/nndoc
~/Repos/el/gnus/lisp/sieve-manage hides /home/horn/Repos/el/emacs/lisp/gnus/sieve-manage
~/Repos/el/gnus/lisp/mm-decode hides /home/horn/Repos/el/emacs/lisp/gnus/mm-decode
~/Repos/el/gnus/lisp/starttls hides /home/horn/Repos/el/emacs/lisp/gnus/starttls
~/Repos/el/gnus/lisp/gnus-dired hides /home/horn/Repos/el/emacs/lisp/gnus/gnus-dired
~/Repos/el/gnus/lisp/nnbabyl hides /home/horn/Repos/el/emacs/lisp/gnus/nnbabyl
~/Repos/el/gnus/lisp/nnmbox hides /home/horn/Repos/el/emacs/lisp/gnus/nnmbox
~/Repos/el/gnus/lisp/gnus-win hides /home/horn/Repos/el/emacs/lisp/gnus/gnus-win
~/Repos/el/gnus/lisp/gnus-async hides /home/horn/Repos/el/emacs/lisp/gnus/gnus-async
~/Repos/el/gnus/lisp/mm-url hides /home/horn/Repos/el/emacs/lisp/gnus/mm-url
~/Repos/el/gnus/lisp/gnus-html hides /home/horn/Repos/el/emacs/lisp/gnus/gnus-html
~/Repos/el/gnus/lisp/gssapi hides /home/horn/Repos/el/emacs/lisp/gnus/gssapi
~/Repos/el/gnus/lisp/mml2015 hides /home/horn/Repos/el/emacs/lisp/gnus/mml2015
~/Repos/el/gnus/lisp/nnrss hides /home/horn/Repos/el/emacs/lisp/gnus/nnrss
~/Repos/el/gnus/lisp/gnus-mh hides /home/horn/Repos/el/emacs/lisp/gnus/gnus-mh
~/Repos/el/gnus/lisp/gnus-sum hides /home/horn/Repos/el/emacs/lisp/gnus/gnus-sum
~/Repos/el/gnus/lisp/nnweb hides /home/horn/Repos/el/emacs/lisp/gnus/nnweb
~/Repos/el/gnus/lisp/mail-prsvr hides /home/horn/Repos/el/emacs/lisp/gnus/mail-prsvr
~/Repos/el/gnus/lisp/nnmairix hides /home/horn/Repos/el/emacs/lisp/gnus/nnmairix
~/Repos/el/gnus/lisp/plstore hides /home/horn/Repos/el/emacs/lisp/gnus/plstore
~/Repos/el/gnus/lisp/rfc2045 hides /home/horn/Repos/el/emacs/lisp/gnus/rfc2045
~/Repos/el/gnus/lisp/gnus-msg hides /home/horn/Repos/el/emacs/lisp/gnus/gnus-msg
~/Repos/el/gnus/lisp/spam-wash hides /home/horn/Repos/el/emacs/lisp/gnus/spam-wash
~/Repos/el/gnus/lisp/gnus-score hides /home/horn/Repos/el/emacs/lisp/gnus/gnus-score
~/Repos/el/gnus/lisp/mm-uu hides /home/horn/Repos/el/emacs/lisp/gnus/mm-uu
~/Repos/el/gnus/lisp/spam hides /home/horn/Repos/el/emacs/lisp/gnus/spam
~/Repos/el/gnus/lisp/mm-view hides /home/horn/Repos/el/emacs/lisp/gnus/mm-view
~/Repos/el/gnus/lisp/sieve-mode hides /home/horn/Repos/el/emacs/lisp/gnus/sieve-mode
~/Repos/el/gnus/lisp/html2text hides /home/horn/Repos/el/emacs/lisp/gnus/html2text
~/Repos/el/gnus/lisp/gnus-ems hides /home/horn/Repos/el/emacs/lisp/gnus/gnus-ems
~/Repos/el/gnus/lisp/registry hides /home/horn/Repos/el/emacs/lisp/gnus/registry
~/Repos/el/gnus/lisp/auth-source hides /home/horn/Repos/el/emacs/lisp/gnus/auth-source
~/Repos/el/gnus/lisp/gravatar hides /home/horn/Repos/el/emacs/lisp/gnus/gravatar
~/Repos/el/gnus/lisp/flow-fill hides /home/horn/Repos/el/emacs/lisp/gnus/flow-fill
~/Repos/el/gnus/lisp/gmm-utils hides /home/horn/Repos/el/emacs/lisp/gnus/gmm-utils
~/Repos/el/gnus/lisp/mailcap hides /home/horn/Repos/el/emacs/lisp/gnus/mailcap
~/Repos/el/gnus/lisp/gnus-delay hides /home/horn/Repos/el/emacs/lisp/gnus/gnus-delay
~/Repos/el/gnus/lisp/mm-bodies hides /home/horn/Repos/el/emacs/lisp/gnus/mm-bodies
~/Repos/el/gnus/lisp/mm-archive hides /home/horn/Repos/el/emacs/lisp/gnus/mm-archive
~/Repos/el/gnus/lisp/rfc1843 hides /home/horn/Repos/el/emacs/lisp/gnus/rfc1843
~/Repos/el/gnus/lisp/gnus-kill hides /home/horn/Repos/el/emacs/lisp/gnus/gnus-kill
~/Repos/el/gnus/lisp/qp hides /home/horn/Repos/el/emacs/lisp/gnus/qp
~/Repos/el/gnus/lisp/score-mode hides /home/horn/Repos/el/emacs/lisp/gnus/score-mode
~/Repos/el/gnus/lisp/gnus-topic hides /home/horn/Repos/el/emacs/lisp/gnus/gnus-topic
~/Repos/el/gnus/lisp/gnus-cache hides /home/horn/Repos/el/emacs/lisp/gnus/gnus-cache
~/Repos/el/gnus/lisp/nnmail hides /home/horn/Repos/el/emacs/lisp/gnus/nnmail
~/Repos/el/gnus/lisp/gnus-vm hides /home/horn/Repos/el/emacs/lisp/gnus/gnus-vm
~/Repos/el/gnus/lisp/gnus-sync hides /home/horn/Repos/el/emacs/lisp/gnus/gnus-sync
~/Repos/el/gnus/lisp/nnoo hides /home/horn/Repos/el/emacs/lisp/gnus/nnoo
~/Repos/el/gnus/lisp/nnregistry hides /home/horn/Repos/el/emacs/lisp/gnus/nnregistry
~/Repos/el/gnus/lisp/gnus-dup hides /home/horn/Repos/el/emacs/lisp/gnus/gnus-dup
~/Repos/el/gnus/lisp/parse-time hides /home/horn/Repos/el/emacs/lisp/calendar/parse-time
~/Repos/el/gnus/lisp/time-date hides /home/horn/Repos/el/emacs/lisp/calendar/time-date

Features:
(shadow emacsbug sendmail rcirc-color rcirc-controls rcirc-late-fix
rcirc bug-reference reftex-sel debug reftex-ref reftex-parse reftex-cite
pdf-sync pdf-annot pdf-outline pdf-links pdf-history texmathp preview
prv-emacs auto-dictionary flyspell ispell tex-buf reftex-dcr reftex-auc
reftex reftex-vars font-latex latex tex-style tex dbus crm tex-mode
latexenc pkg-info lisp-mnt epl mm-archive magit-key-mode magit view
git-rebase-mode git-commit-mode log-edit pcvs-util add-log autorevert
filenotify hippie-exp color org-rmail org-mhe org-irc org-info org-gnus
org-docview doc-view org-bibtex bibtex org-bbdb org-w3m swiper sort
smiley gnus-cite gnus-async gnus-bcklg gnus-ml misearch multi-isearch
winner filecache vc vc-dispatcher vc-git diff-mode cus-theme eieio-opt
speedbar sb-image ezimage dframe nndraft nnmh rot13 utf-7 nnml nnnil
gnus-agent gnus-srvr gnus-score score-mode nnvirtual gnus-cache
gnus-demon nntp spam spam-stat gnus-uu yenc gnus-msg gnus-gravatar
mail-extr gravatar gnus-topic nnir gnus-registry registry eieio-base
th-private gnutls network-stream nsm starttls url-http url-gw url-auth
hl-line company-files company-oddmuse company-keywords company-etags
company-gtags company-dabbrev-code company-dabbrev company-capf
company-cmake company-xcode company-clang company-semantic company-eclim
company-template company-css company-nxml company-bbdb
highlight-parentheses company stratego-mode greql-mode tg-mode generic
preview-latex tex-site auto-loads cider tramp-sh cider-debug cider-mode
cider-repl cider-eldoc cider-interaction arc-mode archive-mode cider-doc
org-table cider-test cider-stacktrace cider-client nrepl-client queue
cider-util ewoc etags xref clojure-mode paredit aggressive-indent names
edebug epa-file epa epg rdictcc google-contacts-message google-contacts
derived url-cache google-oauth google-contacts-gnus gnus-art mm-uu
mml2015 mm-view mml-smime smime dig gnus-sum gnus-group gnus-undo
gnus-start gnus-cloud nnimap nnmail mail-source tls utf7 netrc nnoo
parse-time gnus-spec gnus-int gnus-range gnus-win gnus gnus-ems
gnus-compat nnheader em-term term ehelp esh-opt esh-ext esh-util
highlight-symbol boxquote rect ecomplete message rfc822 mml mml-sec
mm-decode mm-bodies mm-encode mail-parse rfc2231 rfc2047 rfc2045
ietf-drums mailabbrev mail-utils gmm-utils mailheader edit-server server
haskell-yas yasnippet help-mode cl disp-table pdf-occur tablist
tablist-filter semantic/wisent/comp semantic/wisent
semantic/wisent/wisent semantic/util-modes semantic/util semantic
semantic/tag semantic/lex semantic/fw mode-local cedet pdf-isearch
pdf-misc imenu pdf-tools cus-edit cus-start cus-load pdf-view jka-compr
pdf-cache pdf-info tq pdf-util image-mode browse-kill-ring recentf
tree-widget wid-edit helm-projectile helm-files image-dired tramp
tramp-compat tramp-loaddefs trampver shell dired-x dired-aux ffap
helm-tags helm-bookmark helm-adaptive helm-info bookmark pp
helm-external helm-net browse-url xml url url-proxy url-privacy
url-expand url-methods url-history url-cookie url-domsuf url-util
url-parse auth-source gnus-util mm-util mail-prsvr password-cache
url-vars mailcap helm-buffers helm-match-plugin helm-help helm-org org
org-macro org-footnote org-pcomplete pcomplete org-list org-faces
org-entities noutline outline org-version ob-emacs-lisp ob ob-tangle
ob-ref ob-lob ob-table ob-exp org-src ob-keys ob-comint ob-core ob-eval
org-compat org-macs org-loaddefs format-spec find-func cal-menu calendar
cal-loaddefs helm-grep helm-regexp helm-plugin grep helm-elscreen
helm-utils dired compile comint ansi-color ring helm-locate helm cl-macs
helm-source eieio-compat eieio eieio-core cl-generic byte-opt gv
bytecomp byte-compile cl-extra seq cconv projectile ibuf-ext ibuffer
dash thingatpt helm-config async-bytecomp async helm-aliases easy-mmode
iedit iedit-lib cap-words superword subword saveplace savehist paren ivy
delsel icomplete mb-depth smart-mode-line-respectful-theme
smart-mode-line-light-theme smart-mode-line rich-minority rx bs windmove
elec-pair edmacro kmacro cl-loaddefs cl-lib gnus-load subr-x pcase
tsdh-light-theme finder-inf info easymenu memory-usage-autoloads advice
help-fns package epg-config mule-util time-date tooltip eldoc electric
uniquify ediff-hook vc-hooks lisp-float-type mwheel x-win x-dnd tool-bar
dnd fontset image regexp-opt fringe tabulated-list newcomment elisp-mode
lisp-mode prog-mode register page menu-bar rfn-eshadow timer select
scroll-bar mouse jit-lock font-lock syntax facemenu font-core frame cham
georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao
korean japanese hebrew greek romanian slovak czech european ethiopic
indian cyrillic chinese case-table epa-hook jka-cmpr-hook help simple
abbrev minibuffer cl-preloaded nadvice loaddefs button faces cus-face
macroexp files text-properties overlay sha1 md5 base64 format env
code-pages mule custom widget hashtable-print-readable backquote
dbusbind gfilenotify dynamic-setting system-font-setting
font-render-setting move-toolbar gtk x-toolkit x multi-tty
make-network-process emacs)

Memory information:
((conses 16 1602223 179931)
 (symbols 48 67807 0)
 (miscs 40 2020 3774)
 (strings 32 263270 26363)
 (string-bytes 1 9086606)
 (vectors 16 77447)
 (vector-slots 8 2097120 172273)
 (floats 8 1915 1406)
 (intervals 56 144016 5164)
 (buffers 976 90)
 (heap 1024 130819 25998))

bug#20404: 25.0.50; Sometimes no fontification with jit-lock-defer-time

[Eli Zaretskii]

> Resent-Sender: help-debbugs@gnu.org
> From: Tassilo Horn <tsdh@gnu.org>
> Date: Wed, 22 Apr 2015 11:45:42 +0200
> 
> I use the settings
> 
>   (setq jit-lock-defer-time 0.0312
>         jit-lock-stealth-nice 0.117
>         jit-lock-stealth-time 0.23)
> 
> to speed up scrolling in large buffers with complex font-lock rules.
> That basically works nice.  However, quite often I'm faced with buffers
> that are not fontified before I do something, e.g., move point.
> 
> This frequently happens when installing/upgrading packages with the
> package manager.  That shows the buffer containing the byte-compliation
> messages with fontified warnings and errors.  Every second time or so,
> I'm faced with a completely unfontified buffer that gets fontified only
> after selecting the window showing the buffer.
> 
> Another and even reproducible case is `report-emacs-bug'.  It goes like
> this:
> 
>   1. emacs -Q
>   2. eval my above settings in *scratch*
>   3. M-x report-emacs-bug
> 
> The new message buffer is completely unfontified initially.  As soon as
> you move point, font-lock kicks in and the header lines get their proper
> fontification.  Or maybe it's the other way round, i.e., the buffer is
> created, then redisplayed, then the fontification takes place but no
> redisplay is performed thereafter.

Sounds like the idle timer that is started by jit-lock-defer-time
never runs after the buffer is displayed.  Could it be that it already
ran before the display?

bug#20404: 25.0.50; Sometimes no fontification with jit-lock-defer-time

[Tassilo Horn]

Eli Zaretskii <eliz@gnu.org> writes:

>> Another and even reproducible case is `report-emacs-bug'.  It goes
>> like this:
>> 
>>   1. emacs -Q
>>   2. eval my above settings in *scratch*
>>   3. M-x report-emacs-bug
>> 
>> The new message buffer is completely unfontified initially.  As soon
>> as you move point, font-lock kicks in and the header lines get their
>> proper fontification.  Or maybe it's the other way round, i.e., the
>> buffer is created, then redisplayed, then the fontification takes
>> place but no redisplay is performed thereafter.
>
> Sounds like the idle timer that is started by jit-lock-defer-time
> never runs after the buffer is displayed.

Indeed, that's the case.  Or wait, it eventually runs but much later
than `jit-lock-defer-time' defines.

> Could it be that it already ran before the display?

No.  The bug report buffer is displayed first, and then it takes two or
three seconds until the first jit-lock kicks in.  And then the buffer
will be redisplayed and appears fontified.

I tested using this code:

--8<---------------cut here---------------start------------->8---
(progn
  (setq jit-lock-defer-time 0.0312
        jit-lock-stealth-nice 0.117
        jit-lock-stealth-time 0.23)

  (require 'cl-lib)
  (defvar i 0)
  (advice-add 'jit-lock-deferred-fontify :before
              (lambda ()
                (message "DEFERRED FONTIFY %s" (cl-incf i)))))
--8<---------------cut here---------------end--------------->8---

Bye,
Tassilo

bug#20404: 25.0.50; Sometimes no fontification with jit-lock-defer-time

[Eli Zaretskii]

> From: Tassilo Horn <tsdh@gnu.org>
> Cc: 20404@debbugs.gnu.org
> Date: Wed, 22 Apr 2015 22:32:14 +0200
> 
> > Sounds like the idle timer that is started by jit-lock-defer-time
> > never runs after the buffer is displayed.
> 
> Indeed, that's the case.  Or wait, it eventually runs but much later
> than `jit-lock-defer-time' defines.
> 
> > Could it be that it already ran before the display?
> 
> No.  The bug report buffer is displayed first, and then it takes two or
> three seconds until the first jit-lock kicks in.  And then the buffer
> will be redisplayed and appears fontified.

So you are saying that something prevents the timer to run at the
prescribed time?  I suggest to add trace printf's in the code that
traverses the idle-timers' list, and see why this timer doesn't run on
time.

bug#20404: 25.0.50; Sometimes no fontification with jit-lock-defer-time

[Tassilo Horn]

Eli Zaretskii <eliz@gnu.org> writes:

>> > Sounds like the idle timer that is started by jit-lock-defer-time
>> > never runs after the buffer is displayed.
>> 
>> Indeed, that's the case.  Or wait, it eventually runs but much later
>> than `jit-lock-defer-time' defines.
>> 
>> > Could it be that it already ran before the display?
>> 
>> No.  The bug report buffer is displayed first, and then it takes two
>> or three seconds until the first jit-lock kicks in.  And then the
>> buffer will be redisplayed and appears fontified.
>
> So you are saying that something prevents the timer to run at the
> prescribed time?

That seems to be the case.

> I suggest to add trace printf's in the code that traverses the
> idle-timers' list, and see why this timer doesn't run on time.

That would be

  static struct timespec
  timer_check_2 (Lisp_Object timers, Lisp_Object idle_timers)

right?

So first I wanted to see if the deferred font-lock timer gets selected
as being ripe in the first place.  But I already failed with that; emacs
now dumps core.

--8<---------------cut here---------------start------------->8---
diff --git a/src/keyboard.c b/src/keyboard.c
index 068a47c..6231747 100644
--- a/src/keyboard.c
+++ b/src/keyboard.c
@@ -4419,6 +4419,8 @@ timer_check_2 (Lisp_Object timers, Lisp_Object idle_timers)
   Lisp_Object chosen_timer;
   struct gcpro gcpro1;
 
+  printf ("timer_check2 ()");
+  
   nexttime = invalid_timespec ();
 
   chosen_timer = Qnil;
@@ -4513,6 +4515,10 @@ timer_check_2 (Lisp_Object timers, Lisp_Object idle_timers)
 	  idle_timers = XCDR (idle_timers);
 	  difference = idle_timer_difference;
 	  ripe = idle_timer_ripe;
+	  if (ripe)
+	    {
+	      printf("Idle timer calling %s is ripe.", AREF (5, chosen_timer));
+	    }
 	}
 
       /* If timer is ripe, run it if it hasn't been run.  */
--8<---------------cut here---------------end--------------->8---

I think the problem is that the AREF returns the timer's function or
maybe the symbol whose function definition is the timer's function.  In
any case, that's not a char* required by printf's %s.  How do I get the
function's name in order to print that?

Bye,
Tassilo

bug#20404: 25.0.50; Sometimes no fontification with jit-lock-defer-time

[Eli Zaretskii]

> From: Tassilo Horn <tsdh@gnu.org>
> Cc: 20404@debbugs.gnu.org
> Date: Thu, 23 Apr 2015 07:59:15 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > So you are saying that something prevents the timer to run at the
> > prescribed time?
> 
> That seems to be the case.

It can also be that Emacs doesn't become idle for some reason.

> > I suggest to add trace printf's in the code that traverses the
> > idle-timers' list, and see why this timer doesn't run on time.
> 
> That would be
> 
>   static struct timespec
>   timer_check_2 (Lisp_Object timers, Lisp_Object idle_timers)
> 
> right?

Yes.

> +	  if (ripe)
> +	    {
> +	      printf("Idle timer calling %s is ripe.", AREF (5, chosen_timer));
> +	    }
>  	}
>  
>        /* If timer is ripe, run it if it hasn't been run.  */
> --8<---------------cut here---------------end--------------->8---
> 
> I think the problem is that the AREF returns the timer's function or
> maybe the symbol whose function definition is the timer's function.

Yes, you cannon printf a Lisp object with %s.

> How do I get the function's name in order to print that?

Try SDATA (SYMBOL_NAME (AREF (5, chosen_timer))).

bug#20404: 25.0.50; Sometimes no fontification with jit-lock-defer-time

[Tassilo Horn]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Tassilo Horn <tsdh@gnu.org>
>> Cc: 20404@debbugs.gnu.org
>> Date: Thu, 23 Apr 2015 07:59:15 +0200
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > So you are saying that something prevents the timer to run at the
>> > prescribed time?
>> 
>> That seems to be the case.
>
> It can also be that Emacs doesn't become idle for some reason.
>
>> > I suggest to add trace printf's in the code that traverses the
>> > idle-timers' list, and see why this timer doesn't run on time.
>> 
>> That would be
>> 
>>   static struct timespec
>>   timer_check_2 (Lisp_Object timers, Lisp_Object idle_timers)
>> 
>> right?
>
> Yes.
>
>> +	  if (ripe)
>> +	    {
>> +	      printf("Idle timer calling %s is ripe.", AREF (5, chosen_timer));
>> +	    }
>>  	}
>>  
>>        /* If timer is ripe, run it if it hasn't been run.  */
>> --8<---------------cut here---------------end--------------->8---
>> 
>> I think the problem is that the AREF returns the timer's function or
>> maybe the symbol whose function definition is the timer's function.
>
> Yes, you cannon printf a Lisp object with %s.
>
>> How do I get the function's name in order to print that?
>
> Try SDATA (SYMBOL_NAME (AREF (5, chosen_timer))).

That works although you have to add more checks for lambdas, etc.  So
now I use this:

--8<---------------cut here---------------start------------->8---
diff --git a/src/keyboard.c b/src/keyboard.c
index 068a47c..fe906b0 100644
--- a/src/keyboard.c
+++ b/src/keyboard.c
@@ -4410,6 +4410,24 @@ decode_timer (Lisp_Object timer, struct timespec *result)
    In that case we return 0 to indicate that a new timer_check_2 call
    should be done.  */
 
+void print_timer(Lisp_Object chosen_timer, const char* kind) {
+  const char* timer_fn_name = "<no function>";
+  const Lisp_Object fn = AREF (chosen_timer, 5);
+  if (fn) {
+    timer_fn_name = "<anon fn>";
+    if (SYMBOLP (fn)) {
+      const Lisp_Object sn = SYMBOL_NAME (fn);
+      if (sn) {
+	timer_fn_name = SDATA (sn);
+      }
+    }
+  }
+  printf("%s timer calling %s is ripe.\n",
+	 kind, timer_fn_name);
+}
+
+int myCounter = 0;
+
 static struct timespec
 timer_check_2 (Lisp_Object timers, Lisp_Object idle_timers)
 {
@@ -4419,6 +4437,8 @@ timer_check_2 (Lisp_Object timers, Lisp_Object idle_timers)
   Lisp_Object chosen_timer;
   struct gcpro gcpro1;
 
+  printf ("timer_check2 () %d\n", myCounter++);
+  
   nexttime = invalid_timespec ();
 
   chosen_timer = Qnil;
@@ -4506,6 +4526,10 @@ timer_check_2 (Lisp_Object timers, Lisp_Object idle_timers)
 	  timers = XCDR (timers);
 	  difference = timer_difference;
 	  ripe = timer_ripe;
+	  if (ripe)
+	    {
+	      print_timer (chosen_timer, "normal");
+	    }  
 	}
       else
 	{
@@ -4513,6 +4537,10 @@ timer_check_2 (Lisp_Object timers, Lisp_Object idle_timers)
 	  idle_timers = XCDR (idle_timers);
 	  difference = idle_timer_difference;
 	  ripe = idle_timer_ripe;
+	  if (ripe)
+	    {
+	      print_timer (chosen_timer, "idle");
+	    }
 	}
 
       /* If timer is ripe, run it if it hasn't been run.  */
--8<---------------cut here---------------end--------------->8---

And when doing `M-x report-emacs-bug RET bug summary RET`, that's the
output in the period in which the buffer is not fontified.

--8<---------------cut here---------------start------------->8---
normal timer calling blink-cursor-timer-function is ripe.
timer_check2 () 1329
timer_check2 () 1330
timer_check2 () 1331
timer_check2 () 1332
timer_check2 () 1333
timer_check2 () 1334
timer_check2 () 1335
timer_check2 () 1336
timer_check2 () 1337
timer_check2 () 1338
timer_check2 () 1339
timer_check2 () 1340
timer_check2 () 1341
timer_check2 () 1343
timer_check2 () 1344
timer_check2 () 1345
timer_check2 () 1346
timer_check2 () 1348
timer_check2 () 1349
timer_check2 () 1350
timer_check2 () 1351
timer_check2 () 1352
timer_check2 () 1353
timer_check2 () 1354
timer_check2 () 1355
timer_check2 () 1356
timer_check2 () 1357
timer_check2 () 1358
timer_check2 () 1359
timer_check2 () 1360
timer_check2 () 1361
timer_check2 () 1362
timer_check2 () 1363
timer_check2 () 1364
timer_check2 () 1365
timer_check2 () 1366
timer_check2 () 1367
idle timer calling jit-lock-deferred-fontify is ripe.
--8<---------------cut here---------------end--------------->8---

So as you can see, in that period there are many calls of timer_check2()
but none of them select some timer as being ripe.  Neither normal nor
idle timers.  That is, the problem isn't really about deferred
fontification but about timer's not being selected for execution.  That
is, for example the cursor doesn't blink, too, which is another idle
timer that's not run in the period.

Here's the output if I also `trace-redisplay'.

--8<---------------cut here---------------start------------->8---
redisplay_internal 0
trying display optimization 1
0x12b1c30 ( *Minibuf-1*): optimization 1
timer_check2 () 4941
timer_check2 () 4942
timer_check2 () 4943
timer_check2 () 4944
timer_check2 () 4945
timer_check2 () 4946
redisplay_preserve_echo_area (2)
redisplay_internal 0
0x12b0c20 (*unsent mail to bug-gnu-emacs@gnu.org*): same window start
0x12b0c20 (*unsent mail to bug-gnu-emacs@gnu.org*): 1
0x12f8d60 (*Bug Help*): same window start
0x12f8d60 (*Bug Help*): 1
timer_check2 () 4947
redisplay_internal 0
0x12b0c20 (*unsent mail to bug-gnu-emacs@gnu.org*): same window start
0x12b0c20 (*unsent mail to bug-gnu-emacs@gnu.org*): 1
0x12f8d60 (*Bug Help*): same window start
0x12f8d60 (*Bug Help*): 1
timer_check2 () 4948
timer_check2 () 4949
timer_check2 () 4950
timer_check2 () 4951
timer_check2 () 4952
timer_check2 () 4953
timer_check2 () 4954
timer_check2 () 4955
timer_check2 () 4956
timer_check2 () 4957
timer_check2 () 4958
timer_check2 () 4959
timer_check2 () 4960
timer_check2 () 4961
timer_check2 () 4962
timer_check2 () 4963
timer_check2 () 4964
timer_check2 () 4965
timer_check2 () 4966
timer_check2 () 4967
timer_check2 () 4968
timer_check2 () 4969
timer_check2 () 4970
timer_check2 () 4971
timer_check2 () 4972
timer_check2 () 4973
timer_check2 () 4974
timer_check2 () 4975
timer_check2 () 4976
timer_check2 () 4977
redisplay_internal 0
timer_check2 () 4978
timer_check2 () 4979
timer_check2 () 4980
timer_check2 () 4981
timer_check2 () 4982
timer_check2 () 4983
timer_check2 () 4984
redisplay_internal 0
timer_check2 () 4985
timer_check2 () 4986
timer_check2 () 4987
timer_check2 () 4988
idle timer calling jit-lock-deferred-fontify is ripe.
timer_check2 () 4989
timer_check2 () 4990
timer_check2 () 4991
redisplay_preserve_echo_area (2)
redisplay_internal 0
0x12b0c20 (*unsent mail to bug-gnu-emacs@gnu.org*): same window start
0x12b0c20 (*unsent mail to bug-gnu-emacs@gnu.org*): 1
--8<---------------cut here---------------end--------------->8---

HTH,
Tassilo

bug#20404: 25.0.50; Sometimes no fontification with jit-lock-defer-time

[Alan Mackenzie]

In article <mailman.1344.1429696032.904.bug-gnu-emacs@gnu.org> you wrote:

[ .... ]

> Fun anecdote: I really have a pretty fast machine but still scrolling,
> e.g., buffer.c, by pressing and holding `C-v' causes visible scrolling
> for 1-2 seconds, and then the display freezes.  Emacs burns one core,
> and the next visible change I can observe is that the end of the buffer
> is displayed after the ~6 seconds where emacs seems frozen.  That lead
> me to looking for solutions, and deferred JIT lock and stealth lock
> seemed like a good idea (and they are!).

There's a solution (or, depending on your point of view, a workaround)
for this.  You're using Emacs-25, so set fast-but-imprecise-scrolling to
t, or customize it (it's in customisation group "scrolling").  What it
does is only to fontify screen-worths it's about to display, rather than
fontifying the entire buffer portion that is scrolled over.  The
disadvantage is that if you're using faces with different heights, the
exact place the scrolling ends up at might not be quite correct, but if
you're holding down C-v anyway, this might not matter too much.

[ .... ]

> In GNU Emacs 25.0.50.12 (x86_64-unknown-linux-gnu, GTK+ Version 3.16.2)
> of 2015-04-21 on thinkpad-t440p

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#20404: 25.0.50; Sometimes no fontification with jit-lock-defer-time

[Stefan Monnier]

[ OFFTOPIC: the bug report is not about performance but about
  a functional bug in jit-lock-defer.  ]

> There's a solution (or, depending on your point of view, a workaround)
> for this.  You're using Emacs-25, so set fast-but-imprecise-scrolling to
> t, or customize it (it's in customisation group "scrolling").

FWIW, on my machine, this doesn't make much difference: a single
"font-lock one screen's worth" takes much too long, so after that one
screen is displayed, Emacs takes a while catching up (with no display at
all in the mean time).

That's why after you installed your change, I installed the one I had
suggested (which tweaks the decision about when to skip redisplay and
when to defer font-lock and relies on using jit-lock-defer-time).
In my tests, it worked significantly better (was able to keep up).

So I recommend you try jit-lock-defer-time set to 0 instead of using
fast-but-imprecise-scrolling, and see if you like the
resulting behavior.

IIUC if Emacs can *almost* keep up, then fast-but-imprecise-scrolling
might be just enough to let it keep up, in which case the behavior may
look be better than with jit-lock-defer-time (because the text is never
displayed unfontified). But if the repeat rate is faster than that, then
jit-lock-defer-time should give noticeably better results.


        Stefan

bug#20404: 25.0.50; Sometimes no fontification with jit-lock-defer-time

[Alan Mackenzie]

Hello, Stefan.

On Wed, Apr 22, 2015 at 01:37:13PM -0400, Stefan Monnier wrote:
> [ OFFTOPIC: the bug report is not about performance but about
>   a functional bug in jit-lock-defer.  ]

> > There's a solution (or, depending on your point of view, a workaround)
> > for this.  You're using Emacs-25, so set fast-but-imprecise-scrolling to
> > t, or customize it (it's in customisation group "scrolling").

> FWIW, on my machine, this doesn't make much difference: a single
> "font-lock one screen's worth" takes much too long, so after that one
> screen is displayed, Emacs takes a while catching up (with no display at
> all in the mean time).

How long is "too long"?  Do you mean that, with
fast-but-imprecise-scrolling non-nil, jit-lock-defer-time nil, holding
down C-v, that there is no redisplay for several seconds?  That would
be suggesting that the time to scroll one screen (without fontification)
is longer than your auto-repeat interval, which sounds implausible.

> That's why after you installed your change, I installed the one I had
> suggested (which tweaks the decision about when to skip redisplay and
> when to defer font-lock and relies on using jit-lock-defer-time).
> In my tests, it worked significantly better (was able to keep up).

> So I recommend you try jit-lock-defer-time set to 0 instead of using
> fast-but-imprecise-scrolling, and see if you like the
> resulting behavior.

No, I don't.  ;-)  What I see is somewhat jerky scrolling, mainly
unfontified, but with some fontification flashing on each screen some
short time (?10-100 milliseconds) before the next scroll.

What I see with fast-but-imprecise-scrolling is somewhat jerky scrolling,
but each displayed screen being fully fontified.  The delay between
drawing the screens is never more than about half a second, mostly less
than that, and the delay between releasing C-v and the screen stabilising
is likewise no more than half a second.

Also, when I attempt to disable jit-lock-defer-time (through the
customisation interface) the jit-lock-defer-timer keeps running, and the
"defer" mechanism keeps running with it.  This seems worth a bug report
in its own right.

> IIUC if Emacs can *almost* keep up, then fast-but-imprecise-scrolling
> might be just enough to let it keep up, in which case the behavior may
> look be better than with jit-lock-defer-time (because the text is never
> displayed unfontified). But if the repeat rate is faster than that, then
> jit-lock-defer-time should give noticeably better results.

I don't understand how what you're seeing is so bad.  I thought you had a
powerful workstation, a class above a typical PC, and that you had your
auto-repeat rate at a conservative figure (25 or 30 per second) rather
than the insane rate (~40 per second) I have.  I have a 5 year old
machine, not a blazingly fast super-modern one, and my window is 64 lines
deep.

fast-but-imprecise-scrolling doesn't require Emacs almost to keep up.  It
merely prevents a redisplay being directly triggered by the auto-repeated
scrolling, instead allowing a redisplay only when there is no input
waiting.  On my machine, the scrolling easily keeps up with the ~25ms
auto-repeat time.

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#20404: 25.0.50; Sometimes no fontification with jit-lock-defer-time

[Stefan Monnier]

> How long is "too long"?  Do you mean that, with
> fast-but-imprecise-scrolling non-nil, jit-lock-defer-time nil, holding
> down C-v, that there is no redisplay for several seconds?

No, it's usually shorter than "several seconds", but it's only refreshed
maybe twice a second (I use a repeat rate of 40/s) with a few waits that
are longer than that.
In contrast with jit-lock-defer-time set to 0, the scrolling is mostly
smooth for me.

> That would be suggesting that the time to scroll one screen (without
> fontification) is longer than your auto-repeat interval, which
> sounds implausible.

It's not the case, indeed, no.
[ Though, if I push the test with a sufficiently tall window (1600
vertical pixels, and a small enough font), I do get it to skip some
redisplays even with font-lock-mode off.  But that's also because my
processor is slow ("Atom-Z530 @ 1.60GHz") and I compile with all kinds
of extra debugging code.  ]

> No, I don't.  ;-)  What I see is somewhat jerky scrolling, mainly
> unfontified, but with some fontification flashing on each screen some
> short time (?10-100 milliseconds) before the next scroll.

Hmm... interesting.  I don't see the "some fontification flashing": for
me it scrolls unfontified until I stop scrolling (at which point it's
shown immediately fontified).
I guess in your case, your machine is just fast enough to perform the
fontification every few steps, whereas mine is always "trying to catch up".

Maybe for you to see the same behavior as I see, you'd have to use
a non-0 setting for jit-lock-defer-time to try and prevent the
occasional deferred fontification (probably using a defer-time that's just
above the repeat-time of your keyboard), but my hack (which basically
disables the deferring when there's no pending input) would need to be
tweaked (it treats timer==0 specially).

> What I see with fast-but-imprecise-scrolling is somewhat jerky scrolling,
> but each displayed screen being fully fontified.

Yes, the always-fontified display is clearly the obvious advantage of
fast-but-imprecise-scrolling.  I'm personally bothered by the jerkiness
of the scrolling, but if jit-lock-defer-time also gives you jerky
scrolling, then clearly it's a loser.

> Also, when I attempt to disable jit-lock-defer-time (through the
> customisation interface) the jit-lock-defer-timer keeps running, and the
> "defer" mechanism keeps running with it.  This seems worth a bug report
> in its own right.

Yup, sounds like it deserves its own bug report.

> I don't understand how what you're seeing is so bad.  I thought you had a
> powerful workstation, a class above a typical PC, and that you had your
> auto-repeat rate at a conservative figure (25 or 30 per second) rather
> than the insane rate (~40 per second) I have.

I have various machines, but my desktops are a Fit-PC2 (at work) and
a AMD E-350 (at home).  So "powerful workstation" is not exactly the
term I would employ.
This said, "insane" is not the term I'd use to describe 40 repetitions
per seconds either.

> I have a 5 year old machine, not a blazingly fast super-modern one,
> and my window is 64 lines deep.

Right windows are typically between 65 and 80 lines tall, as well.


        Stefan

bug#20404: 25.0.50; Sometimes no fontification with jit-lock-defer-time

[Tassilo Horn]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

> [ OFFTOPIC: the bug report is not about performance but about
>   a functional bug in jit-lock-defer.  ]
>
>> There's a solution (or, depending on your point of view, a workaround)
>> for this.  You're using Emacs-25, so set fast-but-imprecise-scrolling to
>> t, or customize it (it's in customisation group "scrolling").
>
> FWIW, on my machine, this doesn't make much difference: a single
> "font-lock one screen's worth" takes much too long, so after that one
> screen is displayed, Emacs takes a while catching up (with no display
> at all in the mean time).

Setting `fast-but-imprecise-scrolling' to t works quite nicely for me.
Scrolling is indeed a bit bumpy but there are no noticeable hangs.  And
it's nice that the text scrolls by fontified.

> So I recommend you try jit-lock-defer-time set to 0 instead of using
> fast-but-imprecise-scrolling, and see if you like the resulting
> behavior.

This doesn't work for me.  I get the same display freezes as with
`jit-lock-defer-time' set to nil.  I guess that's not how it's indended
to be.  Maybe keyboard events don't count as pending input?

Bye,
Tassilo

bug#20404: 25.0.50; Sometimes no fontification with jit-lock-defer-time

[Eli Zaretskii]

> From: Tassilo Horn <tsdh@gnu.org>
> Date: Wed, 22 Apr 2015 22:10:21 +0200
> Cc: Alan Mackenzie <acm@muc.de>, 20404@debbugs.gnu.org
> 
> Maybe keyboard events don't count as pending input?

??? Of course they do.

bug#20404: 25.0.50; Sometimes no fontification with jit-lock-defer-time

[Tassilo Horn]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Tassilo Horn <tsdh@gnu.org>
>> Date: Wed, 22 Apr 2015 22:10:21 +0200
>> Cc: Alan Mackenzie <acm@muc.de>, 20404@debbugs.gnu.org
>> 
>> Maybe keyboard events don't count as pending input?
>
> ??? Of course they do.

Well, then jit-lock-defer-time == 0 doesn't work here as documented.  It
doesn't seem to defer anything although there must be dozens or hundreds
of pending keyboard events when I press and hold `C-v'.

Bye,
Tassilo

bug#20404: 25.0.50; Sometimes no fontification with jit-lock-defer-time

[Eli Zaretskii]

> From: Tassilo Horn <tsdh@gnu.org>
> Cc: monnier@IRO.UMontreal.CA,  acm@muc.de,  20404@debbugs.gnu.org
> Date: Thu, 23 Apr 2015 06:15:25 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: Tassilo Horn <tsdh@gnu.org>
> >> Date: Wed, 22 Apr 2015 22:10:21 +0200
> >> Cc: Alan Mackenzie <acm@muc.de>, 20404@debbugs.gnu.org
> >> 
> >> Maybe keyboard events don't count as pending input?
> >
> > ??? Of course they do.
> 
> Well, then jit-lock-defer-time == 0 doesn't work here as documented.  It
> doesn't seem to defer anything although there must be dozens or hundreds
> of pending keyboard events when I press and hold `C-v'.

I think I explained this back when there was a very long discussion of
Alan's fast-but-imprecise-scrolling patch.

bug#20404: 25.0.50; Sometimes no fontification with jit-lock-defer-time

[Stefan Monnier]

> I think I explained this back when there was a very long discussion of
> Alan's fast-but-imprecise-scrolling patch.

What was it?  That input-pending-p returns nil even though there are
lots of pending events (because they haven't been processed enough for
Emacs to realize they're there)?  Could you make a bug report about this
issue, describing under which circumstances this happens?  Or do we have
one already?


        Stefan

bug#20404: 25.0.50; Sometimes no fontification with jit-lock-defer-time

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Cc: Tassilo Horn <tsdh@gnu.org>, acm@muc.de, 20404@debbugs.gnu.org
> Date: Thu, 23 Apr 2015 09:40:08 -0400
> 
> > I think I explained this back when there was a very long discussion of
> > Alan's fast-but-imprecise-scrolling patch.
> 
> What was it?  That input-pending-p returns nil even though there are
> lots of pending events (because they haven't been processed enough for
> Emacs to realize they're there)?  Could you make a bug report about this
> issue, describing under which circumstances this happens?  Or do we have
> one already?

Maybe we are not talking about the same thing.  I meant these
messages:

  http://lists.gnu.org/archive/html/emacs-devel/2014-10/msg00705.html
  http://lists.gnu.org/archive/html/emacs-devel/2014-10/msg01039.html
  http://lists.gnu.org/archive/html/emacs-devel/2014-10/msg01055.html

I don't see the special case of zero discussed back there, so perhaps
this is a different problem.

bug#20404: 25.0.50; Sometimes no fontification with jit-lock-defer-time

[Stefan Monnier]

>> What was it?  That input-pending-p returns nil even though there are
>> lots of pending events (because they haven't been processed enough for
>> Emacs to realize they're there)?  Could you make a bug report about this
>> issue, describing under which circumstances this happens?  Or do we have
>> one already?
> Maybe we are not talking about the same thing.  I meant these
> messages:
>   http://lists.gnu.org/archive/html/emacs-devel/2014-10/msg00705.html

This message of yours seems to be talking about the problem I'm
referring to above.  At least, the way I read it, it seems to say that
`input-pending-p' can return nil even if there is pending input
(because that pending input is still in the system's input queue).
If that's the case, we should have a bug report about that.

>   http://lists.gnu.org/archive/html/emacs-devel/2014-10/msg01039.html
>   http://lists.gnu.org/archive/html/emacs-devel/2014-10/msg01055.html

These talk about something else, which is partly outdated.

> I don't see the special case of zero discussed back there, so perhaps
> this is a different problem.

That's because the code has changed since.  Now redisplay (which happens
after running a command) is skipped based on the value of
`input-pending-p' at the beginning of the *command*, rather than based
on its value at the beginning of redisplay.  And jit-lock.el has been
changed to use (not (input-pending-p)), but only if jit-lock-defer-time
is 0, so the special case for 0 is new.


        Stefan

bug#20404: 25.0.50; Sometimes no fontification with jit-lock-defer-time

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: tsdh@gnu.org,  acm@muc.de,  20404@debbugs.gnu.org
> Date: Thu, 23 Apr 2015 12:23:02 -0400
> 
> >> What was it?  That input-pending-p returns nil even though there are
> >> lots of pending events (because they haven't been processed enough for
> >> Emacs to realize they're there)?  Could you make a bug report about this
> >> issue, describing under which circumstances this happens?  Or do we have
> >> one already?
> > Maybe we are not talking about the same thing.  I meant these
> > messages:
> >   http://lists.gnu.org/archive/html/emacs-devel/2014-10/msg00705.html
> 
> This message of yours seems to be talking about the problem I'm
> referring to above.  At least, the way I read it, it seems to say that
> `input-pending-p' can return nil even if there is pending input
> (because that pending input is still in the system's input queue).
> If that's the case, we should have a bug report about that.
> 
> >   http://lists.gnu.org/archive/html/emacs-devel/2014-10/msg01039.html
> >   http://lists.gnu.org/archive/html/emacs-devel/2014-10/msg01055.html
> 
> These talk about something else, which is partly outdated.
> 
> > I don't see the special case of zero discussed back there, so perhaps
> > this is a different problem.
> 
> That's because the code has changed since.  Now redisplay (which happens
> after running a command) is skipped based on the value of
> `input-pending-p' at the beginning of the *command*, rather than based
> on its value at the beginning of redisplay.  And jit-lock.el has been
> changed to use (not (input-pending-p)), but only if jit-lock-defer-time
> is 0, so the special case for 0 is new.

FWIW, when I set jit-lock-defer-time to zero and lean on the PageDown
key, I see somewhat clunky scrolling with no fontifications, until I
release the key.

But fontification inside the scrolling code itself still happens, I
think.

bug#20404: 25.0.50; Sometimes no fontification with jit-lock-defer-time

[Stefan Monnier]

> FWIW, when I set jit-lock-defer-time to zero and lean on the PageDown
> key, I see somewhat clunky scrolling with no fontifications, until I
> release the key.
> But fontification inside the scrolling code itself still happens, I
> think.

Hmm... while I guess it's still possible, it normally shouldn't happen:
if the fontification takes place during the scrolling itself, then the
next redisplay should not be skipped (and hence you should see fontified
text rather than unfontified text, tho, some of the displayed text may
still be unfontified, because it was marked as "fontified=defer" in the
previous redisplay).

So, if the text scrolls by fully unfontified, it should be the case that
fontification did not happen during scrolling.


        Stefan

bug#20404: 25.0.50; Sometimes no fontification with jit-lock-defer-time

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: tsdh@gnu.org,  acm@muc.de,  20404@debbugs.gnu.org
> Date: Thu, 23 Apr 2015 13:27:20 -0400
> 
> > FWIW, when I set jit-lock-defer-time to zero and lean on the PageDown
> > key, I see somewhat clunky scrolling with no fontifications, until I
> > release the key.
> > But fontification inside the scrolling code itself still happens, I
> > think.
> 
> Hmm... while I guess it's still possible, it normally shouldn't happen:
> if the fontification takes place during the scrolling itself, then the
> next redisplay should not be skipped

How does this work?

By "fontification inside scrolling" I mean fontification triggered by
calling move_it_* functions, which simulate display.  They will only
fontify those parts of text over which they iterate, which might be
much less than the window.

> (and hence you should see fontified text rather than unfontified
> text, tho, some of the displayed text may still be unfontified,
> because it was marked as "fontified=defer" in the previous
> redisplay).

It's hard to say, but I'm almost sure I sometimes indeed see a small
part of the scrolling window, near its beginning, fontified.  Or maybe
I'm just imagining things.

bug#20404: 25.0.50; Sometimes no fontification with jit-lock-defer-time

[Stefan Monnier]

>> Hmm... while I guess it's still possible, it normally shouldn't happen:
>> if the fontification takes place during the scrolling itself, then the
>> next redisplay should not be skipped
> How does this work?

The decision whether redisplay is skipped or not is based on the
input-pending-p at the beginning of the command, i.e. before the call to
move_it.
So, if move_it triggers jit-lock which triggers font-lock, that means
jit-lock found (during move_it) that input-pending-p was nil, which
should imply that input-pending-p was also nil at the beginning of the
command, and hence the next redisplay will not be skipped.

> It's hard to say, but I'm almost sure I sometimes indeed see a small
> part of the scrolling window, near its beginning, fontified.  Or maybe
> I'm just imagining things.

Then I guess it's within the "explainable behavior".


        Stefan

bug#20404: 25.0.50; Sometimes no fontification with jit-lock-defer-time

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: tsdh@gnu.org,  acm@muc.de,  20404@debbugs.gnu.org
> Date: Thu, 23 Apr 2015 15:35:18 -0400
> 
> >> Hmm... while I guess it's still possible, it normally shouldn't happen:
> >> if the fontification takes place during the scrolling itself, then the
> >> next redisplay should not be skipped
> > How does this work?
> 
> The decision whether redisplay is skipped or not is based on the
> input-pending-p at the beginning of the command, i.e. before the call to
> move_it.
> So, if move_it triggers jit-lock which triggers font-lock, that means
> jit-lock found (during move_it) that input-pending-p was nil, which
> should imply that input-pending-p was also nil at the beginning of the
> command, and hence the next redisplay will not be skipped.

I'm not sure I see all this; perhaps I'm looking at the wrong places.

What I see is that the only reference to input-pending-p in
jit-lock.el is in jit-lock-function, where it is used to decide
whether to defer fontifications, which I believe is not what you were
describing above.

So how does "jit-lock find (during move_it) that input-pending-p is
nil"?

bug#20404: 25.0.50; Sometimes no fontification with jit-lock-defer-time

[Stefan Monnier]

> So how does "jit-lock find (during move_it) that input-pending-p is nil"?

With the input-pending-p in jit-lock-function.


        Stefan

bug#20404: 25.0.50; Sometimes no fontification with jit-lock-defer-time

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Cc: tsdh@gnu.org, acm@muc.de, 20404@debbugs.gnu.org
> Date: Fri, 24 Apr 2015 10:03:39 -0400
> 
> > So how does "jit-lock find (during move_it) that input-pending-p is nil"?
> 
> With the input-pending-p in jit-lock-function.

So you were describing something that happens only when
jit-lock-defer-time is zero, not the general behavior.  Nice
misunderstanding.

bug#20404: 25.0.50; Sometimes no fontification with jit-lock-defer-time

[Stefan Monnier]

> So you were describing something that happens only when
> jit-lock-defer-time is zero, not the general behavior.

That's right.


        Stefan

bug#20404: 25.0.50; Sometimes no fontification with jit-lock-defer-time

[Tassilo Horn]

Eli Zaretskii <eliz@gnu.org> writes:

>> That's because the code has changed since.  Now redisplay (which
>> happens after running a command) is skipped based on the value of
>> `input-pending-p' at the beginning of the *command*, rather than
>> based on its value at the beginning of redisplay.  And jit-lock.el
>> has been changed to use (not (input-pending-p)), but only if
>> jit-lock-defer-time is 0, so the special case for 0 is new.
>
> FWIW, when I set jit-lock-defer-time to zero and lean on the PageDown
> key, I see somewhat clunky scrolling with no fontifications, until I
> release the key.

Oh, I've told you that I get the display freezes even with
jit-lock-defer-time set to zero but I've been wrong.  My error was that
I called `emacs -Q src/buffer.c', then set jit-lock-defer-time to zero
in *scratch*, and then scrolled in buffer.c.  But the effect applies
only to new buffers.

So with jit-lock-defer-time = 0, scrolling doesn't freeze the display,
and also the "no fontification problem" in bug report buffers or
byte-compilation output buffers doesn't occur anymore.

> But fontification inside the scrolling code itself still happens, I
> think.

Yes, most text that scrolls by is black on white but occassionally I can
catch sight of something reddish (a string or comment) or bluish
(function name).

Bye,
Tassilo

bug#20404: 25.0.50; Sometimes no fontification with jit-lock-defer-time

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: tsdh@gnu.org,  acm@muc.de,  20404@debbugs.gnu.org
> Date: Thu, 23 Apr 2015 12:23:02 -0400
> 
> >   http://lists.gnu.org/archive/html/emacs-devel/2014-10/msg00705.html
> 
> This message of yours seems to be talking about the problem I'm
> referring to above.  At least, the way I read it, it seems to say that
> `input-pending-p' can return nil even if there is pending input
> (because that pending input is still in the system's input queue).

I don't see how what you say can follow from what I wrote in that
message.  What I say there is that while we process keystrokes, we
don't call read_socket_hook.  This observation is based on the code of
get_input_pending, which is the workhorse of input-pending-p:

 static bool
 get_input_pending (int flags)
 {
   /* First of all, have we already counted some input?  */
   input_pending = (!NILP (Vquit_flag) || readable_events (flags));

   /* If input is being read as it arrives, and we have none, there is none.  */
   if (!input_pending && (!interrupt_input || interrupts_deferred))
     {
       /* Try to read some input and see how much we get.  */
       gobble_input ();
       input_pending = (!NILP (Vquit_flag) || readable_events (flags));
     }

   return input_pending;

The call to read_socket_hook is in gobble_input, so if the first call
to readable_events returns non-zero, as it happens when we have events
waiting in our keyboard queue, we won't call read_socket_hook.

But that doesn't mean input-pending-p will return nil when there's
input unread by read_socket_hook, because once our keyboard queue gets
emptied, readable_events will return zero, and then we _will_ call
read_socket_hook to fill our queue, and then the second call to
readable_events will return non-zero, and input-pending-p will return
non-nil.

bug#20404: 25.0.50; Sometimes no fontification with jit-lock-defer-time

[Stefan Monnier]

> I don't see how what you say can follow from what I wrote in that message.

I just misunderstood.

> But that doesn't mean input-pending-p will return nil when there's
> input unread by read_socket_hook, because once our keyboard queue gets
> emptied, readable_events will return zero, and then we _will_ call
> read_socket_hook to fill our queue, and then the second call to
> readable_events will return non-zero, and input-pending-p will return
> non-nil.

Good.  But then I don't understand why Tassilo gets the same display
freezes with jit-lock-defer-time set to 0 as when jit-lock-defer-time is
set to nil.

You seem to say that you know why (and that you explained it in those
old discussions), but I guess I don't understand those explanations.


        Stefan

bug#20404: 25.0.50; Sometimes no fontification with jit-lock-defer-time

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: tsdh@gnu.org,  acm@muc.de,  20404@debbugs.gnu.org
> Date: Thu, 23 Apr 2015 15:31:56 -0400
> 
> > I don't see how what you say can follow from what I wrote in that message.
> 
> I just misunderstood.
> 
> > But that doesn't mean input-pending-p will return nil when there's
> > input unread by read_socket_hook, because once our keyboard queue gets
> > emptied, readable_events will return zero, and then we _will_ call
> > read_socket_hook to fill our queue, and then the second call to
> > readable_events will return non-zero, and input-pending-p will return
> > non-nil.
> 
> Good.  But then I don't understand why Tassilo gets the same display
> freezes with jit-lock-defer-time set to 0 as when jit-lock-defer-time is
> set to nil.

Neither do I.  I see a very big difference: with jit-lock-defer-time
set to zero, the freezes are minor and almost non-existent.  With it
set to nil, they are much more evident and much longer.

> You seem to say that you know why (and that you explained it in those
> old discussions), but I guess I don't understand those explanations.

I never said the value of zero should have no effect, nor do I thing
it should, or does.

bug#20404: 25.0.50; Sometimes no fontification with jit-lock-defer-time

[Tassilo Horn]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> But that doesn't mean input-pending-p will return nil when there's
>> input unread by read_socket_hook, because once our keyboard queue
>> gets emptied, readable_events will return zero, and then we _will_
>> call read_socket_hook to fill our queue, and then the second call to
>> readable_events will return non-zero, and input-pending-p will return
>> non-nil.
>
> Good.  But then I don't understand why Tassilo gets the same display
> freezes with jit-lock-defer-time set to 0 as when jit-lock-defer-time is
> set to nil.

Sorry, sorry, sorry.  I've been wrong.  I had opened the test buffer.c
buffer already before I've set jit-lock-defer-time to zero, and then the
new value had no effect on that buffer.

Now I can confirm that j-l-d-t = 0 fixes both the display freezes and
the non-fontified bug report or byte-compile buffers.

Bye,
Tassilo

bug#20404: 25.0.50; Sometimes no fontification with jit-lock-defer-time

[Eli Zaretskii]

> From: Tassilo Horn <tsdh@gnu.org>
> Date: Wed, 22 Apr 2015 11:45:42 +0200
> 
> I use the settings
> 
>   (setq jit-lock-defer-time 0.0312
>         jit-lock-stealth-nice 0.117
>         jit-lock-stealth-time 0.23)
> 
> to speed up scrolling in large buffers with complex font-lock rules.
> That basically works nice.  However, quite often I'm faced with buffers
> that are not fontified before I do something, e.g., move point.
> 
> This frequently happens when installing/upgrading packages with the
> package manager.  That shows the buffer containing the byte-compliation
> messages with fontified warnings and errors.  Every second time or so,
> I'm faced with a completely unfontified buffer that gets fontified only
> after selecting the window showing the buffer.
> 
> Another and even reproducible case is `report-emacs-bug'.  It goes like
> this:
> 
>   1. emacs -Q
>   2. eval my above settings in *scratch*
>   3. M-x report-emacs-bug
> 
> The new message buffer is completely unfontified initially.

I've reproduced this recipe.  The fontification happens about 2.5 sec
after the buffer is initially displayed.  I've noticed that the cursor
also doesn't blink during these few seconds, which seems to imply that
Emacs simply doesn't "become idle" during that time, or maybe doesn't
run _any_ idle timers during that time for some other reason.

Btw, it is enough to set jit-lock-defer-time to something non-zero, to
have this effect; the other 2 customizations seem not to affect this.

bug#20404: 25.0.50; Sometimes no fontification with jit-lock-defer-time

[Eli Zaretskii]

> Date: Thu, 23 Apr 2015 10:54:24 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: 20404@debbugs.gnu.org
> 
> >   1. emacs -Q
> >   2. eval my above settings in *scratch*
> >   3. M-x report-emacs-bug
> > 
> > The new message buffer is completely unfontified initially.
> 
> I've reproduced this recipe.  The fontification happens about 2.5 sec
> after the buffer is initially displayed.  I've noticed that the cursor
> also doesn't blink during these few seconds, which seems to imply that
> Emacs simply doesn't "become idle" during that time, or maybe doesn't
> run _any_ idle timers during that time for some other reason.
> 
> Btw, it is enough to set jit-lock-defer-time to something non-zero, to
> have this effect; the other 2 customizations seem not to affect this.

Moreover, here's what seems to be a much simpler recipe that shows the
underlying problem:

  emacs -Q
  M-x blink-cursor-mode RET
  M-x blink-cursor-mode RET

And note that after the second "M-x blink-cursor-mode RET", which
turns ON the blinking, Emacs waits for about the same 2 sec before it
actually starts blinking.

And here's why: _any_ command that goes through
execute-extended-command will run this code:

            (when suggest-key-bindings
              (sit-for (cond
                        ((zerop (length (current-message))) 0)
                        ((numberp suggest-key-bindings) suggest-key-bindings)
                        (t 2))))))

The default value of suggest-key-bindings is t, so this means we
_always_ sit-for 2 seconds, unless the echo area shows nothing (a rare
situation in Emacs).  When Emacs is parked inside sit-for, it doesn't
become idle, I think for good reasons.  So the idle timers will not
run until those 2 sec have expired, or some input becomes available.

And indeed setting suggest-key-bindings to nil causes the buffer
displayed by report-emacs-bug to become fontified almost immediately,
under Tassilo's settings of jit-lock-defer-time.

The upshot of all this is that any command that displays a buffer via
execute-extended-command and also says something in the echo area,
will always delay JIT deferred fontifications (and any other features
that run via idle timers) for at least 2 sec.

bug#20404: 25.0.50; Sometimes no fontification with jit-lock-defer-time

[Tassilo Horn]

Eli Zaretskii <eliz@gnu.org> writes:

> And here's why: _any_ command that goes through
> execute-extended-command will run this code:
>
>             (when suggest-key-bindings
>               (sit-for (cond
>                         ((zerop (length (current-message))) 0)
>                         ((numberp suggest-key-bindings) suggest-key-bindings)
>                         (t 2))))))
>
> The default value of suggest-key-bindings is t, so this means we
> _always_ sit-for 2 seconds, unless the echo area shows nothing (a rare
> situation in Emacs).  When Emacs is parked inside sit-for, it doesn't
> become idle, I think for good reasons.  So the idle timers will not
> run until those 2 sec have expired, or some input becomes available.

Oh, yes, this explains things.

> And indeed setting suggest-key-bindings to nil causes the buffer
> displayed by report-emacs-bug to become fontified almost immediately,
> under Tassilo's settings of jit-lock-defer-time.

Confirmed.

Hm, sometimes I actually like `suggest-key-bindings' namely whenever it
tells that I could have used `C-x <whatever>' instead of `M-x ...'.  But
I don't like its suggestions to use `M-x fi-fi' instead of `M-x
find-file'.

I totally agree that sometimes there should be messages that should
stick around for a given amount of time but using `sit-for` to stop the
world turning as a whole seems like a bit overkill.

Maybe such sticky messages could be shown in the `header-line' and then
be removed by a timer.  But of course, what should one do if the
header-line is already in use?

> The upshot of all this is that any command that displays a buffer via
> execute-extended-command and also says something in the echo area,
> will always delay JIT deferred fontifications (and any other features
> that run via idle timers) for at least 2 sec.

But how does that explain the occassionally non-fontified compile
buffers I get during package upgrades/installs?  Those don't go through
`execute-extended-command'.  But compile.el has some own `sit-for'
invocations that might delay timers.

Bye,
Tassilo

bug#20404: 25.0.50; Sometimes no fontification with jit-lock-defer-time

[Eli Zaretskii]

> From: Tassilo Horn <tsdh@gnu.org>
> Cc: 20404@debbugs.gnu.org
> Date: Thu, 23 Apr 2015 11:04:43 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > And here's why: _any_ command that goes through
> > execute-extended-command will run this code:
> >
> >             (when suggest-key-bindings
> >               (sit-for (cond
> >                         ((zerop (length (current-message))) 0)
> >                         ((numberp suggest-key-bindings) suggest-key-bindings)
> >                         (t 2))))))
> >
> > The default value of suggest-key-bindings is t, so this means we
> > _always_ sit-for 2 seconds, unless the echo area shows nothing (a rare
> > situation in Emacs).  When Emacs is parked inside sit-for, it doesn't
> > become idle, I think for good reasons.  So the idle timers will not
> > run until those 2 sec have expired, or some input becomes available.
> 
> Oh, yes, this explains things.

This problem doesn't exist in Emacs 24.5, it only happens on master.
And the reason seems to be the change in execute-extended-command:
Emacs 24 only calls sit-for when the command actually _has_ a key
binding:

    (when binding
      ;; But first wait, and skip the message if there is input.
      (let* ((waited
              ;; If this command displayed something in the echo area;
              ;; wait a few seconds, then display our suggestion message.
              (sit-for (cond
                        ((zerop (length (current-message))) 0)
                        ((numberp suggest-key-bindings) suggest-key-bindings)
                        (t 2)))))

On master, we wait unconditionally (and then wait some more if we
actually show the suggested key binding, but that happens only once
per given command).

So perhaps some small change there, which removes this unnecessary
unconditional wait whenever possible could solve the issue with
jit-lock-defer-time and other similar idle timers.

> Maybe such sticky messages could be shown in the `header-line' and then
> be removed by a timer.  But of course, what should one do if the
> header-line is already in use?

And in addition, this will cause an unpleasant redisplay of the
current window, something that display in the echo area avoids.

> But how does that explain the occassionally non-fontified compile
> buffers I get during package upgrades/installs?  Those don't go through
> `execute-extended-command'.  But compile.el has some own `sit-for'
> invocations that might delay timers.

Yes, something similar.  If you can reproduce these problems at will,
I can tell how to figure out which code caused the delay, similarly to
what I did to figure out the problem with execute-extended-command.

bug#20404: 25.0.50; Sometimes no fontification with jit-lock-defer-time

[Stefan Monnier]

> And here's why:
[...]
>             (when suggest-key-bindings
>               (sit-for (cond

Oh, right.  Another one of those "sit-for considered harmful".

I think we need a new (run-after-if-still-idle TIME FUNCTION) which sets
up a timer to run FUNCTION after TIME seconds, but where that timer is
canceled as soon as "something happens" (I guess we could start by
canceling that timer from pre-command-hook and hope it's sufficient).


        Stefan

bug#20404: 25.0.50; Sometimes no fontification with jit-lock-defer-time

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>   emacs -Q
>   M-x blink-cursor-mode RET
>   M-x blink-cursor-mode RET
>
> And note that after the second "M-x blink-cursor-mode RET", which
> turns ON the blinking, Emacs waits for about the same 2 sec before it
> actually starts blinking.
>
> And here's why: _any_ command that goes through
> execute-extended-command will run this code:
>
>             (when suggest-key-bindings
>               (sit-for (cond
>                         ((zerop (length (current-message))) 0)
>                         ((numberp suggest-key-bindings) suggest-key-bindings)
>                         (t 2))))))

This was fixed in 2018 by this commit, I think.

commit 48a28f8e389c33029ab4aa3d65445f42ed457e11
Author: Dmitry Gutov <dgutov@yandex.ru>
Date:   Mon Dec 24 04:36:08 2018 +0200

    execute-extended-command: Skip waiting in more cases
    
    * lisp/simple.el (execute-extended-command): Don't wait when
    there's no binding the current command, and the user doesn't want
    to see "shorter" suggestions, or TYPED is nil anyway.

Skimming this thread, it's unclear whether there was anything else to
fix here.  Can this bug be closed?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#20404: 25.0.50; Sometimes no fontification with jit-lock-defer-time

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: tsdh@gnu.org,  20404@debbugs.gnu.org
> Date: Thu, 31 Oct 2019 15:23:16 +0100
> 
> > And here's why: _any_ command that goes through
> > execute-extended-command will run this code:
> >
> >             (when suggest-key-bindings
> >               (sit-for (cond
> >                         ((zerop (length (current-message))) 0)
> >                         ((numberp suggest-key-bindings) suggest-key-bindings)
> >                         (t 2))))))
> 
> This was fixed in 2018 by this commit, I think.
> 
> commit 48a28f8e389c33029ab4aa3d65445f42ed457e11
> Author: Dmitry Gutov <dgutov@yandex.ru>
> Date:   Mon Dec 24 04:36:08 2018 +0200
> 
>     execute-extended-command: Skip waiting in more cases
>     
>     * lisp/simple.el (execute-extended-command): Don't wait when
>     there's no binding the current command, and the user doesn't want
>     to see "shorter" suggestions, or TYPED is nil anyway.
> 
> Skimming this thread, it's unclear whether there was anything else to
> fix here.  Can this bug be closed?

Yes; done.

Thanks.

bug#20404: 25.0.50; Sometimes no fontification with jit-lock-defer-time

[Tassilo Horn]

Eli Zaretskii <eliz@gnu.org> writes:

> I've reproduced this recipe.  The fontification happens about 2.5 sec
> after the buffer is initially displayed.  I've noticed that the cursor
> also doesn't blink during these few seconds, which seems to imply that
> Emacs simply doesn't "become idle" during that time, or maybe doesn't
> run _any_ idle timers during that time for some other reason.

It doesn't run _any_ timers, no matter if normal or idle.  See my other
mail which also includes redisplay traces.

> Btw, it is enough to set jit-lock-defer-time to something non-zero, to
> have this effect; the other 2 customizations seem not to affect this.

Right.

Bye,
Tassilo