bug#28795: 25.2; UTF-8 language environment causes extreme sluggishness

bug#28795: 25.2; UTF-8 language environment causes extreme sluggishness

[Mark Laws]

M-x set-language-environment UTF-8
M-: (insert #x2207)

Then hold down C-o until you have a bunch of empty lines, and start moving
around the buffer with the usual movement keys.  You will notice it is very,
very slow.  Then delete the character from the buffer, and notice how things are
fast again.

The issue doesn't seem to happen in the Linux console version of Emacs (even
with UTF-8 set).  It also does not happen in the Windows (NT) GUI version if the
language environment is set to something other than UTF-8.

I can reproduce this 100% of the time under emacs -Q with the instructions given above.



In GNU Emacs 25.2.1 (x86_64-w64-mingw32)
 of 2017-04-30 built on HAYAI
Repository revision: d8cf7c798722e58aa4ebb9ecfcaa6724f905e18e
Windowing system distributor 'Microsoft Corp.', version 10.0.14393
Configured using:
 'configure --host=x86_64-w64-mingw32 --target=x86_64-w64-mingw32
 --build=x86_64-w64-mingw32 --prefix=/c/Users/mdl/local/emacs --without-dbus
 --without-gconf --without-gsettings --with-modules --with-file-notification=w32
 CFLAGS=-O2 CPPFLAGS=-I/mingw64/include LDFLAGS=-L/mingw64/lib'

Configured features:
XPM JPEG TIFF GIF PNG RSVG SOUND NOTIFY ACL GNUTLS LIBXML2 ZLIB
TOOLKIT_SCROLL_BARS MODULES W32_IME

Important settings:
  value of $LANG: JPN
  locale-coding-system: cp1252

Major mode: Emacs-Lisp

Minor modes in effect:
  company-quickhelp-mode: t
  company-quickhelp-local-mode: t
  elisp-slime-nav-mode: t
  rainbow-mode: t
  flycheck-pos-tip-mode: t
  company-mode: t
  global-git-commit-mode: t
  async-bytecomp-package-mode: t
  shell-dirtrack-mode: t
  diff-auto-refine-mode: t
  desktop-save-mode: t
  eldoc-in-minibuffer-mode: t
  global-form-feed-mode: t
  form-feed-mode: t
  global-whitespace-cleanup-mode: t
  whitespace-cleanup-mode: t
  whitespace-mode: t
  global-undo-tree-mode: t
  undo-tree-mode: t
  show-paren-mode: t
  global-hl-line-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  column-number-mode: t
  line-number-mode: t
  auto-fill-function: do-auto-fill
  transient-mark-mode: t

Recent messages:
Wrote c:/Users/mdl/AppData/Roaming/.emacs.d/elisp/mdl-loaddefs.el
Loading c:/Users/mdl/AppData/Roaming/.emacs.d/elisp/mdl-loaddefs.el (source)...done
Please type y, n, or !: y
Error while checking syntax automatically: (irony-server-error "irony-server version mismatch: type ‘M-x irony-install-server’ to reinstall")
Please type y, n, or !: y
Error while checking syntax automatically: (irony-server-error "irony-server is broken! Invalid version syntax: ‘1.1.0’")
Please type y, n, or !: y
Error while checking syntax automatically: (irony-server-error "irony-server is broken! Invalid version syntax: ‘1.1.0’")
Wrote c:/Users/mdl/AppData/Roaming/.emacs.d/.emacs.desktop.lock
Desktop: 3 frames, 46 buffers restored.

Load-path shadows:
c:/Users/mdl/AppData/Roaming/.emacs.d/elpa/seq-2.20/seq hides c:/Users/mdl/local/emacs/share/emacs/25.2/lisp/emacs-lisp/seq

Features:
(shadow sort mail-extr warnings emacsbug sendmail bat-mode make-mode
irony-cdb-libclang irony-cdb-json pp goto-addr view matlab derived tempo
pandoc-mode pandoc-mode-utils hydra lv vc-git irony-eldoc
irony-cdb-clang-complete irony-cdb modern-cpp-font-lock cc-mode cc-fonts
cc-guess cc-menus cc-cmds cc-styles cc-align cc-engine cc-vars cc-defs
markdown-mode thingatpt noutline outline jka-compr company-quickhelp
fsharp-mode-font fsharp-mode-indent info-look fsharp-mode
fsharp-mode-indent-smie inf-fsharp-mode fsharp-doc flycheck-fsharp
fsharp-mode-completion fsharp-mode-util tramp tramp-compat tramp-loaddefs
trampver ucs-normalize stupid-indent-mode clean-aindent-mode wolfram-mode smie
autoinsert elisp-slime-nav rainbow-mode company-oddmuse company-keywords
company-etags etags xref project company-gtags company-dabbrev-code
company-dabbrev company-files company-capf company-cmake company-xcode
company-clang company-semantic company-eclim company-css company-nxml
company-bbdb company-irony company-template irony-completion irony-snippet
flycheck-irony flycheck-pos-tip pos-tip flycheck find-func irony-diagnostics
irony irony-iotask color company proof-site proof-autoloads pg-vars
magit-obsolete magit-blame magit-stash magit-bisect magit-remote magit-commit
magit-sequence magit-notes magit-worktree magit-branch magit-files magit-refs
magit-status magit magit-repos magit-apply magit-wip magit-log magit-diff
smerge-mode magit-core magit-autorevert autorevert filenotify magit-process
magit-margin magit-mode magit-git magit-section magit-popup git-commit
magit-utils crm log-edit message format-spec rfc822 mml mml-sec epg mm-decode
mm-bodies mm-encode mail-parse rfc2231 rfc2047 rfc2045 ietf-drums mailabbrev
mail-utils gmm-utils mailheader pcvs-util add-log with-editor async-bytecomp
async shell pcomplete p4 ps-print ps-def lpr diff-mode dired compile comint
ansi-color ring server revive desktop frameset eldoc-eval form-feed easy-mmode
whitespace-cleanup-mode whitespace ido undo-tree diff paren hl-line vlf-setup
visual-regexp-steroids advice visual-regexp dim mdl-utils mdl-loaddefs autoload
lisp-mnt tex-site edmacro kmacro gh-common gh-profile url-parse auth-source
gnus-util mm-util help-fns mail-prsvr password-cache url-vars rx s marshal
eieio-compat ht json map dash eieio eieio-core cl-seq cl-macs cl finder-inf info
package epg-config seq byte-opt gv bytecomp byte-compile cl-extra help-mode
easymenu cconv cl-loaddefs cl-lib subr-x pcase time-date mule-util japan-util
tooltip eldoc electric uniquify ediff-hook vc-hooks lisp-float-type mwheel
dos-w32 ls-lisp w32-ime disp-table w32-win w32-vars term/common-win tool-bar dnd
fontset image regexp-opt fringe tabulated-list newcomment elisp-mode lisp-mode
prog-mode register page menu-bar rfn-eshadow timer select scroll-bar mouse
jit-lock font-lock syntax facemenu font-core frame cl-generic cham georgian
utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean japanese
eucjp-ms cp51932 hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese charscript case-table epa-hook jka-cmpr-hook help simple abbrev
minibuffer cl-preloaded nadvice loaddefs button faces cus-face macroexp files
text-properties overlay sha1 md5 base64 format env code-pages mule custom widget
hashtable-print-readable backquote w32notify w32 multi-tty make-network-process
emacs)

Memory information:
((conses 16 546773 36422)
 (symbols 56 44493 0)
 (miscs 48 281 814)
 (strings 32 125737 19777)
 (string-bytes 1 3681556)
 (vectors 16 68989)
 (vector-slots 8 1133465 9793)
 (floats 8 1100 285)
 (intervals 56 2889 0)
 (buffers 976 70))

bug#28795: 25.2; UTF-8 language environment causes extreme sluggishness

[Eli Zaretskii]

> From: Mark Laws <mdl@60hz.org>
> Date: Thu, 12 Oct 2017 18:15:53 +0900
> 
> M-x set-language-environment UTF-8

On Windows, doing this is a mistake (but not because it causes
sluggish redisplay).

> M-: (insert #x2207)
> 
> Then hold down C-o until you have a bunch of empty lines, and start moving
> around the buffer with the usual movement keys.  You will notice it is very,
> very slow.  Then delete the character from the buffer, and notice how things are
> fast again.

I cannot reproduce this here.  What font is used to display that
character?  You can find this out with "C-u C-x =" with the cursor on
the character.

On my system, I have the Symbola font installed, which is used for
displaying this character, and the problem doesn't happen.

As another thing to try, set inhibit-compacting-font-caches to a
non-nil value, and see if that helps.

bug#28795: 25.2; UTF-8 language environment causes extreme sluggishness

[Eli Zaretskii]

[Please keep the bug address on the CC list.]

> From: Mark Laws <mdl@60hz.org>
> Date: Thu, 12 Oct 2017 18:40:25 +0900
> 
> On Thu, Oct 12, 2017 at 6:31 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> >> From: Mark Laws <mdl@60hz.org>
> >> Date: Thu, 12 Oct 2017 18:15:53 +0900
> >>
> >> M-x set-language-environment UTF-8
> >
> > On Windows, doing this is a mistake (but not because it causes
> > sluggish redisplay).
> 
> What is the correct way to use UTF-8 as the default encoding for
> things like file-visiting buffers without causing other potential
> problems?

Put this in your .emacs:

  (prefer-coding-system 'utf-8)

or even

  (setq-default buffer-file-coding-system 'utf-8-dos)

(the latter doesn't affect process I/O).

> > I cannot reproduce this here.  What font is used to display that
> > character?  You can find this out with "C-u C-x =" with the cursor on
> > the character.
> 
> uniscribe:-outline-PragmataPro
> Mono-normal-normal-normal-mono-15-*-*-*-c-*-iso8859-1 (#x8FC)

And which font is used if you don't set your language-environment to
UTF-8?

> If it doesn't work for you off the bat, do this after inserting the character:
> 
> M-: (put-text-property 1 2 'face font-lock-warning-face)
> 
> Then try moving around again.

Doesn't change anything here.

I do see slow redisplay on another system, where Symbola is not
installed, and the BatangChe font is then used.  But there I see slow
redisplay whether or not I set the language-environment to UTF-8.

> > On my system, I have the Symbola font installed, which is used for
> > displaying this character, and the problem doesn't happen.
> >
> > As another thing to try, set inhibit-compacting-font-caches to a
> > non-nil value, and see if that helps.
> 
> This seems to help

By "this" do you mean installing Symbola or setting
inhibit-compacting-font-caches non-nil?

> though the fact it only happens due to changes with encoding and/or
> font locking makes me suspicious.

It could be that the default language-environment causes a different
font to be used for that character.

bug#28795: 25.2; UTF-8 language environment causes extreme sluggishness

[Mark Laws]

On Thu, Oct 12, 2017 at 7:00 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> [Please keep the bug address on the CC list.]

Sorry, mashed the wrong key.

> And which font is used if you don't set your language-environment to
> UTF-8?

uniscribe:-outline-Malgun
Gothic-bold-normal-normal-sans-15-*-*-*-p-*-ksc5601.1987-0 (#xF7)

>> > As another thing to try, set inhibit-compacting-font-caches to a
>> > non-nil value, and see if that helps.
>>
>> This seems to help
>
> By "this" do you mean installing Symbola or setting
> inhibit-compacting-font-caches non-nil?

(setq inhibit-compacting-font-caches t) seems to help. I did not try Symbola.

>> though the fact it only happens due to changes with encoding and/or
>> font locking makes me suspicious.
>
> It could be that the default language-environment causes a different
> font to be used for that character.

If I use (prefer-coding-system 'utf-8), I do not get the slowness
anymore, even if inhibit-compacting-font-caches is nil, and even after
propertizing the text. However, the font changes from PragmataPro Mono
to Malgun Gothic, which seems odd too.

if I use both (set-locale-environment "English") and
(prefer-coding-system 'utf-8), the issue returns.

-- 
|v\ /\ |\ |< |_ /\ \^| //

bug#28795: 25.2; UTF-8 language environment causes extreme sluggishness

[Eli Zaretskii]

> From: Mark Laws <mdl@60hz.org>
> Date: Thu, 12 Oct 2017 19:31:57 +0900
> Cc: 28795@debbugs.gnu.org
> 
> > And which font is used if you don't set your language-environment to
> > UTF-8?
> 
> uniscribe:-outline-Malgun
> Gothic-bold-normal-normal-sans-15-*-*-*-p-*-ksc5601.1987-0 (#xF7)

OK, so now we know why you get slow redisplay only with UTF-8 as
language-environment: doing that changes the preferred charset, which
affects font selection (Malgun Gothic is for the Korean character set,
as you see from the end of its spec, so it is rejected when the
preferred charset is unicode-bmp or Unicode).

> If I use (prefer-coding-system 'utf-8), I do not get the slowness
> anymore, even if inhibit-compacting-font-caches is nil, and even after
> propertizing the text. However, the font changes from PragmataPro Mono
> to Malgun Gothic, which seems odd too.

Why is that odd?  Emacs selects fonts according to what it thinks is
the user's preferences, and by default those preferences are deduced
from the culture defined in the environment.

> if I use both (set-locale-environment "English") and
> (prefer-coding-system 'utf-8), the issue returns.

"English" nowadays defaults to Unicode as well.

In general, there should be no reason for you to set a non-default
language-environment on Windows, because Emacs takes that from your
system's setup.  If you just want to control the default encodings,
use prefer-coding-system, its effect is limited to encodings, and
doesn't change anything else.

Is there any other problem left to solve, or we can close this bug
report?

bug#28795: 25.2; UTF-8 language environment causes extreme sluggishness

[Mark Laws]

On 2017/10/12, at 20:55, Eli Zaretskii <eliz@gnu.org> wrote:

> Is there any other problem left to solve, or we can close this bug
> report?

Nope, we're good. Thanks as always for the help!

bug#28795: 25.2; UTF-8 language environment causes extreme sluggishness

[Eli Zaretskii]

> From: Mark Laws <mdl@60hz.org>
> Date: Thu, 12 Oct 2017 21:17:47 +0900
> Cc: 28795@debbugs.gnu.org
> 
> On 2017/10/12, at 20:55, Eli Zaretskii <eliz@gnu.org> wrote:
> 
> > Is there any other problem left to solve, or we can close this bug
> > report?
> 
> Nope, we're good. Thanks as always for the help!

OK, you're welcome; closing.