bug#24205: 25.1.50; Re-display issue (lag) when opening certain files [sml-mode]

bug#24205: 25.1.50; Re-display issue (lag) when opening certain files [sml-mode]

[Kaushal Modi]

[-- Attachment #1: Type: text/plain, Size: 7392 bytes --]

Hi,

Below code easily recreates the issue on emacs -Q on the latest master
build of emacs as of now (2e0a2b3).

Please evaluate the macro definition and the macro call form below.

In short, this will install the sml-mode package from GNU Elpa to the
temporary-file-directory for your system, and also generate a test .sml
file that happened to show this issue. I do not see this lag on opening
other .sml files. For some reason, this .sml file (that the below example
creates) has ONLY comments and that causes emacs to stall for 10's of
seconds.

After evaluating the code below, you should (too, hopefully) see emacs
stall for 10-20 seconds. All it was doing was opening this file:

=====
(* ''a is a special equality type (it is not the same as the generic 'a
type. It *)
(* means that both arguments can be of any type as long as you can use the
*)
(* equality operation between them. So both can be int, both can be
strings, *)
(* both can be int tuples. TH*)
=====

After that, on doing M-x profiler-report, I get:

=====
+ redisplay_internal (C function)                               14724  99%
+ command-execute                                                  42   0%
+ ...                                                              27   0%
=====

I am copying Eli (for possibly re-display issue) and Stefan (for I have
seen for the first and only time in sml-mode).

I have started using sml-mode just recently (2 weeks), and my emacs froze
as I was typing out that comment in the test file. So I have no idea if the
bug is specific to 25.1/25.2 versions.

=====

(defmacro emacs-pkg-debug-setup (pkg-alist &rest body)
  "Install packages in PKG-ALIST and evaluate BODY.
Each element of PKG-ALIST has the form (((ID . LOCATION) . (PKG1 PKG2 ..))
..).
The ID and LOCATION are the same as the ones in `package-archives'.
PKG1, PKG2, .. are package names from the ID archive.

Example usage:

1. Launch 'emacs -Q'.
2. Copy this macro definition to its scratch buffer and evaluate it.
3. Evaluate a minimum working example using this macro as below:
     (emacs-pkg-debug-setup '(;; Install hydra from GNU Elpa
                              (nil . (hydra))
                              ;; Install org from Org Elpa
                              ((\"org\" . \"http://orgmode.org/elpa/\") .
(org)))
       ;; Then evaluate the below forms
       (org-mode)) "
  (declare (indent 1) (debug t))
  `(progn
     (require 'package)
     (setq user-emacs-directory (concat temporary-file-directory
                                        (getenv "USER") "/"
".emacs.d-debug/"))
     (setq package-user-dir (concat user-emacs-directory "elpa/"))
     (let (archive pkgs)
       (dolist (archive-alist ,pkg-alist)
         (setq archive (car archive-alist))
         (when archive
           (add-to-list 'package-archives archive :append))
         (setq pkgs (append pkgs (cdr archive-alist))))
       (package-initialize)
       (package-refresh-contents)

       (dolist (pkg pkgs)
         (when (and pkg (not (package-installed-p pkg)))
           (package-install pkg))
         (require pkg))

       ,@body)))

(emacs-pkg-debug-setup '(;; Install sml-mode from GNU Elpa
                         (nil . (sml-mode)))
  (let ((test-file (expand-file-name "lag_test.sml"
temporary-file-directory)))
    (with-temp-buffer
      (insert (concat
               "(* ''a is a special equality type (it is not the same as
the generic 'a type. It *)\n"
               "(* means that both arguments can be of any type as long as
you can use the *)\n"
               "(* equality operation between them. So both can be int,
both can be strings, *)\n"
               "(* both can be int tuples. TH*)"))
      (write-file test-file))
    (profiler-start 'cpu)
    (find-file test-file)))

=====

In GNU Emacs 25.1.50.25 (x86_64-unknown-linux-gnu, GTK+ Version 2.24.23)
 of 2016-08-11 built on
Repository revision: 2e0a2b376f51dd515ffeb6449181cc37fe912f5e
Windowing system distributor 'The X.Org Foundation', version 11.0.60900000
System Description: Red Hat Enterprise Linux Workstation release 6.6
(Santiago)

Configured using:
 'configure --with-modules
 --prefix=/home/kmodi/usr_local/apps/6/emacs/master
 'CPPFLAGS=-fgnu89-inline -I/home/kmodi/usr_local/6/include
 -I/usr/include/freetype2 -I/usr/include' 'CFLAGS=-ggdb3 -O0'
 'CXXFLAGS=-ggdb3 -O0' 'LDFLAGS=-L/home/kmodi/usr_local/6/lib
 -L/home/kmodi/usr_local/6/lib64 -ggdb3''

Configured features:
XPM JPEG TIFF GIF PNG RSVG IMAGEMAGICK SOUND GPM DBUS GCONF GSETTINGS
NOTIFY ACL LIBSELINUX GNUTLS LIBXML2 FREETYPE LIBOTF XFT ZLIB
TOOLKIT_SCROLL_BARS GTK2 X11 MODULES

Important settings:
  value of $LANG: en_US.UTF-8
  value of $XMODIFIERS: @im=none
  locale-coding-system: utf-8-unix

Major mode: Profiler-Report

Minor modes in effect:
  diff-auto-refine-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  buffer-read-only: t
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug sendmail profiler vc-git diff-mode
easy-mmode sml-mode skeleton warnings smie cl compile comint ansi-color
ring autoload radix-tree lisp-mnt cus-edit cus-start cus-load wid-edit
mm-archive message dired dired-loaddefs format-spec rfc822 mml mml-sec
epa derived gnus-util rmail rmail-loaddefs mailabbrev gmm-utils
mailheader mm-decode mm-bodies mm-encode mail-utils network-stream
starttls url-http tls gnutls mail-parse rfc2231 rfc2047 rfc2045 mm-util
ietf-drums mail-prsvr url-gw nsm subr-x puny url-cache url-auth url
url-proxy url-privacy url-expand url-methods url-history url-cookie
url-domsuf url-util mailcap epg finder-inf package epg-config
url-handlers url-parse auth-source cl-seq eieio eieio-core cl-macs
eieio-loaddefs password-cache url-vars seq byte-opt gv bytecomp
byte-compile cl-extra cconv help-mode easymenu cl-loaddefs pcase cl-lib
debug time-date mule-util tooltip eldoc electric uniquify ediff-hook
vc-hooks lisp-float-type mwheel term/x-win x-win term/common-win x-dnd
tool-bar dnd fontset image regexp-opt fringe tabulated-list newcomment
elisp-mode lisp-mode prog-mode register page menu-bar rfn-eshadow timer
select scroll-bar mouse jit-lock font-lock syntax facemenu font-core
term/tty-colors frame cl-generic cham georgian utf-8-lang misc-lang
vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms cp51932
hebrew greek romanian slovak czech european ethiopic indian cyrillic
chinese charscript case-table epa-hook jka-cmpr-hook help simple abbrev
obarray minibuffer cl-preloaded nadvice loaddefs button faces cus-face
macroexp files text-properties overlay sha1 md5 base64 format env
code-pages mule custom widget hashtable-print-readable backquote
dbusbind inotify dynamic-setting system-font-setting font-render-setting
move-toolbar gtk x-toolkit x multi-tty make-network-process emacs)

Memory information:
((conses 16 153840 33952)
 (symbols 48 25942 2)
 (miscs 40 104 407)
 (strings 32 35522 6421)
 (string-bytes 1 1074371)
 (vectors 16 46905)
 (vector-slots 8 1777619 18518)
 (floats 8 269 573)
 (intervals 56 782 163)
 (buffers 976 16)
 (heap 1024 36043 2798))

-- 

Kaushal Modi

[-- Attachment #2: Type: text/html, Size: 10377 bytes --]

bug#24205: 25.1.50; Re-display issue (lag) when opening certain files [sml-mode]

[Eli Zaretskii]

> From: Kaushal Modi <kaushal.modi@gmail.com>
> Date: Thu, 11 Aug 2016 17:03:44 +0000
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>, Eli Zaretskii <eliz@gnu.org>
> 
> Below code easily recreates the issue on emacs -Q on the latest master build of emacs as of now
> (2e0a2b3).
> 
> Please evaluate the macro definition and the macro call form below.
> 
> In short, this will install the sml-mode package from GNU Elpa to the temporary-file-directory for your system,
> and also generate a test .sml file that happened to show this issue. I do not see this lag on opening other .sml
> files. For some reason, this .sml file (that the below example creates) has ONLY comments and that causes
> emacs to stall for 10's of seconds. 

I'd prefer not to install anything.  I do have the Git repo for ELPA,
though.  So could you please modify your recipe so that the required
packages are loaded from the Git repo (by modifying load-path), and
then the rest is done as needed?

Better yet, just post the file that causes the trouble, there's no
need for everybody to recreate it, right?

Thanks.

bug#24205: 25.1.50; Re-display issue (lag) when opening certain files [sml-mode]

[Kaushal Modi]

[-- Attachment #1: Type: text/plain, Size: 1610 bytes --]

On Thu, Aug 11, 2016 at 1:22 PM Eli Zaretskii <eliz@gnu.org> wrote:

> I'd prefer not to install anything.


Hi Eli,

To avoid people's personal emacs config from polluting, I change the
user-emacs-directory and package-user-dir in the macro. Things get
installed to the /tmp/ dir by default on unix systems.


> I do have the Git repo for ELPA,
> though.  So could you please modify your recipe so that the required
> packages are loaded from the Git repo (by modifying load-path), and
> then the rest is done as needed?
>

Please let me know if this is still necessary to help debug this problem.
The current macro definition is very generic and works for GNU Elpa + other
Elpas out there. It does not pollute your personal area as I mentioned
above. After done with this, just rm -rf (concat temporary-file-directory
(getenv "USER") "/" ".emacs.d-debug/")


> Better yet, just post the file that causes the trouble, there's no
> need for everybody to recreate it, right?
>

I wanted the mininum working example to be all inclusive. If I do not do
so, then I need to rely on manual steps like saving the file, opening the
exact same file using M-x find-file (and all that). Isn't my method more
reliable? All you do is eval the whole thing and check the profiler report.

Here is the file that my example creates:


(* ''a is a special equality type (it is not the same as the generic 'a
type. It *)
(* means that both arguments can be of any type as long as you can use the
*)
(* equality operation between them. So both can be int, both can be
strings, *)
(* both can be int tuples. TH*)
-- 

Kaushal Modi

[-- Attachment #2: Type: text/html, Size: 2471 bytes --]

bug#24205: 25.1.50; Re-display issue (lag) when opening certain files [sml-mode]

[Kaushal Modi]

[-- Attachment #1: Type: text/plain, Size: 1313 bytes --]

It turns out that just one line in that example is responsible for causing
that lag:

(* ''a is a special equality type (it is not the same as the generic 'a
type. It *)

I have updated the MWE. After evaluating the macro definition I posted
earlier and the new example below, hit "C-c r".

Installing a package from Elpa is also good for sanity. That way I do not
worry about local changes made in your or my Elpa git clones. Installing
from Elpa ensures that you and I are using the exact same version of the
package (assuming that the package did not update in the meanwhile :)).

=====
(emacs-pkg-debug-setup '(;; Install sml-mode from GNU Elpa
                         (nil . (sml-mode)))
  (let ((test-file (expand-file-name "lag_test.sml"
                                     user-emacs-directory))) ; the macro
sets user-emacs-directory to (concat temporary-file-directory (getenv
"USER") "/" ".emacs.d-debug/")
    (defun my-report ()
      (interactive)
      (profiler-report)
      (profiler-stop)
      (profiler-reset))
    (global-set-key (kbd "C-c r") #'my-report)

    (with-temp-buffer
      (insert "(* ''a is a special equality type (it is not the same as the
generic 'a type. It *)")
      (write-file test-file))
    (profiler-start 'cpu)
    (find-file test-file)))
=====
-- 

Kaushal Modi

[-- Attachment #2: Type: text/html, Size: 2120 bytes --]

bug#24205: 25.1.50; Re-display issue (lag) when opening certain files [sml-mode]

[Stefan Monnier]

> (* ''a is a special equality type (it is not the same as the generic 'a
> type. It *)

Sounds like we're hitting some pathological regexp-matching situation.


        Stefan

bug#24205: 25.1.50; Re-display issue (lag) when opening certain files [sml-mode]

[Eli Zaretskii]

> From: Kaushal Modi <kaushal.modi@gmail.com>
> Date: Thu, 11 Aug 2016 17:30:23 +0000
> Cc: bug-gnu-emacs@gnu.org, monnier@iro.umontreal.ca
> 
> To avoid people's personal emacs config from polluting, I change the user-emacs-directory and
> package-user-dir in the macro. Things get installed to the /tmp/ dir by default on unix systems.
> 
>  I do have the Git repo for ELPA,
>  though. So could you please modify your recipe so that the required
>  packages are loaded from the Git repo (by modifying load-path), and
>  then the rest is done as needed?
> 
> Please let me know if this is still necessary to help debug this problem.

It would help immensely.  Unless visiting the file is all that's
needed, see below.

> Here is the file that my example creates:
> 
> (* ''a is a special equality type (it is not the same as the generic 'a type. It *)
> (* means that both arguments can be of any type as long as you can use the *)
> (* equality operation between them. So both can be int, both can be strings, *)
> (* both can be int tuples. TH*)

Are you saying that just visiting this file exhibits the problem?  If
so, what modes should be turned on in the visiting buffer to see the
problem?

Thanks.

bug#24205: 25.1.50; Re-display issue (lag) when opening certain files [sml-mode]

[Kaushal Modi]

[-- Attachment #1: Type: text/plain, Size: 2035 bytes --]

On Thu, Aug 11, 2016 at 3:42 PM Eli Zaretskii <eliz@gnu.org> wrote:

> It would help immensely.  Unless visiting the file is all that's
> needed, see below.
>

Correct, visiting the file is all that's needed, with the correct major
mode turning on.
That's why the file name ends in .sml and I made my MWE install the
sml-mode package from GNU Elpa so that that mode is autoloaded on visiting
any .sml file.

To clarify, this issue does not occur for ANY .sml file. It happens to
occur only due to this example file:

The file has to contain just this sml comment:

=====
(* ''a is a special equality type (it is not the same as the generic 'a
type. It *)
=====

I cannot figure out what's special about this comment that freaks the
re-display.

Are you saying that just visiting this file exhibits the problem?


Yes, in sml-mode.


> If
> so, what modes should be turned on in the visiting buffer to see the
> problem?
>

All my MWE is doing is:
- Install sml-mode from GNU Elpa
- Create a specific example file that causes this issue
- Start profiler
- Visit the file

You will then see a 10-20 second lag.

After which you hit C-c r (that shows the profiler report and also resets
the profiler. My MWE sets that binding.).

Here's the MWE once again:

=====
(emacs-pkg-debug-setup
    '((nil . (sml-mode))) ; Install sml-mode from GNU Elpa
  (let ((test-file (expand-file-name
                    "lag_test.sml"
                    user-emacs-directory)))
    ;; The `emacs-pkg-debug-setup' macro sets user-emacs-directory to
    ;; (concat temporary-file-directory (getenv "USER") "/"
".emacs.d-debug/")
    (defun my-report ()
      (interactive)
      (profiler-report)
      (profiler-stop)
      (profiler-reset))
    (global-set-key (kbd "C-c r") #'my-report)

    (with-temp-buffer
      (insert (concat "(* ''a is a special equality type (it is not "
                      "the same as the generic 'a type. It *)"))
      (write-file test-file))
    (profiler-start 'cpu)
    (find-file test-file)))
=====
-- 

Kaushal Modi

[-- Attachment #2: Type: text/html, Size: 3425 bytes --]

bug#24205: 25.1.50; Re-display issue (lag) when opening certain files [sml-mode]

[Eli Zaretskii]

> From: Kaushal Modi <kaushal.modi@gmail.com>
> Date: Thu, 11 Aug 2016 19:53:05 +0000
> Cc: bug-gnu-emacs@gnu.org, monnier@iro.umontreal.ca
> 
> The file has to contain just this sml comment:
> 
> =====
> (* ''a is a special equality type (it is not the same as the generic 'a type. It *) 
> ===== 
> 
> I cannot figure out what's special about this comment that freaks the re-display.

Most probably the unbalanced parentheses.  Something in sml-mode
causes the display engine to hit on that many times.

I will look into this.

>  Are you saying that just visiting this file exhibits the problem? 
> 
> Yes, in sml-mode.
> 
>  If
>  so, what modes should be turned on in the visiting buffer to see the
>  problem?
> 
> All my MWE is doing is:
> - Install sml-mode from GNU Elpa
> - Create a specific example file that causes this issue
> - Start profiler
> - Visit the file

I must explain something so that my requests don't sound arbitrary.
When I work on a problem, I usually run Emacs many times in different
ways, most of the times under a debugger.  Since the debugger (and
other tools) need to read my init files for working properly as I'm
used to, I cannot easily change my home directory, or mess with my
init files during the debugging session.  In addition, with 10-20 sec
delay I don't need a profiler to notice it.

So it is much more helpful to have a recipe that is as simple as
possible (so invoking it many times won't be painful), avoids
interfering with my init files and system configuration, and includes
only the stuff that is directly relevant to the bug reproduction.

A good recipe along the above lines could easily speed up a bugfix by
hours and even days.  So if you can afford that, please try to follow
these guidelines when you prepare a recipe.

I hope this clarifies the reason for my requests.

Thanks.

bug#24205: 25.1.50; Re-display issue (lag) when opening certain files [sml-mode]

[Andreas Schwab]

On Aug 11 2016, Eli Zaretskii <eliz@gnu.org> wrote:

>> From: Kaushal Modi <kaushal.modi@gmail.com>
>> Date: Thu, 11 Aug 2016 19:53:05 +0000
>> Cc: bug-gnu-emacs@gnu.org, monnier@iro.umontreal.ca
>> 
>> The file has to contain just this sml comment:
>> 
>> =====
>> (* ''a is a special equality type (it is not the same as the generic 'a type. It *) 
>> ===== 
>> 
>> I cannot figure out what's special about this comment that freaks the re-display.
>
> Most probably the unbalanced parentheses.  Something in sml-mode
> causes the display engine to hit on that many times.

Maybe because comments in SML can be nested.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

bug#24205: 25.1.50; Re-display issue (lag) when opening certain files [sml-mode]

[Kaushal Modi]

[-- Attachment #1: Type: text/plain, Size: 1966 bytes --]

On Thu, Aug 11, 2016 at 4:28 PM Eli Zaretskii <eliz@gnu.org> wrote:

> I must explain something so that my requests don't sound arbitrary.
> When I work on a problem, I usually run Emacs many times in different
> ways, most of the times under a debugger.  Since the debugger (and
> other tools) need to read my init files for working properly as I'm
> used to, I cannot easily change my home directory, or mess with my
> init files during the debugging session.  In addition, with 10-20 sec
> delay I don't need a profiler to notice it.
>
> So it is much more helpful to have a recipe that is as simple as
> possible (so invoking it many times won't be painful), avoids
> interfering with my init files and system configuration, and includes
> only the stuff that is directly relevant to the bug reproduction.
>
> A good recipe along the above lines could easily speed up a bugfix by
> hours and even days.  So if you can afford that, please try to follow
> these guidelines when you prepare a recipe.
>
> I hope this clarifies the reason for my requests.
>

Makes sense. I was under the impression that having an all-inclusive MWE
was more useful and so I went through the effort of creating one :) I was
under the impression that you would run "r -Q" in gdb and simply paste my
code in there to get going. But now I understand that you need your
personal config too to help you debug in the most efficient way.

As for the recipe for this one:

1. Install sml-mode from GNU Elpa.
2. Save a file called foo.sml with below content:
=====
 (* ''a is a special equality type (it is not the same as the generic 'a
type. It *)
=====
Note that while doing so, if the buffer is already in sml-mode, you might
not even need to go to step 3. Because the same thing happened to me when I
encountered this bug for the first time. I was actually typing out that
comment, and my emacs started freezing as I was typing.
3. Open it.

Thanks for working on this.
-- 

Kaushal Modi

[-- Attachment #2: Type: text/html, Size: 2751 bytes --]

bug#24205: 25.1.50; Re-display issue (lag) when opening certain files [sml-mode]

[Stefan Monnier]

>  (* ''a is a special equality type (it is not the same as the generic 'a
> type. It *)

Apparently, it's the following font-lock rule which takes forever:

    (,(concat "\\_<\\(\\(?:data\\|abs\\|with\\|eq\\)?type\\)\\s-+"
              sml-tyvarseq-re "\\(" sml-id-re "\\)")
     (1 font-lock-keyword-face)
     (2 font-lock-type-def-face))

The sml-tyvarseq-re part of it matches type annotation which typically
take the form ('a, 'b), so it normally matches things like

     datatype ('a, 'b) foo
or
     type ('a, 'b) foo

and the regexp is loose enough to match any mix of , ' and identifiers
inside parentheses.  This in itself is not a big problem, except that
the regexp is not careful enough and match "abc" either as a single
identifier, or as 2 identifiers (in two different ways: "a" followed by
"bc" or "ab" "followed by "c"), or as 3 identifiers.  For longish words
like "generic" you can guess that it ends up with very many different
ways to match, so when the match fails, it takes many trials to notice
that none of the ways succeeds.

Can you try the patch below, which tightens the regexp a bit and tries
to remove these redundant ways to match the same thing?


        Stefan


diff --git a/packages/sml-mode/sml-mode.el b/packages/sml-mode/sml-mode.el
index e94b571..f4adf11 100644
--- a/packages/sml-mode/sml-mode.el
+++ b/packages/sml-mode/sml-mode.el
@@ -257,8 +257,8 @@ notion of \"the end of an outline\".")
   (defconst sml-id-re "\\sw\\(?:\\sw\\|\\s_\\)*"))
 
 (defconst sml-tyvarseq-re
-  (concat "\\(?:\\(?:'+" sml-id-re "\\|(\\(?:[,']\\|" sml-id-re
-          "\\|\\s-\\)+)\\)\\s-+\\)?"))
+  (concat "\\(?:\\(?:'+" sml-id-re "\\|(\\(?:[,' \t\n]+" sml-id-re
+          "\\)+)\\)\\s-+\\)?"))
 
 ;;; Font-lock settings ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
 

bug#24205: 25.1.50; Re-display issue (lag) when opening certain files [sml-mode]

[Stefan Monnier]

> Can you try the patch below, which tightens the regexp a bit and tries
> to remove these redundant ways to match the same thing?

For the lack of answer I assume that it fixes the problem and doesn't
introduce new ones (and hopefully it might bring world peace while it's
at it), so installed,


        Stefan

bug#24205: 25.1.50; Re-display issue (lag) when opening certain files [sml-mode]

[Kaushal Modi]

[-- Attachment #1: Type: text/plain, Size: 436 bytes --]

On Sun, Aug 14, 2016 at 9:26 AM Stefan Monnier <monnier@iro.umontreal.ca>
wrote:

> For the lack of answer I assume that it fixes the problem and doesn't
> introduce new ones (and hopefully it might bring world peace while it's
> at it), so installed,
>

Hi Stefan,

Oddly, I missed that that email had a patch. I now tried it out and it
works! I do not see emacs get freeze any more because of that comment.

Thanks!
-- 

Kaushal Modi

[-- Attachment #2: Type: text/html, Size: 871 bytes --]