bug#74281: 30.0.91; font-lock mode hangs on scrolling large Scheme file

[Eli Zaretskii <eliz@gnu.org>]

> Date: Sat, 09 Nov 2024 16:04:13 +0000
> From:  Divya Ranjan via "Bug reports for GNU Emacs,
>  the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>
> 
> 
> 1. In emacs -Q, open a fairly big Scheme file (> 70k) and try to move in
> the file with C-n, C-f or mouse scrolling. It sverely lags for several
> seconds, doesn’t even respond to C-g.

I can only reproduce this with C-p when moving from the end of the
file.  C-n and C-f are instantaneous here, even though this is an
unoptimized build of Emacs 30.

> 2. Toggle font-lock-mode off through M-x, then try to scroll. It works
> as expected, no serious lags.
> 
> 3. Running the profiler also shows font-lock as the culprit.

The profile I measured is below.  It seems like the bottleneck is
parse-partial-sexp, which means the syntax table
scheme-mode-syntax-table might be the immediate suspect.

> This makes working on big projects, in my case the source code of GNU
> Guix distribution an impossible situation. I haven’t tried setting up
> (Guile) Scheme with Tree-Sitter, but shouldn’t simple
> syntax-highlighting be not a serious problem?
> 
> To try a big Scheme file and reproduce this, open tex.scm in
> gnu/packages/tex.scm in Guix source tree:
> 
> https://git.savannah.gnu.org/cgit/guix.git

Stefan, what tools do we have to investigate slowness related to
parse-partial-sexp?  Or maybe you have suggestions for how to speed up
font-lock in this case?

Here's the profile I get while moving with C-p through the above file:

        1151  66% - ...
        1151  66%  - command-execute
        1151  66%   - call-interactively
        1151  66%    - funcall-interactively
        1151  66%     - previous-line
        1151  66%      - line-move
         798  45%       - line-pixel-height
         798  45%        - jit-lock-function
         798  45%         - jit-lock-fontify-now
         798  45%          - jit-lock--run-functions
         798  45%           - run-hook-wrapped
         798  45%            - #<byte-code-function CEC>
         798  45%             - font-lock-fontify-region
         798  45%              - font-lock-default-fontify-region
         798  45%               - font-lock-fontify-syntactically-region
         798  45%                - font-lock-default-fontify-syntactically
         798  45%                   parse-partial-sexp
         353  20%       - line-move-visual
         353  20%        - vertical-motion
         353  20%         - jit-lock-function
         353  20%          - jit-lock-fontify-now
         353  20%           - jit-lock--run-functions
         353  20%            - run-hook-wrapped
         353  20%             - #<byte-code-function BCE>
         353  20%              - font-lock-fontify-region
         353  20%               - font-lock-default-fontify-region
         353  20%                - font-lock-fontify-syntactically-region
         353  20%                 - font-lock-default-fontify-syntactically
         353  20%                    parse-partial-sexp
         585  33% - redisplay_internal (C function)
         577  33%  - jit-lock-function
         577  33%   - jit-lock-fontify-now
         577  33%    - jit-lock--run-functions
         577  33%     - run-hook-wrapped
         577  33%      - #<byte-code-function 5A0>
         577  33%       - font-lock-fontify-region
         577  33%        - font-lock-default-fontify-region
         577  33%         - font-lock-fontify-syntactically-region
         577  33%          - font-lock-default-fontify-syntactically
         577  33%             parse-partial-sexp
           6   0%   Automatic GC