bug#29889: 27.0.50; Slow visual selection

bug#29889: 27.0.50; Slow visual selection

[Sujith]

Visual selection of text becomes very slow and hogs the
CPU in some cases.

For example, open the file lisp/progmodes/vhdl-mode.el in
the emacs codebase. And then, to reproduce this issue:

* Scroll patiently to the bottom using C-v.
  (this is essential, jumping to the bottom doesn't seem to bring
  up this issue).
* Set mark with C-SPC.
* Go to the beginning with M-<.
* Now move the cursor up and down.

The selection is jerky and CPU usage is very high.
I have tried this with emacs -Q and can see the issue. I am using
the master branch.

A profile report with this issue:

- #<compiled 0x41937d>                                           2499  53%
 - filter-buffer-substring                                       2499  53%
  - buffer-substring--filter                                     2499  53%
   - #<compiled 0x17002e5>                                       2499  53%
      apply                                                      2499  53%
- ...                                                            2102  44%
   Automatic GC                                                  2085  44%
 - minibuffer-complete                                             17   0%
  - completion-in-region                                           17   0%
   - completion--in-region                                         17   0%
    - #<compiled 0x10e5b45>                                        17   0%
     - apply                                                       17   0%
      - #<compiled 0x243ba7>                                       17   0%
       - completion--in-region-1                                   17   0%
        - completion--do-completion                                17   0%
         - completion-try-completion                               17   0%
          - completion--nth-completion                             17   0%
           - completion--some                                      17   0%
            - #<compiled 0x52cfb1>                                 17   0%
             - completion-basic-try-completion                     17   0%
              - try-completion                                     17   0%
               - #<compiled 0x24664f>                              17   0%
                  complete-with-action                             17   0%
+ command-execute                                                  87   1%
+ gui-set-selection                                                16   0%
+ redisplay_internal (C function)                                   3   0%


If any more information is needed, please let me know !


In GNU Emacs 27.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version 2.24.31)
 of 2017-12-28 built on the-damned
Repository revision: b19df8ae78cdebe76512a70f76ec68677de41c11
Windowing system distributor 'The X.Org Foundation', version 11.0.11906000
Recent messages:
mwheel-scroll: Beginning of buffer [6 times]
scroll-down-command: Beginning of buffer
Undo!
uncompressing simple.el.gz...done
Note: file is write protected
uncompressing simple.el.gz...done
Note: file is write protected
scroll-down-command: Beginning of buffer [5 times]
Quit [2 times]
Making completion list...

Configured using:
 'configure --prefix=/usr --without-gconf --without-gsettings
 --without-selinux --without-gnutls --without-libsystemd
 --without-threads --without-dbus --with-x-toolkit=gtk2'

Configured features:
XPM JPEG TIFF GIF PNG RSVG SOUND GPM NOTIFY ACL LIBXML2 FREETYPE
M17N_FLT LIBOTF XFT ZLIB TOOLKIT_SCROLL_BARS GTK2 X11 LCMS2

Important settings:
  value of $LANG: en_IN.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Shell

Minor modes in effect:
  global-magit-file-mode: t
  diff-auto-refine-mode: t
  magit-auto-revert-mode: t
  global-git-commit-mode: t
  async-bytecomp-package-mode: t
  shell-dirtrack-mode: t
  display-time-mode: t
  iswitchb-mode: t
  savehist-mode: t
  override-global-mode: t
  save-place-mode: t
  cl-old-struct-compat-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  mouse-wheel-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  column-number-mode: 1
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow flyspell ispell face-remap emacsbug cc-mode cc-fonts cc-guess
cc-menus cc-cmds cc-styles cc-align cc-engine cc-vars cc-defs help-fns
radix-tree find-func profiler dired-aux elec-pair mu4e-alert pcase ht s
alert log4e rx notifications dbus xml gntp magit-obsolete magit-blame
magit-stash magit-bisect magit-remote magit-commit magit-sequence
magit-notes magit-worktree magit-branch magit-collab ghub url-auth url
url-proxy url-privacy url-expand url-methods url-history url-cookie
url-domsuf url-util mailcap let-alist magit-files magit-refs
magit-status magit magit-repos magit-apply magit-wip magit-log
magit-diff smerge-mode diff-mode magit-core magit-autorevert autorevert
filenotify magit-process magit-margin magit-mode git-commit magit-git
magit-section magit-utils crm magit-popup log-edit pcvs-util add-log
with-editor cl-extra help-mode async-bytecomp async shell pcomplete dash
advice mu4e-contrib mu4e desktop frameset mu4e-speedbar speedbar
sb-image ezimage dframe mu4e-main mu4e-context mu4e-view cal-menu
calendar cal-loaddefs thingatpt browse-url comint ansi-color
mu4e-headers mu4e-compose mu4e-draft mu4e-actions ido rfc2368 smtpmail
sendmail mu4e-mark mu4e-message html2text mu4e-proc mu4e-utils doc-view
jka-compr image-mode mu4e-lists mu4e-vars message rmc puny format-spec
rfc822 mml mml-sec epa derived epg gnus-util rmail rmail-loaddefs
mm-decode mm-bodies mm-encode mail-parse rfc2231 rfc2047 rfc2045 mm-util
ietf-drums mail-prsvr mailabbrev mail-utils gmm-utils mailheader hl-line
cl mu4e-meta battery time dired-x dired dired-loaddefs edmacro kmacro
xcscope ring zenburn-theme server iswitchb savehist bind-key easy-mmode
saveplace finder-inf info package easymenu epg-config url-handlers
url-parse auth-source cl-seq eieio eieio-core cl-macs eieio-loaddefs
password-cache json map url-vars seq byte-opt gv bytecomp byte-compile
cconv cl-loaddefs cl-lib time-date mule-util tooltip eldoc electric
uniquify ediff-hook vc-hooks lisp-float-type mwheel term/x-win x-win
term/common-win x-dnd tool-bar dnd fontset image regexp-opt fringe
tabulated-list replace newcomment text-mode elisp-mode lisp-mode
prog-mode register page menu-bar rfn-eshadow isearch timer select
scroll-bar mouse jit-lock font-lock syntax facemenu font-core
term/tty-colors frame cl-generic cham georgian utf-8-lang misc-lang
vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms cp51932
hebrew greek romanian slovak czech european ethiopic indian cyrillic
chinese composite charscript charprop case-table epa-hook jka-cmpr-hook
help simple abbrev obarray minibuffer cl-preloaded nadvice loaddefs
button faces cus-face macroexp files text-properties overlay sha1 md5
base64 format env code-pages mule custom widget hashtable-print-readable
backquote inotify lcms2 dynamic-setting font-render-setting move-toolbar
gtk x-toolkit x multi-tty make-network-process emacs)

Memory information:
((conses 16 385737 51929)
 (symbols 48 34919 1)
 (miscs 40 105 376)
 (strings 32 75656 5006)
 (string-bytes 1 2341110)
 (vectors 16 54734)
 (vector-slots 8 1167949 49516)
 (floats 8 149 357)
 (intervals 56 42540 1784)
 (buffers 992 18))

bug#29889: 27.0.50; Slow visual selection

[Eli Zaretskii]

> From: Sujith <m.sujith@gmail.com>
> Date: Fri, 29 Dec 2017 09:22:22 +0530
> 
> Visual selection of text becomes very slow and hogs the
> CPU in some cases.
> 
> For example, open the file lisp/progmodes/vhdl-mode.el in
> the emacs codebase. And then, to reproduce this issue:
> 
> * Scroll patiently to the bottom using C-v.
>   (this is essential, jumping to the bottom doesn't seem to bring
>   up this issue).
> * Set mark with C-SPC.
> * Go to the beginning with M-<.
> * Now move the cursor up and down.
> 
> The selection is jerky and CPU usage is very high.
> I have tried this with emacs -Q and can see the issue. I am using
> the master branch.

Confirmed.  Additional info:

This happens also on the emacs-26 branch and in the 26.0.90 pretest,
but not in Emacs 25.2.

It also isn't limited to *.el files: I see it with, e.g., xterm.c.

The painfully slow first step can be replaced with this much simpler
step:

  M-: (font-lock-fontify-region (point-min) (point-max)) RET

I guess some bisecting is in order.

bug#29889: 27.0.50; Slow visual selection

[Sujith]

Eli Zaretskii <eliz@gnu.org> writes:
> Confirmed.  Additional info:
>
> This happens also on the emacs-26 branch and in the 26.0.90 pretest,
> but not in Emacs 25.2.

I checked out the 'emacs-25.2' tag and the issue happens there also.

To get a valid starting point which could be seen as a 'good' commit
for git-bisect, I tried to go back in the tree. I went up to Jun-1-2016
and then I ran into build errors. But the issue was present at that
point too.

Anything else which I could do to help ?

bug#29889: 27.0.50; Slow visual selection

[Eli Zaretskii]

> From: Sujith <m.sujith@gmail.com>
> Cc: 29889@debbugs.gnu.org
> Date: Sat, 30 Dec 2017 07:06:05 +0530
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> > Confirmed.  Additional info:
> >
> > This happens also on the emacs-26 branch and in the 26.0.90 pretest,
> > but not in Emacs 25.2.
> 
> I checked out the 'emacs-25.2' tag and the issue happens there also.

You are right.  It turns out the issue all but disappears in an
optimized build; once I rebuilt Emacs 25.3 without optimizations, I
see this in that version as well.

However, I don't seem to be able to see the problem in a -nw session,
or maybe the slowness in non-GUI frames is just below the threshold of
keeping up with the keyboard auto-repeat rate.

Starting with Emacs 25.1, we switched to using an overlay for showing
the region, so the prime suspect at this point is the relatively
expensive redisplay when buffer overlays have changed since the
previous redisplay, especially in a buffer with many text properties
(produced by font-locking).  E.g., I see that with region highlighted,
every C-f causes a full redisplay of the window, because all the other
redisplay optimizations are disabled.

> To get a valid starting point which could be seen as a 'good' commit
> for git-bisect, I tried to go back in the tree. I went up to Jun-1-2016
> and then I ran into build errors. But the issue was present at that
> point too.

This sounds consistent with the hypothesis that the overlay-based
implementation of region highlighting is the trigger.

> Anything else which I could do to help ?

Run Emacs under 'perf' (or build with C-level profiling), and show the
C-level profile while moving the cursor with region highlighted?  That
should tell whether my hypothesis above holds any water, and if not,
point out some other suspects.

Thanks.

bug#29889: 27.0.50; Slow visual selection

[Sujith]

Eli Zaretskii <eliz@gnu.org> writes:
> Run Emacs under 'perf' (or build with C-level profiling), and show the
> C-level profile while moving the cursor with region highlighted?  That
> should tell whether my hypothesis above holds any water, and if not,
> point out some other suspects.

I did 'perf record emacs' and then 'perf report --stdio'.

Profile report (just the top few lines):

# Overhead  Command      Shared Object               Symbol                                        
# ........  ...........  ..........................  ................................
#
    41.09%  emacs        emacs-27.0.50               [.] mark_object
    10.90%  emacs        emacs-27.0.50               [.] balance_an_interval
     4.23%  emacs        emacs-27.0.50               [.] mark_interval
     3.77%  emacs        emacs-27.0.50               [.] Flength
     3.45%  emacs        emacs-27.0.50               [.] sweep_strings
     3.11%  emacs        emacs-27.0.50               [.] sweep_conses
     2.94%  emacs        emacs-27.0.50               [.] balance_intervals_internal
     2.85%  emacs        emacs-27.0.50               [.] sweep_intervals
     1.91%  emacs        emacs-27.0.50               [.] traverse_intervals_noorder
     1.70%  emacs        emacs-27.0.50               [.] mark_char_table
     1.65%  emacs        emacs-27.0.50               [.] next_interval
     1.52%  emacs        emacs-27.0.50               [.] copy_intervals
     1.50%  emacs        emacs-27.0.50               [.] concat
     1.43%  emacs        emacs-27.0.50               [.] sweep_vectors
     1.28%  emacs        emacs-27.0.50               [.] scan_sexps_forward
     0.93%  emacs        emacs-27.0.50               [.] Fcons
     0.84%  emacs        emacs-27.0.50               [.] exec_byte_code
     0.81%  emacs        emacs-27.0.50               [.] sweep_symbols
     0.72%  emacs        emacs-27.0.50               [.] re_match_2_internal


Using gprof with --enable-profiling, the report looks like:

Flat profile:

Each sample counts as 0.01 seconds.
  %   cumulative   self              self     total           
 time   seconds   seconds    calls  ms/call  ms/call  name    
 33.97      6.33     6.33 111195700     0.00     0.00  mark_object
 10.26      8.24     1.91 31639099     0.00     0.00  balance_an_interval
  3.54      8.90     0.66 22239242     0.00     0.00  mark_interval
  2.95      9.45     0.55     7431     0.07     0.25  balance_intervals_internal
  2.79      9.97     0.52      223     2.33    10.50  sweep_strings
  2.42     10.42     0.45  7826133     0.00     0.00  Flength
  2.42     10.87     0.45      223     2.02     2.26  sweep_conses
  2.09     11.26     0.39      223     1.75     1.89  sweep_intervals
  2.09     11.65     0.39   419665     0.00     0.00  assq_no_quit
  2.04     12.03     0.38 11148108     0.00     0.00  next_interval
  1.93     12.39     0.36    42738     0.01     0.02  scan_sexps_forward
  1.88     12.74     0.35  7482960     0.00     0.00  concat
  1.66     13.05     0.31 24810407     0.00     0.00  FETCH_MULTIBYTE_CHAR
  1.40     13.31     0.26    34595     0.01     0.01  mark_char_table
  1.34     13.56     0.25     7431     0.03     0.12  traverse_intervals_noorder
  1.29     13.80     0.24   422335     0.00     0.00  exec_byte_code
  1.18     14.02     0.22      223     0.99     1.14  sweep_vectors
  1.07     14.22     0.20      282     0.71     0.71  evxprintf

bug#29889: 27.0.50; Slow visual selection

[Eli Zaretskii]

On December 31, 2017 7:25:32 AM GMT+02:00, Sujith <m.sujith@gmail.com> wrote:
> Eli Zaretskii <eliz@gnu.org> writes:
> > Run Emacs under 'perf' (or build with C-level profiling), and show
> the
> > C-level profile while moving the cursor with region highlighted? 
> That
> > should tell whether my hypothesis above holds any water, and if not,
> > point out some other suspects.
> 
> I did 'perf record emacs' and then 'perf report --stdio'.
> 
> Profile report (just the top few lines):
> 
> # Overhead  Command      Shared Object               Symbol           
>                             
> # ........  ...........  .......................... 
> ................................
> #
>     41.09%  emacs        emacs-27.0.50               [.] mark_object
> 10.90%  emacs        emacs-27.0.50               [.]
> balance_an_interval
>      4.23%  emacs        emacs-27.0.50               [.] mark_interval
>      3.77%  emacs        emacs-27.0.50               [.] Flength
>      3.45%  emacs        emacs-27.0.50               [.] sweep_strings
>      3.11%  emacs        emacs-27.0.50               [.] sweep_conses
> 2.94%  emacs        emacs-27.0.50               [.]
> balance_intervals_internal
>    2.85%  emacs        emacs-27.0.50               [.] sweep_intervals
> 1.91%  emacs        emacs-27.0.50               [.]
> traverse_intervals_noorder
>    1.70%  emacs        emacs-27.0.50               [.] mark_char_table
>      1.65%  emacs        emacs-27.0.50               [.] next_interval
>     1.52%  emacs        emacs-27.0.50               [.] copy_intervals
>      1.50%  emacs        emacs-27.0.50               [.] concat
>      1.43%  emacs        emacs-27.0.50               [.] sweep_vectors
> 1.28%  emacs        emacs-27.0.50               [.] scan_sexps_forward
>      0.93%  emacs        emacs-27.0.50               [.] Fcons
>     0.84%  emacs        emacs-27.0.50               [.] exec_byte_code
>      0.81%  emacs        emacs-27.0.50               [.] sweep_symbols
> 0.72%  emacs        emacs-27.0.50               [.]
> re_match_2_internal
> 
> 
> Using gprof with --enable-profiling, the report looks like:
> 
> Flat profile:
> 
> Each sample counts as 0.01 seconds.
>   %   cumulative   self              self     total           
>  time   seconds   seconds    calls  ms/call  ms/call  name    
>  33.97      6.33     6.33 111195700     0.00     0.00  mark_object
> 10.26      8.24     1.91 31639099     0.00     0.00 
> balance_an_interval
>   3.54      8.90     0.66 22239242     0.00     0.00  mark_interval
> 2.95      9.45     0.55     7431     0.07     0.25 
> balance_intervals_internal
>   2.79      9.97     0.52      223     2.33    10.50  sweep_strings
>   2.42     10.42     0.45  7826133     0.00     0.00  Flength
>   2.42     10.87     0.45      223     2.02     2.26  sweep_conses
>   2.09     11.26     0.39      223     1.75     1.89  sweep_intervals
>   2.09     11.65     0.39   419665     0.00     0.00  assq_no_quit
>   2.04     12.03     0.38 11148108     0.00     0.00  next_interval
> 1.93     12.39     0.36    42738     0.01     0.02  scan_sexps_forward
>   1.88     12.74     0.35  7482960     0.00     0.00  concat
> 1.66     13.05     0.31 24810407     0.00     0.00 
> FETCH_MULTIBYTE_CHAR
>   1.40     13.31     0.26    34595     0.01     0.01  mark_char_table
> 1.34     13.56     0.25     7431     0.03     0.12 
> traverse_intervals_noorder
>   1.29     13.80     0.24   422335     0.00     0.00  exec_byte_code
>   1.18     14.02     0.22      223     0.99     1.14  sweep_vectors
>   1.07     14.22     0.20      282     0.71     0.71  evxprintf

Thanks a lot!

Since mark_object appears high in the profile, could you please
rerun the experiment after setting gc-cons-threshold and
gc-cons-percentage so as to avoid GC for the time of the expdriment?

bug#29889: 27.0.50; Slow visual selection

[Eli Zaretskii]

On December 31, 2017 9:20:19 AM GMT+02:00, Eli Zaretskii <eliz@gnu.org> wrote:
> On December 31, 2017 7:25:32 AM GMT+02:00, Sujith <m.sujith@gmail.com>
> wrote:
> > Eli Zaretskii <eliz@gnu.org> writes:
> > > Run Emacs under 'perf' (or build with C-level profiling), and show
> > the
> > > C-level profile while moving the cursor with region highlighted? 
> > That
> > > should tell whether my hypothesis above holds any water, and if
> not,
> > > point out some other suspects.
> > 
> > I did 'perf record emacs' and then 'perf report --stdio'.
> > 
> > Profile report (just the top few lines):
> > 
> > # Overhead  Command      Shared Object               Symbol         
>  
> >                             
> > # ........  ...........  .......................... 
> > ................................
> > #
> >     41.09%  emacs        emacs-27.0.50               [.] mark_object
> > 10.90%  emacs        emacs-27.0.50               [.]
> > balance_an_interval
> >      4.23%  emacs        emacs-27.0.50               [.]
> mark_interval
> >      3.77%  emacs        emacs-27.0.50               [.] Flength
> >      3.45%  emacs        emacs-27.0.50               [.]
> sweep_strings
> >      3.11%  emacs        emacs-27.0.50               [.]
> sweep_conses
> > 2.94%  emacs        emacs-27.0.50               [.]
> > balance_intervals_internal
> >    2.85%  emacs        emacs-27.0.50               [.]
> sweep_intervals
> > 1.91%  emacs        emacs-27.0.50               [.]
> > traverse_intervals_noorder
> >    1.70%  emacs        emacs-27.0.50               [.]
> mark_char_table
> >      1.65%  emacs        emacs-27.0.50               [.]
> next_interval
> >     1.52%  emacs        emacs-27.0.50               [.]
> copy_intervals
> >      1.50%  emacs        emacs-27.0.50               [.] concat
> >      1.43%  emacs        emacs-27.0.50               [.]
> sweep_vectors
> > 1.28%  emacs        emacs-27.0.50               [.]
> scan_sexps_forward
> >      0.93%  emacs        emacs-27.0.50               [.] Fcons
> >     0.84%  emacs        emacs-27.0.50               [.]
> exec_byte_code
> >      0.81%  emacs        emacs-27.0.50               [.]
> sweep_symbols
> > 0.72%  emacs        emacs-27.0.50               [.]
> > re_match_2_internal
> > 
> > 
> > Using gprof with --enable-profiling, the report looks like:
> > 
> > Flat profile:
> > 
> > Each sample counts as 0.01 seconds.
> >   %   cumulative   self              self     total           
> >  time   seconds   seconds    calls  ms/call  ms/call  name    
> >  33.97      6.33     6.33 111195700     0.00     0.00  mark_object
> > 10.26      8.24     1.91 31639099     0.00     0.00 
> > balance_an_interval
> >   3.54      8.90     0.66 22239242     0.00     0.00  mark_interval
> > 2.95      9.45     0.55     7431     0.07     0.25 
> > balance_intervals_internal
> >   2.79      9.97     0.52      223     2.33    10.50  sweep_strings
> >   2.42     10.42     0.45  7826133     0.00     0.00  Flength
> >   2.42     10.87     0.45      223     2.02     2.26  sweep_conses
> >   2.09     11.26     0.39      223     1.75     1.89 
> sweep_intervals
> >   2.09     11.65     0.39   419665     0.00     0.00  assq_no_quit
> >   2.04     12.03     0.38 11148108     0.00     0.00  next_interval
> > 1.93     12.39     0.36    42738     0.01     0.02 
> scan_sexps_forward
> >   1.88     12.74     0.35  7482960     0.00     0.00  concat
> > 1.66     13.05     0.31 24810407     0.00     0.00 
> > FETCH_MULTIBYTE_CHAR
> >   1.40     13.31     0.26    34595     0.01     0.01 
> mark_char_table
> > 1.34     13.56     0.25     7431     0.03     0.12 
> > traverse_intervals_noorder
> >   1.29     13.80     0.24   422335     0.00     0.00  exec_byte_code
> >   1.18     14.02     0.22      223     0.99     1.14  sweep_vectors
> >   1.07     14.22     0.20      282     0.71     0.71  evxprintf
> 
> Thanks a lot!
> 
> Since mark_object appears high in the profile, could you please
> rerun the experiment after setting gc-cons-threshold and
> gc-cons-percentage so as to avoid GC for the time of the expdriment?

Also, please profile only when you move the cursor in the last step
of the recipe.  I suspect that you profiled also the time during
fontification of the buffer in step 1, otherwise I don't understand
why balance_an_interval is called so many times.

bug#29889: 27.0.50; Slow visual selection

[Sujith]

I have set 'select-active-regions' to 'only' in my .emacs
to address this issue. Is this the recommended solution for
this problem ?

bug#29889: 27.0.50; Slow visual selection

[Eli Zaretskii]

> From: Sujith <m.sujith@gmail.com>
> Cc: bug-gnu-emacs@gnu.org, 29889@debbugs.gnu.org
> Date: Sat, 06 Jan 2018 06:45:22 +0530
> 
> I have set 'select-active-regions' to 'only' in my .emacs
> to address this issue. Is this the recommended solution for
> this problem ?

That would be my recommendation, yes.  Especially if you happen to
deal frequently with large regions, and you did not disable
transient-mark-mode.

In fact, I cannot understand why the default was changed to t in
7c23dd4.  The discussions leading to those changes all mention the
value 'lazy' (later renamed to 'only') as the default, and there's
nothing I could find explaining why t was eventually deemed a better
default.

Regardless, I still wonder whether region-extract-function should call
buffer-substring-no-properties, at least when it's used to set the
primary selection.  Stefan, any thoughts?

bug#29889: 27.0.50; Slow visual selection

[Stefan Monnier]

> In fact, I cannot understand why the default was changed to t in
> 7c23dd4.  The discussions leading to those changes all mention the
> value 'lazy' (later renamed to 'only') as the default, and there's
> nothing I could find explaining why t was eventually deemed a better
> default.

FWIW I was also surprised to see its default is t rather than `only`
(and even more surprised that I hadn't noticed it all this time).

I think to make t work well (i.e. to avoid the obvious performance issue
discussed in the current bug-report), we'd need to rework the code so as
to try and avoid re-allocating a complete brand new string all the time.

E.g. special-case the common situation where nothing in the buffer has
been modified since last time and only extract some kind of "adjustment"
(i.e. something that says "remove last N chars" or "append this string").

Or maybe extract the region lazily: only remember the start and end
position of the region in the post-command-hook and postpone extracting
the region until either the primary selection is requested or the text
is about to be changed/destroyed (i.e. from before-change-function or
kill-buffer-hook).

> Regardless, I still wonder whether region-extract-function should call
> buffer-substring-no-properties, at least when it's used to set the
> primary selection.  Stefan, any thoughts?

We could try to do that to reduce the pain a bit, but
region-extract-function should generally preserve text properties (when
its output is used by Emacs rather than by another application), so we'd
need to add some way to specify whether we want the properties or not.
I think I'd rather "fix it right" than add such a hack.


        Stefan

bug#29889: 27.0.50; Slow visual selection

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Sujith <m.sujith@gmail.com>,  29889@debbugs.gnu.org
> Date: Sat, 06 Jan 2018 10:37:27 -0500
> 
> > In fact, I cannot understand why the default was changed to t in
> > 7c23dd4.  The discussions leading to those changes all mention the
> > value 'lazy' (later renamed to 'only') as the default, and there's
> > nothing I could find explaining why t was eventually deemed a better
> > default.
> 
> FWIW I was also surprised to see its default is t rather than `only`
> (and even more surprised that I hadn't noticed it all this time).

So do we change it back to 'only' for Emacs 26?  Sounds a bit risky to
me, after it has been t for 2 major releases, with no one complaining
until now.

> I think to make t work well (i.e. to avoid the obvious performance issue
> discussed in the current bug-report), we'd need to rework the code so as
> to try and avoid re-allocating a complete brand new string all the time.
> 
> E.g. special-case the common situation where nothing in the buffer has
> been modified since last time and only extract some kind of "adjustment"
> (i.e. something that says "remove last N chars" or "append this string").

But do we have a way of "adjusting" a string like that?  Or did you
think of a primitive that avoids consing a new string (assuming the
text properties were already stripped from the string)?

> Or maybe extract the region lazily: only remember the start and end
> position of the region in the post-command-hook and postpone extracting
> the region until either the primary selection is requested or the text
> is about to be changed/destroyed (i.e. from before-change-function or
> kill-buffer-hook).

That sounds cleaner to me.  Also much easier to implement.

But it can only be done on master, I think.

bug#29889: [SUSPECTED SPAM] Re: bug#29889: 27.0.50; Slow visual selection

[Stefan Monnier]

> But do we have a way of "adjusting" a string like that?

I was thinking of storing the "selected text" not as a single string but
a list of "operations" (from which we could generate the actual string
upon request).

> That sounds cleaner to me.  Also much easier to implement.

I think so too (I wrote them in the order in which they occurred to me).

> But it can only be done on master, I think.

Definitely.


        Stefan

bug#29889: 27.0.50; Slow visual selection

[martin rudalics]

 > So do we change it back to 'only' for Emacs 26?  Sounds a bit risky to
 > me, after it has been t for 2 major releases, with no one complaining
 > until now.

See also Bug#29661.

martin

bug#29889: 27.0.50; Slow visual selection

[Eli Zaretskii]

> Date: Sun, 07 Jan 2018 17:08:28 +0100
> From: martin rudalics <rudalics@gmx.at>
> CC: 29889@debbugs.gnu.org, m.sujith@gmail.com
> 
>  > So do we change it back to 'only' for Emacs 26?  Sounds a bit risky to
>  > me, after it has been t for 2 major releases, with no one complaining
>  > until now.
> 
> See also Bug#29661.

Yes.  However, I don't feel I can defend the change of the default
value in Emacs 26, unless I hear more support from developers.

bug#29889: 27.0.50; Slow visual selection

[Stefan Monnier]

> Yes.  However, I don't feel I can defend the change of the default
> value in Emacs 26, unless I hear more support from developers.

I don't have a strong opinion either way:
- on the one hand, it seems very harmless in terms of the risk to break
  existing Elisp packages: the change should only affect
  interactive behavior.
- on the other hand, we lived with this for 2 years now, so it's not
  terribly urgent.


        Stefan

bug#29889: 27.0.50; Slow visual selection

[Sujith]

Eli Zaretskii <eliz@gnu.org> writes:
> That would be my recommendation, yes.  Especially if you happen to
> deal frequently with large regions, and you did not disable
> transient-mark-mode.

I select large regions to run 'clang-format-region' to cleanup
code in a preferred format, so I can see this quite frequently.

Thanks for looking into this.

bug#29889: 27.0.50; Slow visual selection

[Sujith]

Eli Zaretskii <eliz@gnu.org> writes:
> Since mark_object appears high in the profile, could you please
> rerun the experiment after setting gc-cons-threshold and
> gc-cons-percentage so as to avoid GC for the time of the expdriment?

After doing (setq gc-cons-threshold 1000000000), the issue doesn't
seem to happen. The cursor moved around freely except for one
interruption - maybe the GC kicked in then.

Profile report using 'perf record -p `pidof emacs`' just before
starting to move the cursor:

# Samples: 39K of event 'cycles:ppp'
# Event count (approx.): 21976730020
#
# Overhead  Command  Shared Object               Symbol                                   
# ........  .......  ..........................  .........................................
#
    35.09%  emacs    emacs-27.0.50               [.] balance_an_interval
     9.36%  emacs    emacs-27.0.50               [.] Flength
     7.38%  emacs    emacs-27.0.50               [.] lisp_align_free
     5.12%  emacs    emacs-27.0.50               [.] next_interval
     5.12%  emacs    emacs-27.0.50               [.] concat
     3.74%  emacs    emacs-27.0.50               [.] copy_intervals
     2.54%  emacs    emacs-27.0.50               [.] Fcons
     2.13%  emacs    libc-2.26.so                [.] __memmove_sse2_unaligned_erms
     1.94%  emacs    emacs-27.0.50               [.] assq_no_quit
     1.44%  emacs    emacs-27.0.50               [.] copy_properties
     1.39%  emacs    emacs-27.0.50               [.] mem_insert
     1.35%  emacs    emacs-27.0.50               [.] sweep_intervals
     1.34%  emacs    libc-2.26.so                [.] _int_malloc
     1.17%  emacs    emacs-27.0.50               [.] make_interval
     1.14%  emacs    emacs-27.0.50               [.] Fmake_list
     1.02%  emacs    emacs-27.0.50               [.] sweep_conses
     0.76%  emacs    emacs-27.0.50               [.] x_produce_glyphs
     0.76%  emacs    emacs-27.0.50               [.] split_interval_right
     0.76%  emacs    libc-2.26.so                [.] _int_free
     0.65%  emacs    emacs-27.0.50               [.] lookup_char_property
     0.61%  emacs    libc-2.26.so                [.] malloc
     0.57%  emacs    emacs-27.0.50               [.] mem_find.part.9
     0.55%  emacs    emacs-27.0.50               [.] Fcopy_sequence

bug#29889: 27.0.50; Slow visual selection

[Eli Zaretskii]

> From: Sujith <m.sujith@gmail.com>
> Cc: 29889@debbugs.gnu.org
> Date: Sun, 31 Dec 2017 14:12:23 +0530
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> > Since mark_object appears high in the profile, could you please
> > rerun the experiment after setting gc-cons-threshold and
> > gc-cons-percentage so as to avoid GC for the time of the expdriment?
> 
> After doing (setq gc-cons-threshold 1000000000), the issue doesn't
> seem to happen. The cursor moved around freely except for one
> interruption - maybe the GC kicked in then.

Oh, you are right: if I set garbage-collection-messages non-nil, I see
a GC message each time I move the cursor.

So I guess my original theory was probably wrong, and the actual
suspect is some code, yet to be discovered, that conses such large
amounts of Lisp data.  I will look into it if no one beats me to it.

Thanks!

bug#29889: 27.0.50; Slow visual selection

[Eli Zaretskii]

> Date: Sun, 31 Dec 2017 17:37:52 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: 29889@debbugs.gnu.org
> 
> > After doing (setq gc-cons-threshold 1000000000), the issue doesn't
> > seem to happen. The cursor moved around freely except for one
> > interruption - maybe the GC kicked in then.
> 
> Oh, you are right: if I set garbage-collection-messages non-nil, I see
> a GC message each time I move the cursor.
> 
> So I guess my original theory was probably wrong, and the actual
> suspect is some code, yet to be discovered, that conses such large
> amounts of Lisp data.  I will look into it if no one beats me to it.

Long story short: set select-active-regions to 'only' or nil, and the
problem goes away.

Here's what happens: select-active-regions is now t by default, since
Emacs 24.1.  When that variable is t, every command, except those in
the list selection-inhibit-update-commands (a variable that is not
documented in any manual, btw), causes us to set the primary X
selection with the text in the active region.  Doing that invokes the
value of region-extract-function, which makes a string out of the
active region, which in this case is the entire buffer text.  For a
large buffer, such as vhdl.el, this conses a large string, thus
triggering GC _on_every_keystroke_.  And that _does_ make Emacs slow.

To add insult to injury, region-extract-function calls
buffer-substring--filter, which calls buffer-substring.  In a buffer
with a lot of text properties, such as the one that is fully
fontified, this involves copying the properties from the buffer to the
newly created string, most probably just to remove the properties
right after that (because the X selection doesn't need them).  Which
explains why a fully fontified buffer made things even more slow.

So how do we resolve this issue?  Should we set select-active-regions
to 'only' by default?  Should we arrange for region-extract-function
to call buffer-substring-no-properties instead, at least in this
specific use case?  Should we do both?  Something else?

bug#29889: 27.0.50; Slow visual selection

[Lars Ingebrigtsen]

Sujith <m.sujith@gmail.com> writes:

> Visual selection of text becomes very slow and hogs the
> CPU in some cases.
>
> For example, open the file lisp/progmodes/vhdl-mode.el in
> the emacs codebase. And then, to reproduce this issue:
>
> * Scroll patiently to the bottom using C-v.
>   (this is essential, jumping to the bottom doesn't seem to bring
>   up this issue).
> * Set mark with C-SPC.
> * Go to the beginning with M-<.
> * Now move the cursor up and down.
>
> The selection is jerky and CPU usage is very high.

(I'm going through old bug reports that unfortunately weren't resolved
at the time.)

I'm unable to reproduce the issue in recent Emacs versions (like 28.1).
Are you still seeing the problem?

Skimming the rest of the thread, the discussion was about
select-active-regions and whether to have it to `only' or t.  It's still
t (which, if I understand correctly, was the problematic value).  

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#29889: 27.0.50; Slow visual selection

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Skimming the rest of the thread, the discussion was about
> select-active-regions and whether to have it to `only' or t.  It's still
> t (which, if I understand correctly, was the problematic value).  

The doc string here is confusing (and wrong?)

--------
If non-nil, an active region automatically sets the primary selection.
If the value is ‘only’, only temporarily active regions (usually made
by mouse-dragging or shift-selection) set the window selection.

This takes effect only when Transient Mark mode is enabled.
--------

"window selection" should probably be "primary selection"?  The "This"
is confusing -- is it referring to `only' or to all non-nil values?  I
think it's the latter.

In any case, I think changing the default to `only' here would make a
lot of sense, but on the other hand, the t value has been the default
for a long time, so changing it now might just be too annoying.

Eli, what do you think?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#29889: 27.0.50; Slow visual selection

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: 29889@debbugs.gnu.org, Eli Zaretskii <eliz@gnu.org>
> Date: Fri, 20 May 2022 04:13:09 +0200
> 
> Lars Ingebrigtsen <larsi@gnus.org> writes:
> 
> > Skimming the rest of the thread, the discussion was about
> > select-active-regions and whether to have it to `only' or t.  It's still
> > t (which, if I understand correctly, was the problematic value).  
> 
> The doc string here is confusing (and wrong?)
> 
> --------
> If non-nil, an active region automatically sets the primary selection.
> If the value is ‘only’, only temporarily active regions (usually made
> by mouse-dragging or shift-selection) set the window selection.
> 
> This takes effect only when Transient Mark mode is enabled.
> --------
> 
> "window selection" should probably be "primary selection"?

Probably what was meant is "window-system's primary selection".

> The "This" is confusing -- is it referring to `only' or to all
> non-nil values?  I think it's the latter.

Yes.

> In any case, I think changing the default to `only' here would make a
> lot of sense, but on the other hand, the t value has been the default
> for a long time, so changing it now might just be too annoying.
> 
> Eli, what do you think?

I have that set to 'only' long ago.  What does that tell you? ;-)

bug#29889: 27.0.50; Slow visual selection

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> "window selection" should probably be "primary selection"?
>
> Probably what was meant is "window-system's primary selection".
>
>> The "This" is confusing -- is it referring to `only' or to all
>> non-nil values?  I think it's the latter.
>
> Yes.

I've now updated the doc string.

>> In any case, I think changing the default to `only' here would make a
>> lot of sense, but on the other hand, the t value has been the default
>> for a long time, so changing it now might just be too annoying.
>> 
>> Eli, what do you think?
>
> I have that set to 'only' long ago.  What does that tell you? ;-)

That we should consider flipping the default to `only'.  Let's see
whether anybody objects, and then do it.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#29889: 27.0.50; Slow visual selection

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> In any case, I think changing the default to `only' here would make a
> lot of sense, but on the other hand, the t value has been the default
> for a long time, so changing it now might just be too annoying.

I didn't yet read the rest of the bug report, but AFAIK we already have
a way to set a selection to a pair of positions in a buffer.  Requestors
get the contents of the buffer between those two positions, but no
string is consed until a program actually asks for the contents of the
selection.

We could have a new value of `select-active-regions' that tells Emacs to
set the primary selections to buffer positions if the active region is
not temporary, thereby avoiding the unreasonably high amount of string
consing.

The only problem is that this feature is only implemented on X and
Haiku, and not consing a string every time the selection is set is
impossible outside X.

bug#29889: 27.0.50; Slow visual selection

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Po Lu <luangruo@yahoo.com> writes:

> I didn't yet read the rest of the bug report, but AFAIK we already have
> a way to set a selection to a pair of positions in a buffer.  Requestors
> get the contents of the buffer between those two positions, but no
> string is consed until a program actually asks for the contents of the
> selection.
>
> We could have a new value of `select-active-regions' that tells Emacs to
> set the primary selections to buffer positions if the active region is
> not temporary, thereby avoiding the unreasonably high amount of string
> consing.
>
> The only problem is that this feature is only implemented on X and
> Haiku, and not consing a string every time the selection is set is
> impossible outside X.

We could also set the selection to two buffer positions or avoid setting
the primary selection if the region is more than N characters in length.

bug#29889: 27.0.50; Slow visual selection

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: 29889@debbugs.gnu.org,  Eli Zaretskii <eliz@gnu.org>,  Sujith
>  <m.sujith@gmail.com>
> Date: Fri, 20 May 2022 16:39:54 +0800
> 
> We could also set the selection to two buffer positions or avoid setting
> the primary selection if the region is more than N characters in length.

Not on MS-Windows, I think.

bug#29889: 27.0.50; Slow visual selection

[Lars Ingebrigtsen]

Po Lu <luangruo@yahoo.com> writes:

> I didn't yet read the rest of the bug report, but AFAIK we already have
> a way to set a selection to a pair of positions in a buffer.  Requestors
> get the contents of the buffer between those two positions, but no
> string is consed until a program actually asks for the contents of the
> selection.
>
> We could have a new value of `select-active-regions' that tells Emacs to
> set the primary selections to buffer positions if the active region is
> not temporary, thereby avoiding the unreasonably high amount of string
> consing.
>
> The only problem is that this feature is only implemented on X and
> Haiku, and not consing a string every time the selection is set is
> impossible outside X.

Yes, that would solve the performance problem, but the surprising
behaviour would still exist.

That is, I think most people would not expect that just setting the mark
in a buffer would make the part between mark and point available in the
selection in other programs.

So I think just flipping the default to `only' would be the best thing
here, even if the current value has been in effect since 24.1,
apparently.  (And then mention this as an incompatible change in NEWS,
of course.)

So I think I'll do that, unless somebody objects a lot.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#29889: 27.0.50; Slow visual selection

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Yes, that would solve the performance problem, but the surprising
> behaviour would still exist.
>
> That is, I think most people would not expect that just setting the mark
> in a buffer would make the part between mark and point available in the
> selection in other programs.

Why is that surprising, though?  Activating the mark makes the region
display in the "region" face, so its availability in the primary
selection becomes obvious to the user, since it really is "selected".

Consider typing M-h and then inserting the contents of the region into
another program with a middle-click, or typing C-h a and doing the same.

That is how other X programs behave, and isn't possible if
`select-active-regions' is `only'.

bug#29889: 27.0.50; Slow visual selection

[Lars Ingebrigtsen]

Po Lu <luangruo@yahoo.com> writes:

> Why is that surprising, though?  Activating the mark makes the region
> display in the "region" face, so its availability in the primary
> selection becomes obvious to the user, since it really is "selected".
>
> Consider typing M-h and then inserting the contents of the region into
> another program with a middle-click, or typing C-h a and doing the same.
>
> That is how other X programs behave, and isn't possible if
> `select-active-regions' is `only'.

Other programs don't have a concept of a point and mark, so that's
pretty moot.  In other programs, if you mark stuff with the mouse, or
the equivalent of the selecting directional keys, then things (may) end
up on the clipboard -- which is what `only' does.

(Sometimes even that doesn't make things end up on the clipboard, and
you have to type Cmd-C or the like.)

`M-h M-w' is the expected way to put the paragraph on the clipboard in
Emacs.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#29889: 27.0.50; Slow visual selection

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Sujith <m.sujith@gmail.com>,  Eli Zaretskii <eliz@gnu.org>,
>   29889@debbugs.gnu.org
> Date: Fri, 20 May 2022 12:11:12 +0200
> 
> `M-h M-w' is the expected way to put the paragraph on the clipboard in
> Emacs.

"M-h M-w" is the equivalent of "Select All" followed by Ctrl-C.  M-h
alone is the equivalent of just "Select All".  Aren't there programs
that set the primary selection when the user performs "Select All"?

bug#29889: 27.0.50; Slow visual selection

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> "M-h M-w" is the equivalent of "Select All" followed by Ctrl-C.  M-h
> alone is the equivalent of just "Select All".  Aren't there programs
> that set the primary selection when the user performs "Select All"?

`M-h' just marks the paragraph -- are you thinking of `C-x h'?  But the
principle is the same with that, I guess.

And there probably are.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#29889: 27.0.50; Slow visual selection

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

> "M-h M-w" is the equivalent of "Select All" followed by Ctrl-C.  M-h
> alone is the equivalent of just "Select All".  Aren't there programs
> that set the primary selection when the user performs "Select All"?

If you mean code in Emacs, then it's the current default value of
`select-active-regions'.  Otherwise, no, not that I know of.

bug#29889: 27.0.50; Slow visual selection

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: Lars Ingebrigtsen <larsi@gnus.org>,  m.sujith@gmail.com,
>   29889@debbugs.gnu.org
> Date: Fri, 20 May 2022 19:47:10 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > "M-h M-w" is the equivalent of "Select All" followed by Ctrl-C.  M-h
> > alone is the equivalent of just "Select All".  Aren't there programs
> > that set the primary selection when the user performs "Select All"?
> 
> If you mean code in Emacs, then it's the current default value of
> `select-active-regions'.  Otherwise, no, not that I know of.

Sorry, I don't understand the question.  I asked specifically about
other applications, where the user can do "Select All" to select the
text of an entire document -- does this set the primary selection in
those other applications?  If not, then I think your objection to use
'only' is not valid, because we aren't doing anything that the users
won't expect.

bug#29889: 27.0.50; Slow visual selection

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

> Sorry, I don't understand the question.  I asked specifically about
> other applications, where the user can do "Select All" to select the
> text of an entire document -- does this set the primary selection in
> those other applications?  If not, then I think your objection to use
> 'only' is not valid, because we aren't doing anything that the users
> won't expect.

Ah, I misunderstood the question.  Yes, "Select All" in other programs
make them set the primary selection.

bug#29889: 27.0.50; Slow visual selection

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: larsi@gnus.org,  m.sujith@gmail.com,  29889@debbugs.gnu.org
> Date: Fri, 20 May 2022 20:16:34 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Sorry, I don't understand the question.  I asked specifically about
> > other applications, where the user can do "Select All" to select the
> > text of an entire document -- does this set the primary selection in
> > those other applications?  If not, then I think your objection to use
> > 'only' is not valid, because we aren't doing anything that the users
> > won't expect.
> 
> Ah, I misunderstood the question.  Yes, "Select All" in other programs
> make them set the primary selection.

Then I think we should allow "C-x h" do the same, at least optionally.

bug#29889: 27.0.50; Slow visual selection

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Then I think we should allow "C-x h" do the same, at least optionally.

So a new value that's like `only', but also puts things into the primary
selection for commands like `C-x h' and `M-h'?  Yes, perhaps that'd be
nice -- but do we have a way to identify these commands?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#29889: 27.0.50; Slow visual selection

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Po Lu <luangruo@yahoo.com>,  m.sujith@gmail.com,  29889@debbugs.gnu.org
> Date: Fri, 20 May 2022 14:47:43 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Then I think we should allow "C-x h" do the same, at least optionally.
> 
> So a new value that's like `only', but also puts things into the primary
> selection for commands like `C-x h' and `M-h'?  Yes, perhaps that'd be
> nice -- but do we have a way to identify these commands?

Some property on the command's symbol, perhaps?

P.S. I've removed the OP from the CC list, since the address bounces.

bug#29889: 27.0.50; Slow visual selection

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> So a new value that's like `only', but also puts things into the primary
>> selection for commands like `C-x h' and `M-h'?  Yes, perhaps that'd be
>> nice -- but do we have a way to identify these commands?
>
> Some property on the command's symbol, perhaps?

Yes, that should work.  And there probably aren't that many of these
commands, so it should be possible to get them all marked with
reasonable confidence.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#29889: 27.0.50; Slow visual selection

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Yes, that should work.  And there probably aren't that many of these
> commands, so it should be possible to get them all marked with
> reasonable confidence.

But what if some other command moves the region?  For example, if you
type "C-x h" and then "C-f"?

The selection should also be updated then, I think, to match the text
displayed in the region face.  The best way to solve the performance
problem is probably to fall back to using buffer positions instead of
strings if the region exceeds a certain amount of characters in length.

bug#29889: 27.0.50; Slow visual selection

[Lars Ingebrigtsen]

Po Lu <luangruo@yahoo.com> writes:

> But what if some other command moves the region?  For example, if you
> type "C-x h" and then "C-f"?

I think for the new setting we're discussing, then that `C-f' should not
alter the selection (just like `open' doesn't today with the selections
it chooses).  That is, `C-x h' would say "put all of this in the
selection", just like `C-a' in Firefox does.

> The selection should also be updated then, I think, to match the text
> displayed in the region face.  The best way to solve the performance
> problem is probably to fall back to using buffer positions instead of
> strings if the region exceeds a certain amount of characters in length.

I think that sounds good for the t setting.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#29889: 27.0.50; Slow visual selection

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> I think for the new setting we're discussing, then that `C-f' should not
> alter the selection (just like `open' doesn't today with the selections
> it chooses).  That is, `C-x h' would say "put all of this in the
> selection", just like `C-a' in Firefox does.

But Firefox only says "own the selection", and transfers the contents of
whatever happens to be highlighted at the time of a selection request.

So if you move the highlighted area (with the moral equivalent of C-f),
requestors will get the contents of the new area.

bug#29889: 27.0.50; Slow visual selection

[Lars Ingebrigtsen]

Po Lu <luangruo@yahoo.com> writes:

>> I think for the new setting we're discussing, then that `C-f' should not
>> alter the selection (just like `open' doesn't today with the selections
>> it chooses).  That is, `C-x h' would say "put all of this in the
>> selection", just like `C-a' in Firefox does.
>
> But Firefox only says "own the selection", and transfers the contents of
> whatever happens to be highlighted at the time of a selection request.
>
> So if you move the highlighted area (with the moral equivalent of C-f),
> requestors will get the contents of the new area.

What's the equivalent of `C-f' in Firefox in this situation?  If you
mean "adjusting the area with the mouse", then that's already covered by
the definition of `only'.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#29889: 27.0.50; Slow visual selection

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> What's the equivalent of `C-f' in Firefox in this situation?  If you
> mean "adjusting the area with the mouse", then that's already covered by
> the definition of `only'.

Programmatically, via JavaScript:

  window.getSelection ().modify ("extend", "forward", "character");

or shift-selection.  The point is, whatever is displayed in the region
face must be sent to requestors asking for the primary selection.  We
should not invent distinctions between shift-selection, mouse dragging,
and cursor movement.

bug#29889: 27.0.50; Slow visual selection

[Lars Ingebrigtsen]

Po Lu <luangruo@yahoo.com> writes:

> Programmatically, via JavaScript:
>
>   window.getSelection ().modify ("extend", "forward", "character");

I don't think we have to care about that a lot...

> or shift-selection.

I tried `C-a' and the `S-<left>' and stuff in Firefox, but nothing seems
to be happening...

> The point is, whatever is displayed in the region
> face must be sent to requestors asking for the primary selection.  We
> should not invent distinctions between shift-selection, mouse dragging,
> and cursor movement.

But we do that already -- that's what `open' is about.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#29889: 27.0.50; Slow visual selection

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> I tried `C-a' and the `S-<left>' and stuff in Firefox, but nothing seems
> to be happening...

Are you testing that on X or Mac OS?  Remember that the primary
selection is inserted using Button2, not "Paste", and it only exists on
X.

> But we do that already -- that's what `open' is about.

It isn't the default.

bug#29889: 27.0.50; Slow visual selection

[Lars Ingebrigtsen]

Po Lu <luangruo@yahoo.com> writes:

>> I tried `C-a' and the `S-<left>' and stuff in Firefox, but nothing seems
>> to be happening...
>
> Are you testing that on X or Mac OS?  Remember that the primary
> selection is inserted using Button2, not "Paste", and it only exists on
> X.

I was testing on X.  I couldn't see that the `S-<left>' key did anything
visible in Firefox.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#29889: 27.0.50; Slow visual selection

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> I was testing on X.  I couldn't see that the `S-<left>' key did anything
> visible in Firefox.

That's odd.  Have you tried with add-ons and customizations disabled?

I remember there used to be an item in the Help menu for that, but
Mozilla seems to have deleted it, so you will have to temporarily rename
~/.mozilla to something else.

bug#29889: 27.0.50; Slow visual selection

[Lars Ingebrigtsen]

Po Lu <luangruo@yahoo.com> writes:

>> I was testing on X.  I couldn't see that the `S-<left>' key did anything
>> visible in Firefox.
>
> That's odd.  Have you tried with add-ons and customizations disabled?

It seems to depend on the web page.  I tried on four different ones, and
`S-<left>' did nothing.  But then I tried on
https://news.ycombinator.com/, and `S-<left>' does indeed adjust the
end of the region after `C-a' there.  (Is there a way to adjust the
start?)

But this is pretty obscure functionality in Firefox, while moving point
in Emacs is fundamental, so it shouldn't be normative for how we design
this.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#29889: 27.0.50; Slow visual selection

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> But this is pretty obscure functionality in Firefox, while moving point
> in Emacs is fundamental, so it shouldn't be normative for how we design
> this.

Emacs should _not_ invent distinctions between different methods of
changing the region, as long as it remains displayed in the region face,
since that is what tells the user what will be inserted with Button2.

If we want to make a distinction, then we should ensure that the region
face reflects the contents of the primary selection (instead of being
unconditionally displayed when the mark is active.)

bug#29889: 27.0.50; Slow visual selection

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Other programs don't have a concept of a point and mark, so that's
> pretty moot.  In other programs, if you mark stuff with the mouse, or
> the equivalent of the selecting directional keys, then things (may) end
> up on the clipboard -- which is what `only' does.

If you select text (i.e. it becomes highlighted) in any program, that
program owns the primary selection.  When another program requests the
primary selection, it is transferred what is currently highlighted in
the selection owner -- it doesn't matter how things were highlighted, as
long as it has been highlighted, the program should own the primary
selection.

When you select stuff with the mouse, nothing _ever_ ends up in the
clipboard.  The clipboard is only set when the user asks the program to
"copy" or "cut" text.

> (Sometimes even that doesn't make things end up on the clipboard, and
> you have to type Cmd-C or the like.)
>
> `M-h M-w' is the expected way to put the paragraph on the clipboard in
> Emacs.

You're mistaking the clipboard for the primary selection.

bug#29889: 27.0.50; Slow visual selection

[Lars Ingebrigtsen]

Po Lu <luangruo@yahoo.com> writes:

>> (Sometimes even that doesn't make things end up on the clipboard, and
>> you have to type Cmd-C or the like.)
>>
>> `M-h M-w' is the expected way to put the paragraph on the clipboard in
>> Emacs.
>
> You're mistaking the clipboard for the primary selection.

Yes, quite likely.  I was actually thinking about how things work on
Macos for a second -- if you mark text in a program there, it doesn't
seem to do anything inter-program (in my limited experience).  You have
to `Cmd-C' for it to happen.

But I guess Macos doesn't have a primary selection concept.

So perhaps the t value here is the one that makes most sense -- but it
should be more efficient, as you sketched in a previous message.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#29889: 27.0.50; Slow visual selection

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: Sujith <m.sujith@gmail.com>,  Eli Zaretskii <eliz@gnu.org>,
>   29889@debbugs.gnu.org
> Date: Fri, 20 May 2022 17:23:17 +0800
> 
> Consider typing M-h and then inserting the contents of the region into
> another program with a middle-click, or typing C-h a and doing the same.
> 
> That is how other X programs behave, and isn't possible if
> `select-active-regions' is `only'.

Maybe we need yet another value for that variable, which would set the
selection not only for shift-selection?

bug#29889: 27.0.50; Slow visual selection

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

> Maybe we need yet another value for that variable, which would set the
> selection not only for shift-selection?

I guess so.  How about a number indicating the maximum amount of
characters to put into the primary selection, which will not happen if
the region exceeds that many characters in length?

bug#29889: 27.0.50; Slow visual selection

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: larsi@gnus.org,  m.sujith@gmail.com,  29889@debbugs.gnu.org
> Date: Fri, 20 May 2022 19:51:17 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Maybe we need yet another value for that variable, which would set the
> > selection not only for shift-selection?
> 
> I guess so.  How about a number indicating the maximum amount of
> characters to put into the primary selection, which will not happen if
> the region exceeds that many characters in length?

That's bad UX, IMO: chances are you don't know that only part of the
text was put in the selection.  Or do you suggest displaying a warning
when text in the selection is truncated?

How do other apps handle request to put huge chunks of text into the
primary selection?  If they honor such requests seamlessly and without
slowing down the response, why cannot Emacs do the same?

bug#29889: 27.0.50; Slow visual selection

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

> That's bad UX, IMO: chances are you don't know that only part of the
> text was put in the selection.  Or do you suggest displaying a warning
> when text in the selection is truncated?

The latter, yes.

> How do other apps handle request to put huge chunks of text into the
> primary selection?  If they honor such requests seamlessly and without
> slowing down the response, why cannot Emacs do the same?

Those programs only "own" the selection without copying anything.  When
another program asks for the contents of the selection, they are sent
directly from the "buffer" containing them.  (That does mean if the
buffer contents change, so will the contents of the selection.)

bug#29889: 27.0.50; Slow visual selection

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: larsi@gnus.org,  m.sujith@gmail.com,  29889@debbugs.gnu.org
> Date: Fri, 20 May 2022 20:39:43 +0800
> 
> > How do other apps handle request to put huge chunks of text into the
> > primary selection?  If they honor such requests seamlessly and without
> > slowing down the response, why cannot Emacs do the same?
> 
> Those programs only "own" the selection without copying anything.  When
> another program asks for the contents of the selection, they are sent
> directly from the "buffer" containing them.  (That does mean if the
> buffer contents change, so will the contents of the selection.)

What if the buffer is killed?

Anyway, if other applications behave like that, why cannot Emacs do
the same?

bug#29889: 27.0.50; Slow visual selection

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

> What if the buffer is killed?

Then the selection is disowned, which means other programs will no
longer ask the buffer's program for the contents of the selection.

> Anyway, if other applications behave like that, why cannot Emacs do
> the same?

That would be setting the selection to a cons of two buffer positions.

We could also only do that upon the region reaching a given size, to
avoid disturbing people who rely on the current behavior.

bug#29889: 27.0.50; Slow visual selection

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: Sujith <m.sujith@gmail.com>,  Eli Zaretskii <eliz@gnu.org>,
>   29889@debbugs.gnu.org
> Date: Fri, 20 May 2022 16:35:46 +0800
> 
> We could have a new value of `select-active-regions' that tells Emacs to
> set the primary selections to buffer positions if the active region is
> not temporary, thereby avoiding the unreasonably high amount of string
> consing.

What will happen if the buffer text changes between the time the
active region is defined and the time some other program requests the
selection?

bug#29889: 27.0.50; Slow visual selection

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

> What will happen if the buffer text changes between the time the
> active region is defined and the time some other program requests the
> selection?

Then the new buffer text will be used.  But we could always store that
text in a temporary buffer, since inserting and deleting text from a
buffer is much faster (and doesn't cons nearly as much as) making a copy
of the text each time.

bug#29889: 27.0.50; Slow visual selection

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: larsi@gnus.org,  m.sujith@gmail.com,  29889@debbugs.gnu.org
> Date: Fri, 20 May 2022 19:49:19 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > What will happen if the buffer text changes between the time the
> > active region is defined and the time some other program requests the
> > selection?
> 
> Then the new buffer text will be used.

Which is not the expected result.

> But we could always store that text in a temporary buffer, since
> inserting and deleting text from a buffer is much faster (and
> doesn't cons nearly as much as) making a copy of the text each time.

It is true that buffers are cheaper than strings, but they still cons
objects and consume memory.  Buffers are also much more visible to
users than strings.  So I wonder why we should do something as
complicated as creating a temporary buffer.  Why not simply allocate
memory and store the text there, if we indeed care enough about this
issue (do we?).