bug#22404: 25.1.50; Forcing `window-scroll-functions` to run.

bug#22404: 25.1.50; Forcing `window-scroll-functions` to run.

[Keith David Bershatsky]

As a feature request, please consider adding an alternative method to force the `window-scroll-functions` hook to run -- but NOT more than it would normally run. [In other words, it shouldn't run 3 times.]  I only found `(set-window-buffer (selected-window) (current-buffer))` as a viable means to achieve this goal.  `run-window-scroll-functions` runs immediately and is not a viable substitute for this feature request.  Some users may not wish to `set-window-buffer` or run the `window-configuration-change-hook` to achieve this feature request.

In my particular use case, I want to run a custom function only one time per command loop that is dependent upon the new `window-start` and `window-end`, which is only reliably available to the user after the `post-command-hook` has run its course.  My function is too costly time-wise to run more than once each command loop.  I have tried, to no avail, to come up with a test to ascertain with 100% accuracy (from the `post-command-hook`) to guess whether the `window-scroll-functions` hook will run and/or whether it will run more than one time (e.g., when cursor/point is partially visible).  I played around with `(pos-visible-in-window-p nil nil t)` and whether it returned a length of 0, 2 or 6 -- however, from the `post-command-hook`, that is insufficient to ascertain with 100% accuracy whe
 ther the `window-scroll-functions` hook will run once or twice or not at all.  The solution, I believe, is to force the `window-scroll-functions` hook to run during every redisplay -- but not more than it would normally run.

In other words, I want the `window-scroll-functions` hook to run every command loop when my custom minor-mode is active -- or twice if point/cursor is only partially visible (at the bottom of the window) and needs to be moved up a smidgen by redisplay.

Background:  my custom function is a cross-hairs created with overlays that works with `word-wrap` -- the vertical line is XPM that matches the character and color underneath.

Thanks,

Keith

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

In GNU Emacs 25.1.50.1 (x86_64-apple-darwin10.8.0, NS appkit-1038.36 Version 10.6.8 (Build 10K549))
 of 2016-01-03 built on server.local
Repository revision: e94b1799d4f4c57266bdbc4801b26fe0121b7c7a
Windowing system distributor 'Apple', version 10.3.1038
Configured using:
 'configure --with-ns --without-imagemagick --enable-checking=glyphs
 CPPFLAGS=-I/Users/HOME/.0.data/.0.emacs/macports/include
 LDFLAGS=-L/Users/HOME/.0.data/.0.emacs/macports/lib'

Configured features:
JPEG RSVG DBUS NOTIFY ACL LIBXML2 ZLIB TOOLKIT_SCROLL_BARS NS

Important settings:
  locale-coding-system: utf-8-unix

Major mode: Grep

Minor modes in effect:
  tabbar-mode: t
  sb-mode: t
  ml-mode: t
  hl-mode: t
  kb-mode: t
  sd-mode: t
  fl-mode: t
  bc-mode: t
  buffer-read-only: t

Recent messages:

Load-path shadows:
None found.

Features:
(shadow emacsbug message mml mml-sec mm-decode mm-bodies mm-encode
gmm-utils mailheader sendmail lawlist-ztree lawlist-ys lawlist-ws
lawlist-wl elmo-imap4 elmo-localdir modb-standard modb-legacy
elmo-internal elmo-flag mmelmo-imap mmelmo-buffer elsp-generic mel-u
epg-config lawlist-w3m doc-view jka-compr image-mode ccl lawlist-vl
lawlist-view lawlist-undo lawlist-txt lawlist-tm lawlist-tex compare-w
diff-mode lawlist-tabbar lawlist-speedbar lawlist-shell info
esh-groups ehelp ange-ftp lawlist-sgml lawlist-sb lawlist-ruler
lawlist-replace lawlist-rectangle lawlist-re-builder lawlist-python
skeleton lawlist-profiler lawlist-print lawlist-php cl-seq cc-langs
lawlist-perl lawlist-parens lawlist-org lawlist-calendar org-agenda
org org-macro org-footnote org-pcomplete org-list org-faces
org-entities org-version ob-emacs-lisp ob ob-tangle ob-ref ob-lob
ob-table ob-exp org-src ob-keys ob-comint ob-core ob-eval org-compat
org-macs org-loaddefs find-func holidays hol-loaddefs cal-menu
calendar cal-loaddefs lawlist-neotree lawlist-movement lawlist-mouse
lawlist-ml lawlist-misc lawlist-messages lawlist-mc lawlist-markdown
noutline outline lawlist-lorem lawlist-linum lawlist-keymap lawlist-js
json map thingatpt cc-mode cc-fonts cc-guess cc-menus cc-cmds
cc-styles cc-align cc-engine cc-vars cc-defs lawlist-ispell
lawlist-isearch lawlist-info lawlist-imenu lawlist-ibuffer lawlist-hl
lawlist-grep lawlist-git pcvs-util ido seq server conf-mode
lawlist-framebufs lawlist-frame lawlist-fm lawlist-files zeroconf dbus
xml lawlist-env lawlist-elscreen lawlist-elisp lawlist-dv
lawlist-image lawlist-ds lawlist-dired dired dired-loaddefs
format-spec lawlist-diff lawlist-desktop frameset lawlist-saveplace
lawlist-debug lawlist-window debug lawlist-css smie lawlist-compile rx
lawlist-color lawlist-cm lawlist-cc lawlist-font-lock cl-macs
lawlist-calc lawlist-calc+ lawlist-bk lawlist-bc lawlist-bbdb gnus
gnus-ems nnheader mail-utils wid-edit mail-parse rfc2231 rfc2047
rfc2045 ietf-drums mailabbrev mail-extr rfc822 timezone
lawlist-minibuffer gv lawlist-auth gnus-util mm-util help-fns
mail-prsvr password-cache lawlist-as lawlist-archive lawlist-apropos
lawlist-+ lawlist-lcl byte-opt bytecomp byte-compile cl-extra cconv
lawlist-help disp-table easy-mmode edmacro kmacro quail help-mode
easymenu cl-loaddefs cl-lib pcase derived advice shell pcomplete
comint ansi-color ring savehist time-date mule-util tooltip eldoc
electric uniquify ediff-hook vc-hooks lisp-float-type mwheel ns-win
ucs-normalize term/common-win tool-bar dnd fontset image regexp-opt
fringe tabulated-list newcomment elisp-mode lisp-mode prog-mode
register page menu-bar rfn-eshadow timer select scroll-bar mouse
jit-lock font-lock syntax facemenu font-core frame cl-generic cham
georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao
korean japanese eucjp-ms cp51932 hebrew greek romanian slovak czech
european ethiopic indian cyrillic chinese charscript case-table
epa-hook jka-cmpr-hook help simple abbrev obarray minibuffer
cl-preloaded nadvice loaddefs button faces cus-face macroexp files
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote dbusbind kqueue cocoa
ns multi-tty make-network-process emacs)

Memory information:
((conses 16 2506629 211607)
 (symbols 48 86281 0)
 (miscs 40 1116 1187)
 (strings 32 198663 21468)
 (string-bytes 1 7537848)
 (vectors 16 61491)
 (vector-slots 8 1294267 27715)
 (floats 8 3369 803)
 (intervals 56 4477 414)
 (buffers 976 17))

bug#22404: 25.1.50; Forcing `window-scroll-functions` to run.

[Eli Zaretskii]

> Date: Mon, 18 Jan 2016 21:49:40 -0800
> From: Keith David Bershatsky <esq@lawlist.com>
> 
> As a feature request, please consider adding an alternative method to force the `window-scroll-functions` hook to run -- but NOT more than it would normally run. [In other words, it shouldn't run 3 times.]  I only found `(set-window-buffer (selected-window) (current-buffer))` as a viable means to achieve this goal.  `run-window-scroll-functions` runs immediately and is not a viable substitute for this feature request.  Some users may not wish to `set-window-buffer` or run the `window-configuration-change-hook` to achieve this feature request.
> 
> In my particular use case, I want to run a custom function only one time per command loop that is dependent upon the new `window-start` and `window-end`, which is only reliably available to the user after the `post-command-hook` has run its course.  My function is too costly time-wise to run more than once each command loop.  I have tried, to no avail, to come up with a test to ascertain with 100% accuracy (from the `post-command-hook`) to guess whether the `window-scroll-functions` hook will run and/or whether it will run more than one time (e.g., when cursor/point is partially visible).  I played around with `(pos-visible-in-window-p nil nil t)` and whether it returned a length of 0, 2 or 6 -- however, from the `post-command-hook`, that is insufficient to ascertain with 100% accuracy w
 hether the `window-scroll-functions` hook will run once or twice or not at all.  The solution, I believe, is to force the `window-scroll-functions` hook to run during every redisplay -- but not more than it would normally run.
> 
> In other words, I want the `window-scroll-functions` hook to run every command loop when my custom minor-mode is active -- or twice if point/cursor is only partially visible (at the bottom of the window) and needs to be moved up a smidgen by redisplay.
> 
> Background:  my custom function is a cross-hairs created with overlays that works with `word-wrap` -- the vertical line is XPM that matches the character and color underneath.

Can you please tell more details about the problem you have?  I;ve
read this description several times, and couldn't figure out why are
you having problems.

For example, why can't you run your function from the
post-command-hook directly?  If you want it to run after all the hook
functions finished, you can use the APPEND argument to add-hook,
right?

Or if the above somehow doesn't work, what about pre-command-hook?

And those are just 2 random thoughts that went through my head while I
was reading your request.

Thanks.

bug#22404: 25.1.50; Forcing `window-scroll-functions` to run.

[Keith David Bershatsky]

Thank you, Eli, for taking a look at this feature request.

The post-command-hook ("PCH") is only sufficient to return the correct window-start and window-end when no scrolling occurs.  If I attach a function to the PCH that places overlays, then I would need to know programmatically with 100% certainty whether the window-scroll-functions ("WSF") will be running one or more times subsequent to the PCH.  If the WSF will be running one or more times subsequent to the PCH, then I want to programmatically *prevent* overlays from being placed with the PCH -- i.e., so that they can instead be placed by the WSF.  If the WSF will run more than once, then I want to wait until the last time it runs to place my overlays -- i.e., because start/end have changed since the first time WSF ran.  E.g., point was partially visible and redisplay brings it into full vi
 ew.

I was not able to come up with a 100% reliable test from the PCH that tells me whether WSF will be running one or more times subsequent thereto.  So, I am using the following work-around to force the WSF to run all the time.  The problem remaining is that I still need a reliable test to determine whether the WSF will be running more than once, so that I can delay my overlay function until the last call.  The feature request is essentially:  trigger WSF (similar to set-window-buffer), but there is no need to set-window-buffer and there is no need to run the window-configuration-change-hook.  If there is any way you can think of to programmatically know with 100% certainty whether the WSF will be running more than one time, your help would be greatly appreciated.

(setq scroll-conservatively 101)

(defun pch-fn ()
  (let ((window-configuration-change-hook nil))
    (set-window-buffer (selected-window) (current-buffer) 'keep-margins)))

(add-hook 'post-command-hook 'pch-fn nil 'local)

(defun wsf-fn (win start)
  (message "win: %s | start: %s | end: %s" win start (window-end win t)))

(add-hook 'window-scroll-functions 'wsf-fn nil 'local)

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

At Tue, 19 Jan 2016 19:50:08 +0200,
Eli Zaretskii wrote:
> 
> * * *
> 
> Can you please tell more details about the problem you have?  I;ve
> read this description several times, and couldn't figure out why are
> you having problems.
> 
> For example, why can't you run your function from the
> post-command-hook directly?  If you want it to run after all the hook
> functions finished, you can use the APPEND argument to add-hook,
> right?
> 
> Or if the above somehow doesn't work, what about pre-command-hook?
> 
> And those are just 2 random thoughts that went through my head while I
> was reading your request.
> 
> Thanks.

bug#22404: 25.1.50; Forcing `window-scroll-functions` to run.

[Eli Zaretskii]

If you call set-window-start with its 3rd argument non-nil, doesn't
that force window-scroll-functions to be run?

bug#22404: 25.1.50; Forcing `window-scroll-functions` to run.

[John Wiegley]

>>>>> Keith David Bershatsky <esq@lawlist.com> writes:

> `run-window-scroll-functions` runs immediately and is not a viable
> substitute for this feature request.

Keith, why is the fact that it runs immediately a problem? The very name of
this function answers your feature request, but you don't explain why it is
insufficient, other than this comment.

How would run-window-scroll-functions need to be changed, since that is the
advertised way to... run the window-scroll-functions?

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

bug#22404: 25.1.50; Forcing `window-scroll-functions` to run.

[Keith David Bershatsky]

I too, John, got really excited when I saw `run-window-scroll-functions`.  Unfortunately, it doesn't return the correct window-start and window-end because it runs too soon.  It's not really a trigger that forces the WSF to run at its normal stages during redisplay.  If you run the following test in a lengthy buffer in a vertical split window pane with a *Messages* buffer in the other pane, then use the down arrow key to force window scrolling as the cursor moves, you'll see the numbers from the PCH are wrong and the numbers from WSF are correct.  The other problem with `run-window-scroll-functions` for this usage is that the WSF still runs when it is supposed to run, but there is an extra call to just the function attached to the WSF hook -- in other words, there might be as many as three
  (3) times the WSF function runs -- once with the wrong results off of the PCH, and once with the initial results from the WSF, and one final time when the WSF does its last call.  That last call on the WSF is the one I'm seeking.  Sometimes WSF only runs once when scrolling if point is fully visible.

(setq scroll-conservatively 101)

(defvar pch-test-var nil)

(defun pch-fn ()
  (let ((pch-test-var t))
  (run-window-scroll-functions (selected-window))))

(add-hook 'post-command-hook 'pch-fn nil 'local)

(defun wsf-fn (win start)
  (message "hook: %s | win: %s | start: %s | end: %s"
    (if pch-test-var "PCH" "WSF") win start (window-end win t)))

(add-hook 'window-scroll-functions 'wsf-fn nil 'local)

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

At Tue, 19 Jan 2016 10:53:32 -0800,
John Wiegley wrote:
> 
> >>>>> Keith David Bershatsky <esq@lawlist.com> writes:
> 
> > `run-window-scroll-functions` runs immediately and is not a viable
> > substitute for this feature request.
> 
> Keith, why is the fact that it runs immediately a problem? The very name of
> this function answers your feature request, but you don't explain why it is
> insufficient, other than this comment.
> 
> How would run-window-scroll-functions need to be changed, since that is the
> advertised way to... run the window-scroll-functions?
> 
> -- 
> John Wiegley

bug#22404: 25.1.50; Forcing `window-scroll-functions` to run.

[Keith David Bershatsky]

Explicitly calling `set-window-start` doesn't appear to trigger the WSF based on the following test -- e.g., moving around the window without scrolling doesn't yield any messages.

(setq scroll-conservatively 101)

(defvar pch-test-var nil)

(defun pch-fn ()
  (set-window-start (selected-window) (window-start (selected-window)) 'noforce))

(add-hook 'post-command-hook 'pch-fn nil 'local)

(defun wsf-fn (win start)
  (message "win: %s | start: %s | end: %s" win start (window-end win t)))

(add-hook 'window-scroll-functions 'wsf-fn nil 'local)

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

At Tue, 19 Jan 2016 21:39:53 +0200,
Eli Zaretskii wrote:
> 
> If you call set-window-start with its 3rd argument non-nil, doesn't
> that force window-scroll-functions to be run?

bug#22404: 25.1.50; Forcing `window-scroll-functions` to run.

[Eli Zaretskii]

> Date:  Tue, 19 Jan 2016 12:35:38 -0800
> From:  Keith David Bershatsky <esq@lawlist.com>
> Cc:  22404@debbugs.gnu.org
> 
> Explicitly calling `set-window-start` doesn't appear to trigger the WSF based on the following test -- e.g., moving around the window without scrolling doesn't yield any messages.
> 
> (setq scroll-conservatively 101)
> 
> (defvar pch-test-var nil)
> 
> (defun pch-fn ()
>   (set-window-start (selected-window) (window-start (selected-window)) 'noforce))

Sorry, I was mistaken: you need to call set-window-start with its 3rd
argument omitted or nil.  Does that solve your problem?

bug#22404: 25.1.50; Forcing `window-scroll-functions` to run.

[Keith David Bershatsky]

The following example uses `pos-visible-in-window-p` to guess as to whether the WSF will run more than one time -- i.e., if point is not fully visible, then WSF will run again.  [The last call to WSF is the most important so that the values of window-start and window-end are truly accurate.]  In addition to the WSF trigger feature request, a more sophisticated C-source solution to ascertain whether the WSF will run more than one time would be helpful.

(setq scroll-conservatively 101)

(defun pch-fn ()
  (let ((window-configuration-change-hook nil))
    (set-window-buffer (selected-window) (current-buffer) 'keep-margins)))

(add-hook 'post-command-hook 'pch-fn nil 'local)

(defun wsf-fn (win start)
  (message "point: %s | win: %s | start: %s | end: %s" 
    (if (pos-visible-in-window-p nil nil nil)
      "visible"
      "NOT visible")
    win start (window-end win t)))

(add-hook 'window-scroll-functions 'wsf-fn nil 'local)

bug#22404: 25.1.50; Forcing `window-scroll-functions` to run.

[Keith David Bershatsky]

[-- Attachment #1: Type: text/plain, Size: 3054 bytes --]

`set-window-start` (without the third option) would indeed trigger the WSF because it contains a key ingredient:  `w->force_start = true`.

Because the second argument (i.e., POS) cannot be known from the PCH when point has moved beyond the visible window, `set-window-start` is not a viable substitute for this feature request.

The attached patch is an example of an implementation of this feature request.  I created a new function called `force-wsf`, whose sole purpose is to trigger the WSF to run during redisplay -- after the PCH has already finished.  As noted in a previous e-mail, `run-window-scroll-functions` doesn't accomplish what `force-wsf` can achieve because the former runs the function attached to the WSF immediately -- instead of waiting until later on during redisplay when the correct values of `window-start` and `window-end` are ascertainable.

This is my first attempt at writing something like this, and I'm not sure exactly if everything is correct.  It does, however, appear to achieve the desired affect -- i.e., run the WSF at least once every command loop (even if no scrolling occurs) so we can always know the correct values for `window-start` and `window-end`.  It is no longer necessary to guess with `elisp` from the PCH regarding whether WSF will run.

I would, however, still like to come up with a test at the C-source code level that tells me whether the WSF will run more than one time -- because I only care about the last call on the WSF when the final values for `window-start` and `window-end` become available.

The usage for the new function `force-wsf` is as follows:

    (defun pch-fn ()
      (force-wsf (selected-window)))
    
    (add-hook 'post-command-hook 'pch-fn nil 'local)
    
    (defun wsf-fn (win start)
      (message "point: %s | win: %s | start: %s | end: %s" 
        ;; A better test at the C-source code level appears to be needed to ascertain whether
        ;; the WSF will run more than one time -- because we want the value for `window-start`
        ;; and `window-end win t` based on the LAST time WSF runs during the command loop.
        (if (pos-visible-in-window-p nil nil nil)
          "visible"
          "NOT visible")
        win start (window-end win t)))
    
    (add-hook 'window-scroll-functions 'wsf-fn nil 'local)

The new function in window.c looks like this:

    DEFUN ("force-wsf", Fforce_wsf, Sforce_wsf, 0, 1, 0,
           doc: /* Set force_start so that redisplay_window will run the
    window-scroll-functions.  */)
      (Lisp_Object window)
    {
      register struct window *w = decode_live_window (window);
        w->optional_new_start = true;
        return;
    }

And, the there is one additional line that may also be required further on down in window.d:

  defsubr (&Sforce_wsf);

Thanks,

Keith

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

At Wed, 20 Jan 2016 15:34:41 +0200,
Eli Zaretskii wrote:
> 
> > . . . you need to call set-window-start with its 3rd
> argument omitted or nil.  Does that solve your problem?


[-- Attachment #2: patch.diff --]
[-- Type: application/diff, Size: 1092 bytes --]

bug#22404: 25.1.50; Forcing `window-scroll-functions` to run.

[Eli Zaretskii]

> Date:  Wed, 20 Jan 2016 18:32:40 -0800
> From:  Keith David Bershatsky <esq@lawlist.com>
> Cc:  22404@debbugs.gnu.org
> 
> `set-window-start` (without the third option) would indeed trigger the WSF because it contains a key ingredient:  `w->force_start = true`.
> 
> Because the second argument (i.e., POS) cannot be known from the PCH when point has moved beyond the visible window, `set-window-start` is not a viable substitute for this feature request.

Then I'm afraid I must insist on understanding your needs better.  I
still don't, not even after re-reading your description several times.

Maybe you could describe your use case from a different perspective:
instead of telling how you tried to make sure window-scroll-functions
are run exactly once, perhaps try to describe the problem for which
you needed to invoke window-scroll-functions in the first place.
After all, window-scroll-functions is just the means towards some
specific goal.  And I don't think I understand the goal.

Once you described what you are trying to accomplish, it might be
easier to understand how you arrived at window-scroll-functions and
the need to run them when you do.

> The attached patch is an example of an implementation of this feature request.  I created a new function called `force-wsf`, whose sole purpose is to trigger the WSF to run during redisplay -- after the PCH has already finished.  As noted in a previous e-mail, `run-window-scroll-functions` doesn't accomplish what `force-wsf` can achieve because the former runs the function attached to the WSF immediately -- instead of waiting until later on during redisplay when the correct values of `window-start` and `window-end` are ascertainable.

I'd prefer to understand the need before we discuss the
implementation.

> I would, however, still like to come up with a test at the C-source code level that tells me whether the WSF will run more than one time

I think this is fundamentally impossible.  But I didn't yet take a
good enough look at the involved code, so maybe I'm missing something.

Thanks.

bug#22404: 25.1.50; Forcing `window-scroll-functions` to run.

[Keith David Bershatsky]

I have a minor mode that draws overlays from `window-start` to `window-end`, and those two values are mission critical to making sure that the overlays are placed correctly.  I have `word-wrap` set to `t`.  I have a function that makes relevant calculations in the visible window, which begins at `window-start` and uses `vertical-motion` for every visual line until `window-end`.  Along the way, I gather 22 different elements of a list for EACH visible line:  points at the beginning/ending of visual line (pbovl/peovl); points at beginning/ending of full line (bol/eol); column at the end of visual line (col-at-eovl); line number `(format-mode-line "%l")`; and all of the following in a vertical line (from top to bottom of the window) where the cursor is:  column, point, character, foreground/b
 ackground colors of character.

When the list is assembled, I then take that list and place overlays to create a horizontal and vertical line compatible with `word-wrap` -- the result is a visual cross-hairs spanning the entire length and width of the visible window.  I am using XPM images for the vertical line, which permits me to achieve a thin vertical line to the left of each character -- it looks just like a thin vbar cursor.  I control the color of the vertical line -- the letters remain the same color they were, but the line color varies -- yellow for even column number; red for odd column number; green for when I am beyond the fill-column on the first visual line; a medium blue for a line that is visually wrapped when point is on a visual line subsequent to the first wrapped line; and a cyan color when point is a
 t the right window boundary.  The horizontal line stretches the entire length of the window, even for the word-wrapped lines.  I have some fancy XPM images that change depending upon where point is.  It works with active regions too.  I have line numbering (in the left-hand margin) for each line.  I placer certain bitmap images in the right fringe depending when point is at the last character at the window edge, or when there is a hard return at the right window edge.  Each line-end (eol) has a pilcrow symbol that is XPM, except when cursor is at eol, in which case it is a different XPM image.  The end of the buffer has a different XPM image and the vertical line extends to the last visual line, even if the point at the end of the buffer is a few inches to the left of the vertical line.

The calculations (primarily because of `vertical-motion`) are too costly time-wise to perform more than once during each command loop because it slows down performance.

If I run the calculations from the `post-command-hook`, I will have wasted precious time IF the `window-scroll-functions` hook will be running.  IF the `window-scroll-functions` hook will be running, then the `window-start` and `window-end` obtained from the `post-command-hook` will be wrong -- so there is no point using those PCH wrong numbers.

If I run the calculations from the INITIAL run of the `window-scroll-functions` hook, I will have wasted precious time IF the WSF hook will be running a SECOND time.  [WSF runs a SECOND time if point was PARTIALLY visible.]  IF the WSF will be running more than one time, then the INITIAL values for `window-start` and `window-end` will be wrong -- so there is no point using those WSF wrong numbers.  Instead, I have to wait until the LAST call on the WSF when the final correct values for `window-start` and `window-end` can be truly ascertained.

Without the benefit of feature request 22404, there are a couple of options that are not ideal.

OPTION # 1(a):  Devise an efficient/accurate test that can be called from the `post-command-hook` to ascertain whether the `window-scroll-functions` hook will be running one or more times.  [Without a forced trigger, WSF only runs SOME of the time.]  If the WSF will be running one or more times, then do NOT perform the overlay calculations because the PCH values for `window-start` and `window-end` are wrong.  If the WSF will NOT be running, then go ahead and perform the time-costly calculations and place the overlays immediately.

OPTION # 1(b):  When the `window-scroll-functions` hook runs, I need to figure out whether it will be running one more time (subsequently) because point was only partially visible.  The only test I am aware of is `(pos-visible-in-window-p nil nil nil)`.  The LAST call on the WSF is the mission critical because that produces the correct values for `window-start` and `window-end`.  If WSF will only be running once, then go ahead and perform the time-costly calculations and place the overlays immediately.  If the WSF will be running more than once, then the *initial* WSF values for `window-start and `window-end` are wrong -- in that case, wait until the last call of WSF to perform the calculations and place overlays.

OPTION # 2:  Call `set-window-buffer` from the `post-command-hook` as a means of forcing WSF to run during the latter part of redisplay.  In this case, we never use the PCH values of `window-start` and `window-end` -- instead, we wait for WSF to give us the correct values.  This solution is not preferred because `set-window-buffer` was not designed to be used solely as a WSF trigger.  In the context of my example, I don't need to set the window buffer and I don't need to run the `window-configuration-change-hook` -- I just want to force WSF to run during the latter part of redisplay.

OPTION # 3:  Trigger the WSF without all the hoopla -- e.g., w->optional_new_start = true;.  E.g., my sample C-function `force_wsf`.

OPTION # 4 (ideal):  Create an entirely new animal that knows whether WSF will run more than once, and only produces the correct values for `window-start` and `window-end` in ALL circumstances, and then permits the user to run a custom function that takes advantage of those values.

Thanks,

Keith

bug#22404: 25.1.50; Forcing `window-scroll-functions` to run.

[Eli Zaretskii]

> Date:  Thu, 21 Jan 2016 11:54:30 -0800
> From:  Keith David Bershatsky <esq@lawlist.com>
> Cc:  22404@debbugs.gnu.org
> 
> The calculations (primarily because of `vertical-motion`) are too costly time-wise to perform more than once during each command loop because it slows down performance.
> 
> If I run the calculations from the `post-command-hook`, I will have wasted precious time IF the `window-scroll-functions` hook will be running.  IF the `window-scroll-functions` hook will be running, then the `window-start` and `window-end` obtained from the `post-command-hook` will be wrong -- so there is no point using those PCH wrong numbers.
> 
> If I run the calculations from the INITIAL run of the `window-scroll-functions` hook, I will have wasted precious time IF the WSF hook will be running a SECOND time.  [WSF runs a SECOND time if point was PARTIALLY visible.]  IF the WSF will be running more than one time, then the INITIAL values for `window-start` and `window-end` will be wrong -- so there is no point using those WSF wrong numbers.  Instead, I have to wait until the LAST call on the WSF when the final correct values for `window-start` and `window-end` can be truly ascertained.
> 
> Without the benefit of feature request 22404, there are a couple of options that are not ideal.

What do you need from window-scroll-functions?  Only the correct
values of window-start and window-end?  Or something else?

The other thing I still don't understand is how forcing
window-scroll-functions to run could fix your problem or _avoiding_
extra calculations.  Where and under what conditions would you call
the function that forces Emacs to run window-scroll-functions?
Doesn't that waste processing in those cases where normally
window-scroll-functions didn't need to be run (because window-start
and window-end don't change)?  IOW, aren't you running those costly
calculations from window-scroll-functions?

Finally, wouldn't running from pre-command-hook solve the problem?
The values of window-start and window-end are known by then, and you
can record the old values to compare them against new, to know when
they change.  No?

bug#22404: 25.1.50; Forcing `window-scroll-functions` to run.

[Michael Heerdegen]

Eli Zaretskii <eliz@gnu.org> writes:

> The other thing I still don't understand is how forcing
> window-scroll-functions to run could fix your problem or _avoiding_
> extra calculations.  Where and under what conditions would you call
> the function that forces Emacs to run window-scroll-functions?
> Doesn't that waste processing in those cases where normally
> window-scroll-functions didn't need to be run (because window-start
> and window-end don't change)?  IOW, aren't you running those costly
> calculations from window-scroll-functions?

FWIW, I ran into this class of problems for several times: after
scrolling had been happening, I want to do something with the newly
visible area of the buffer (e.g., place overlays).

The main problem here is that even when window-scroll-function run,
AFAIK the value of `window-end' can still be outdated, even when called
with the update argument.

I didn't need to use post-command-hook in my cases, so it's a bit
different from what Keith wants.

Anyway, using pre-command-hook is too late for us: it's not executed
unless the next input arrives, so you have to hit a key until
decorations or whatever get updated.

The only solution I know of is to do things in a timer, which works, but
not nicely (you sometimes get delays).


Regards,

Michael.

bug#22404: 25.1.50; Forcing `window-scroll-functions` to run.

[Eli Zaretskii]

> From: Michael Heerdegen <michael_heerdegen@web.de>
> Cc: Keith David Bershatsky <esq@lawlist.com>,  22404@debbugs.gnu.org
> Date: Fri, 29 Jan 2016 13:00:42 +0100
> 
> Anyway, using pre-command-hook is too late for us: it's not executed
> unless the next input arrives, so you have to hit a key until
> decorations or whatever get updated.

Then maybe we need a post-redisplay-hook.

bug#22404: 25.1.50; Forcing `window-scroll-functions` to run.

[Michael Heerdegen]

Eli Zaretskii <eliz@gnu.org> writes:

> Then maybe we need a post-redisplay-hook.

I would like that! - and I think it would solve Keith's problem as well.


Regards,

Michael.

bug#22404: 25.1.50; Forcing `window-scroll-functions` to run.

[Eli Zaretskii]

> From: Michael Heerdegen <michael_heerdegen@web.de>
> Cc: esq@lawlist.com,  22404@debbugs.gnu.org
> Date: Fri, 29 Jan 2016 15:57:57 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Then maybe we need a post-redisplay-hook.
> 
> I would like that! - and I think it would solve Keith's problem as well.

Patches welcome to that effect.

bug#22404: 25.1.50; Forcing `window-scroll-functions` to run.

[Keith David Bershatsky]

> What do you need from window-scroll-functions?  Only the correct
> values of window-start and window-end?  Or something else?


Correct, I just need the 100% accurate `window-start` and `window-end`.  :)  :)


> The other thing I still don't understand is how forcing
> window-scroll-functions to run could fix your problem or _avoiding_
> extra calculations.  Where and under what conditions would you call
> the function that forces Emacs to run window-scroll-functions?


I either need to use BOTH the `post-command-hook` and `window-scroll-functions` hook; OR, just the WSF if it is forced to always run.  WSF is needed (for sure) whenever `window-start` and `window-end` have changed -- so PCH must do the work the rest of the time.  I had previously been using a poor test to GUESS (from the PCH) whether WSF will run.


> Doesn't that waste processing in those cases where normally
> window-scroll-functions didn't need to be run (because window-start
> and window-end don't change)?  IOW, aren't you running those costly
> calculations from window-scroll-functions?


If point moves left or right, the vertical ruler that tracks the cursor position moves with the cursor -- so recalculation is needed.  The vertical ruler is on a 0.3 idle timer, and the horizontal ruler (with line numbers and pilcrows) is on all the time.  For the horizontal ruler, I have a test that compares the previous window-start/end to the new window-start/end -- if it is just cursor movement (instead of self-insert-command), then I use the previously recorded list of calculations.  Just moving the cursor left/right/up/down within the same window bounds is much faster than typing or deleting a character.

      ;; HORIZONTAL RULER (snippet)
      (setq posn-list
        (if
            (and
              hr-prev-start
              (= hr-prev-start start)
              hr-prev-end
              (= hr-prev-end end)
              +-posn-list
              (or
                (eq +-this-command 'lawlist-left-char)
                (eq +-this-command 'lawlist-right-char)
                (eq +-this-command 'lawlist-previous-line)
                (eq +-this-command 'lawlist-next-line)
                (eq +-this-command 'lawlist-forward-entity)
                (eq +-this-command 'lawlist-backward-entity)
                (eq +-this-command 'lawlist-forward-paragraph)
                (eq +-this-command 'lawlist-backward-paragraph)
                (eq +-this-command 'lawlist-end-of-visual-line)
                (eq +-this-command 'lawlist-beginning-of-visual-line)))
          +-posn-list
          (posn start end vcol)))

The vertical ruler (on an idle timer) will use the previously recorded list of calculations under a few limited circumstances.

      ;; VERTICAL RULER (snippet)
      (setq posn-list
        (if
            (or
              (not +-posn-list)
              (and
                (not force)
                (memq last-command '(
                  special-yank
                  special-copy-selected-region
                  copy-selected-region
                  delete-word-or-whitespace
                  delete-forward-char
                  lawlist-backward-delete-char-untabify
                  lawlist-beginning-of-buffer
                  lawlist-end-of-buffer
                  beginning-of-buffer
                  end-of-buffer
                  scroll-up
                  scroll-down
                  lawlist-super-scroll-up
                  lawlist-super-scroll-down
                  lawlist-scroll-up
                  lawlist-scroll-down
                  lawlist-left-char
                  lawlist-right-char
                  left-char
                  right-char
                  lawlist-previous-line
                  lawlist-next-line
                  previous-line
                  next-line
                  lawlist-forward-entity
                  lawlist-backward-entity
                  lawlist-forward-paragraph
                  lawlist-backward-paragraph
                  forward-paragraph
                  backward-paragraph
                  lawlist-end-of-visual-line
                  lawlist-beginning-of-visual-line
                  end-of-visual-line
                  beginning-of-visual-line))))
          (posn start end vcol)
          +-posn-list))

> Finally, wouldn't running from pre-command-hook solve the problem?
> The values of window-start and window-end are known by then, and you
> can record the old values to compare them against new, to know when
> they change.  No?

Basically any change to the text requires some type of recalculation -- e.g., line got shorter/longer, block of test was deleted/pasted.

Keith

bug#22404: 25.1.50; Forcing `window-scroll-functions` to run.

[John Wiegley]

>>>>> Keith David Bershatsky <esq@lawlist.com> writes:

>> What do you need from window-scroll-functions? Only the correct values of
>> window-start and window-end? Or something else?

> Correct, I just need the 100% accurate `window-start` and
> `window-end`. :) :)

Keith, your use case here does strike me as a bit exotic. I'm not saying it's
not a bug, but I'm wondering if your suggested solution is really the best
path to the final result you're reaching for.  The cost of `vertical-motion',
for example, seems to be a driving force behind your request.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

bug#22404: 25.1.50; Forcing `window-scroll-functions` to run.

[Keith David Bershatsky]

Thanks, John, for taking another look at #22404.

Let's say for example that someone has a function named `foo` which uses `window-start` and `window-end`, and the user wants that function to run every command loop.  Everyone will immediately think of the `post-command-hook` because the buffer or position may have been modified by `this-command` (whatever that may be).  The way Emacs is presently designed, people are needlessly calling `foo` multiple times when scrolling occurs because the `post-command-hook` can't provide correct numbers.  Most people probably don't care because `foo` is trivial in terms of the time it takes to run -- so if it runs 3 times per command loop, no problem.  I personally think calling `foo` 3 times per command loop, when it only needed to run once, is a poor design.

There is no 100% guaranteed test from the `post-command-hook` to know whether the `window-scroll-functions` hook will run, and if so, whether it will run 2 times (instead of just one).

Here is an excerpt from `window.h` that tells us what `optional_new_start` is currently used for:

    /* True means we have explicitly changed the value of start,
       but that the next redisplay is not obliged to use the new value.
       This is used in Fdelete_other_windows to force a call to
       Vwindow_scroll_functions; also by Frecenter with argument.  */
    bool_bf optional_new_start : 1;

It is also used for `set-window-buffer`; so perhaps that should be added to the doc-string.

By setting `optional_new_start` to `true`, we force the `window_scroll_functions` hook to run every command loop (during redisplay).  If that happens, then the `post-command-hook` is no longer needed to run `foo`.  So we have just reduced `foo` being called 2 to 3 times, to 1 to 2 times instead.  I have been studying `xdisp.c` and have concluded that it is extremely difficult to know whether the `window-scroll-functions` hook will run 1 or 2 times, because that hook can modify the buffer and there are some other hooks that may modify the buffer too -- thus requiring a second call to the WSF.

So, the moral of the story is whether it is a good thing to call `foo` 2 to 3 times per command loop, instead of 1 to 2 times per command loop.  And, it boils down to whether `foo` is costly.  It is costly for me because I use `vertical-motion`.  It may be costly to other users for a different reason.

Another idea would be to create a new animal that permits users to THROW_ANY_SWITCH from `elisp`.  It doesn't have to be `optional_new_start` specifically.  If users have an `elisp` mechanism to THROW_ANY_SWITCH, they can throw it from anywhere -- `pre-command-hook`, `post-command-hook`, etc.  [By switch, I mean the bool_bf items in `window.h`.]

Keith

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

At Thu, 28 Jan 2016 18:14:43 -0800,
John Wiegley wrote:
> 
> >>>>> Keith David Bershatsky <esq@lawlist.com> writes:
> 
> >> What do you need from window-scroll-functions? Only the correct values of
> >> window-start and window-end? Or something else?
> 
> > Correct, I just need the 100% accurate `window-start` and
> > `window-end`. :) :)
> 
> Keith, your use case here does strike me as a bit exotic. I'm not saying it's
> not a bug, but I'm wondering if your suggested solution is really the best
> path to the final result you're reaching for.  The cost of `vertical-motion',
> for example, seems to be a driving force behind your request.

bug#22404: 25.1.50; Forcing `window-scroll-functions` to run.

[Eli Zaretskii]

> Date:  Thu, 28 Jan 2016 19:08:22 -0800
> From:  Keith David Bershatsky <esq@lawlist.com>
> Cc:  22404@debbugs.gnu.org,Eli Zaretskii <eliz@gnu.org>
> 
> Let's say for example that someone has a function named `foo` which uses `window-start` and `window-end`, and the user wants that function to run every command loop.  Everyone will immediately think of the `post-command-hook` because the buffer or position may have been modified by `this-command` (whatever that may be).  The way Emacs is presently designed, people are needlessly calling `foo` multiple times when scrolling occurs because the `post-command-hook` can't provide correct numbers.

Personally, I think that having a function run by post-command-hook
that crucially depends on window-start and window-end is poor design.
Emacs never promised that these values be accurate before redisplay
runs, since redisplay sometimes needs to change where the window
starts and ends.

Solving this with window-scroll-functions is not the proper solution,
either, for the reasons already discussed at length in this thread.

I still don't understand well enough why pre-command-hook is not a
better solution here.  Until I do, I don't see how this issue can be
resolved better than it is today.  I don't like exposing APIs that
manipulate flags which are there for internal machinery.  Lisp
programs that set these flags arbitrarily can easily disrupt the
redisplay operation, so such APIs is a time bomb.

bug#22404: 25.1.50; Forcing `window-scroll-functions` to run.

[Keith David Bershatsky]

[-- Attachment #1: Type: text/plain, Size: 2204 bytes --]

In reading the comments in `xdisp.c`, there is a lot of mention about checking to see if the function that ran changed `window-start` and whether the cursor is still visible.  In my own tests over the past few days, I created something just like `post-redisplay-hook` that runs towards the end of redisplay.  I felt that my own feeble attempt to create the equivalent of the `post-redisplay-hook` had no checking built-in (due to lack knowledge / experience on my part).

So, I love the idea of a `post-redisplay-hook`, but think there probably needs to be some type of check to see whether the function that ran from it did some things that require certain recalculations -- e.g., maybe cursor was moved and is no longer partially or fully visible (there is even a setting I discovered where a user can decide that a partially visible cursor is just fine), maybe some text was inserted prior to window-start (which means the prior value is outdated), etc.

Sorry that I didn't understand your previous comments about the `pre-command-hook` potentially being a substitute for the contemplated `post-redisplay-hook`.  Michael is correct -- the goal is generally to have the screen looking perfectly once redisplay concludes and the `pre-command-hook` (after the fact) can't accomplish that goal.  When switching windows, the `pre-command-hook` will also have the wrong window in mind -- the correct window/buffer needs to be determined after the main command finishes.

Attached is an example of what I have been using for the past few days while I continue my ongoing tinkering -- it creates a new animal just like `window-scroll-functions` but with a different name.

Keith

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

At Fri, 29 Jan 2016 16:37:16 +0200,
Eli Zaretskii wrote:
> 
> > From: Michael Heerdegen <michael_heerdegen@web.de>
> > Cc: Keith David Bershatsky <esq@lawlist.com>,  22404@debbugs.gnu.org
> > Date: Fri, 29 Jan 2016 13:00:42 +0100
> > 
> > Anyway, using pre-command-hook is too late for us: it's not executed
> > unless the next input arrives, so you have to hit a key until
> > decorations or whatever get updated.
> 
> Then maybe we need a post-redisplay-hook.


[-- Attachment #2: wsf.diff --]
[-- Type: application/diff, Size: 8501 bytes --]

bug#22404: 25.1.50; Forcing `window-scroll-functions` to run.

[Keith David Bershatsky]

[-- Attachment #1: Type: text/plain, Size: 3119 bytes --]

I see that there are a few express circumstances in the function `window-end` where a forced update will not occur, even though a user expressly requested it with the optional argument being set to `t`.  Michael had mentioned an unspecified situation where `window-end` was incorrect from the `window-scroll-functions` hook even with a forced update, so perhaps one or more of those expressly coded exceptions is/are to blame.  I encountered a similar situation today in my testing with interactively calling `previous-line` at the top window edge (and scroll-conservatively 101) where the `window-end` was not updating correctly.

Attached is a diff-patch of the second draft for the new proposed animal that is somewhat similar to the `window-scroll-functions` hook, but this new animal is able to run every command loop even when there is no scrolling.  It automatically updates `window-end`, and it throws four (4) values that can be used by the user's custom function attached to the new hook:

* window-start
* window-end
* point at the beginning of the line of window-start.
* point at the end of the line of window-end.

I added two new symbols for the mode-line:  little `%w` for `window-start` and big `%W` for `window-end`.  This made debugging much easier, and I do a lot with window-start/end, so it comes in handy for writing other related functions.

I removed one double quote in comments in `window.c` that was breaking my font-lock highlighting -- an unmatched double quotes in a comment always causes havoc with my highlighting for the remainder of the buffer, so I try to fix those whenever I come across them.

I still haven't figured out how to entirely substitute throwing the switch `w->wsf_toggle = true` with just a buffer-local variable.  Ideally, I would prefer that the new hook run whenever local variable `wsf-var` is `t` -- without needing a switch.  At the present time, I am throwing the switch each command loop with the `post-command-hook`.  [The switch gets set back to `false` during redisplay, and is needed as sort of a counter so that a section of the redisplay code does not run more than necessary.]

This draft diff-patch can of course still use some polishing up -- e.g., the forced window update only needs to occur when `wsf-var` is `t`.  In future drafts, I'll probably change some of the names to further distinguish this from the built-in WSF.  [The built-in WSF section near a patched comment labeled "2 of 3" should probably now have an exception so that it doesn't run merely because this new animal is running, and I'll think some more about that in the coming days.]  I'll be using this patch in my daily routine to see how it works out.  Here is the sample usage, which is designed to be buffer-local:

(setq scroll-conservatively 101)

(setq wsf-var t)

(defun hr-pch-fn ()
  (force-wsf (selected-window)))

(defun hr-wsf-fn (win start end pbol-start peol-end)
  (message "win: %s | start: %s | end: %s | peol-start: %s | peol-end: %s"
    win start end pbol-start peol-end))

(add-hook 'post-command-hook 'hr-pch-fn t t)

(add-hook 'wsf-hook 'hr-wsf-fn nil t)


[-- Attachment #2: wsf.diff --]
[-- Type: application/diff, Size: 12566 bytes --]

bug#22404: 25.1.50; Forcing `window-scroll-functions` to run.

[Eli Zaretskii]

> Date:  Sun, 31 Jan 2016 19:50:05 -0800
> From:  Keith David Bershatsky <esq@lawlist.com>
> Cc:  22404@debbugs.gnu.org,Michael Heerdegen <eliz@gnu.org>
> 
> I see that there are a few express circumstances in the function `window-end` where a forced update will not occur, even though a user expressly requested it with the optional argument being set to `t`.  Michael had mentioned an unspecified situation where `window-end` was incorrect from the `window-scroll-functions` hook even with a forced update, so perhaps one or more of those expressly coded exceptions is/are to blame.  I encountered a similar situation today in my testing with interactively calling `previous-line` at the top window edge (and scroll-conservatively 101) where the `window-end` was not updating correctly.
> 
> Attached is a diff-patch of the second draft for the new proposed animal that is somewhat similar to the `window-scroll-functions` hook, but this new animal is able to run every command loop even when there is no scrolling.  It automatically updates `window-end`, and it throws four (4) values that can be used by the user's custom function attached to the new hook:
> 
> * window-start
> * window-end
> * point at the beginning of the line of window-start.
> * point at the end of the line of window-end.
> 
> I added two new symbols for the mode-line:  little `%w` for `window-start` and big `%W` for `window-end`.  This made debugging much easier, and I do a lot with window-start/end, so it comes in handy for writing other related functions.
> 
> I removed one double quote in comments in `window.c` that was breaking my font-lock highlighting -- an unmatched double quotes in a comment always causes havoc with my highlighting for the remainder of the buffer, so I try to fix those whenever I come across them.
> 
> I still haven't figured out how to entirely substitute throwing the switch `w->wsf_toggle = true` with just a buffer-local variable.  Ideally, I would prefer that the new hook run whenever local variable `wsf-var` is `t` -- without needing a switch.  At the present time, I am throwing the switch each command loop with the `post-command-hook`.  [The switch gets set back to `false` during redisplay, and is needed as sort of a counter so that a section of the redisplay code does not run more than necessary.]
> 
> This draft diff-patch can of course still use some polishing up -- e.g., the forced window update only needs to occur when `wsf-var` is `t`.  In future drafts, I'll probably change some of the names to further distinguish this from the built-in WSF.  [The built-in WSF section near a patched comment labeled "2 of 3" should probably now have an exception so that it doesn't run merely because this new animal is running, and I'll think some more about that in the coming days.]  I'll be using this patch in my daily routine to see how it works out.  Here is the sample usage, which is designed to be buffer-local:

Keith, I appreciate all the efforts you are putting into this, but I'd
rather we'd not go in that direction.  Running Lisp from within the
bowels of redisplay is inherently dangerous (because Lisp code can
signal an error, which then makes Emacs not responsive due to infinite
attempts to display the error message, which causes another error,
etc.).  Your patches add more places where we call Lisp, and in
particular in places, like redisplay_window, where it simply is out of
the question.

I'd prefer to have a post-redisplay-hook, which is called either at
the end of redisplay cycle, or after redisplay_internal returns to its
caller.  Would that solve your problem, and if so, what information do
you need to be passed to such a hook?

Thanks.

bug#22404: 25.1.50; Forcing `window-scroll-functions` to run.

[Keith David Bershatsky]

The four (4) values that I am most interested in are:

  1.  window-start

  2.  window-end

  3.  Point at the beginning of the line containing window-start.

  4.  Point at the end of the line containing window-end.

Although my proposed usage of the `post-redisplay-hook` will not be to alter buffer (i.e., I'm just adding overlays), other people who may use this new hook in the future may be modifying the buffer content -- e.g., adding/removing fonts, adding/removing text, moving the cursor, etc.  The `window-scroll-functions` hook is strategically placed at three locations of `xdisp.c`, and some of those areas are revisited when things have changed since the first pass -- e.g., continuation line at the beginning of the window, or cursor is partially visible at the bottom of the window.

If the `post-redisplay-hook` runs too late in time to receive any checks regarding cursor position and so forth, some users may be surprised when their cursor ends up below the visible screen at the conclusion of redisplay -- e.g., because they added text or increased a font size that pushed the cursor further down below.

Keith

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

At Mon, 01 Feb 2016 21:54:42 +0200,
Eli Zaretskii wrote:
> 
> * * *
> 
> I'd prefer to have a post-redisplay-hook, which is called either at
> the end of redisplay cycle, or after redisplay_internal returns to its
> caller.  Would that solve your problem, and if so, what information do
> you need to be passed to such a hook?

bug#22404: 25.1.50; Forcing `window-scroll-functions` to run.

[Eli Zaretskii]

> Date:  Mon, 01 Feb 2016 05:18:02 -0800
> From:  Keith David Bershatsky <esq@lawlist.com>
> Cc:  22404@debbugs.gnu.org
> 
> The four (4) values that I am most interested in are:
> 
>   1.  window-start
> 
>   2.  window-end
> 
>   3.  Point at the beginning of the line containing window-start.
> 
>   4.  Point at the end of the line containing window-end.

These are available from Lisp, so there seems to be no need to pass
them as arguments to the hook, right?

> Although my proposed usage of the `post-redisplay-hook` will not be to alter buffer (i.e., I'm just adding overlays), other people who may use this new hook in the future may be modifying the buffer content -- e.g., adding/removing fonts, adding/removing text, moving the cursor, etc.

If those users need this new hook, they can always invoke redisplay
explicitly at the end of their code, right?

> The `window-scroll-functions` hook is strategically placed at three locations of `xdisp.c`, and some of those areas are revisited when things have changed since the first pass -- e.g., continuation line at the beginning of the window, or cursor is partially visible at the bottom of the window.

But the fact they run in the middle of the redisplay cycle is exactly
the source of the problems we are trying to solve!

> If the `post-redisplay-hook` runs too late in time to receive any checks regarding cursor position and so forth, some users may be surprised when their cursor ends up below the visible screen at the conclusion of redisplay -- e.g., because they added text or increased a font size that pushed the cursor further down below.

Calling redisplay explicitly should resolve all those issues.

bug#22404: 25.1.50; Forcing `window-scroll-functions` to run.

[Keith David Bershatsky]

[-- Attachment #1: Type: text/plain, Size: 974 bytes --]

I completely understand and respect your decision to go in a different direction than my draft implementation of this new feature, but I thought it might be helpful for you to compare my ideas to your own proposed implementation as you go forward.  Because my proposed implementation mirrors the current implementation of the `window-scroll-functions` hook, `xdisp.c` performs all of the necessary checking to see whether the user's function attached to the `window-start-end-hook` changes the buffer in such a way that requires further work on the part of redisplay.  Attached is the revised edition of my new animal, whose proposed buffer-local usage is as follows:

(setq scroll-conservatively 101)

(setq window-start-end-var t)

(defun window-start-end-fn (win start end pbol-start peol-end)
 (message "win: %s | start: %s | end: %s | peol-start: %s | peol-end: %s"
   win start end pbol-start peol-end))

(add-hook 'window-start-end-hook 'window-start-end-fn nil t)



[-- Attachment #2: window_start_end.diff --]
[-- Type: application/diff, Size: 13270 bytes --]

bug#22404: 25.1.50; Forcing `window-scroll-functions` to run.

[Keith David Bershatsky]

A forced redisplay every command loop would pose two (2) major problems as far as I am aware.  The first issue will be that an unfinished display will be visible to the user for a split second before it gets finished with whatever code follows the first visible redisplay -- this will be especially visible when moving to an entirely different location of the buffer (e.g., full screen up or full screen down).  The second issue will be a significant slow-down in performance that will be mostly visible to the user when using rapid-fire keystrokes, or simply moving the cursor with the arrow keys by holding the arrow key in any of the four directions.  A forced redisplay is something that should be used sparingly, but not every command loop.

The four (4) values I seek *may* be available via Lisp depending upon where the `post-redisplay-hook` is placed.  I'd be happy to run some tests once you have decided upon a proposed location for the hook.

Keith

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

At Tue, 02 Feb 2016 18:34:58 +0200,
Eli Zaretskii wrote:
> 
> > Date:  Mon, 01 Feb 2016 05:18:02 -0800
> > From:  Keith David Bershatsky <esq@lawlist.com>
> > Cc:  22404@debbugs.gnu.org
> > 
> > The four (4) values that I am most interested in are:
> > 
> >   1.  window-start
> > 
> >   2.  window-end
> > 
> >   3.  Point at the beginning of the line containing window-start.
> > 
> >   4.  Point at the end of the line containing window-end.
> 
> These are available from Lisp, so there seems to be no need to pass
> them as arguments to the hook, right?
> 
> > Although my proposed usage of the `post-redisplay-hook` will not be to alter buffer (i.e., I'm just adding overlays), other people who may use this new hook in the future may be modifying the buffer content -- e.g., adding/removing fonts, adding/removing text, moving the cursor, etc.
> 
> If those users need this new hook, they can always invoke redisplay
> explicitly at the end of their code, right?
> 
> > The `window-scroll-functions` hook is strategically placed at three locations of `xdisp.c`, and some of those areas are revisited when things have changed since the first pass -- e.g., continuation line at the beginning of the window, or cursor is partially visible at the bottom of the window.
> 
> But the fact they run in the middle of the redisplay cycle is exactly
> the source of the problems we are trying to solve!
> 
> > If the `post-redisplay-hook` runs too late in time to receive any checks regarding cursor position and so forth, some users may be surprised when their cursor ends up below the visible screen at the conclusion of redisplay -- e.g., because they added text or increased a font size that pushed the cursor further down below.
> 
> Calling redisplay explicitly should resolve all those issues.

bug#22404: 25.1.50; Forcing `window-scroll-functions` to run.

[Eli Zaretskii]

> Date:  Tue, 02 Feb 2016 10:16:23 -0800
> From:  Keith David Bershatsky <esq@lawlist.com>
> Cc:  22404@debbugs.gnu.org
> 
> A forced redisplay every command loop would pose two (2) major problems as far as I am aware.  The first issue will be that an unfinished display will be visible to the user for a split second before it gets finished with whatever code follows the first visible redisplay

That depends where the hook will be called.  If it is called before
the changed contents is delivered to the glass, you won't see any such
adverse effects, exactly like you don't see them today when
window-scroll-functions are called after a large portion of redisplay
already happened.

> The second issue will be a significant slow-down in performance that will be mostly visible to the user when using rapid-fire keystrokes, or simply moving the cursor with the arrow keys by holding the arrow key in any of the four directions.  A forced redisplay is something that should be used sparingly, but not every command loop.

Emacs display engine is highly optimized: if there's nothing to do
when it is called the second time, it will detect that very quickly
and return without doing anything.  Specifically, if only cursor
motion happened, redisplay is very fast.

And the hook doesn't have to be dumb and call the second redisplay
every time.  It can be smarter, and only do so if it actually made
some changes, like moved an overlay or scrolled the window.  In the
latter case, the comparison with just moving cursor is no longer
valid.

So if such problems do exist, they will need to be reported with
specific use cases, because in general the current code already deals
with this.

> The four (4) values I seek *may* be available via Lisp depending upon where the `post-redisplay-hook` is placed.  I'd be happy to run some tests once you have decided upon a proposed location for the hook.

I don't understand the "maybe" part.  We are talking about values that
are easily obtained from any Lisp program, so what are the problems
you envision?

bug#22404: 25.1.50; Forcing `window-scroll-functions` to run.

[Keith David Bershatsky]

I will go through your most recent e-mail in a little while, but I wanted to get this test minor-mode over to you so that you can visually see exactly what I see when performing these tests.  It is a scaled-down example of my current usage -- this example just draws line numbers in the left margin of the visible window and uses `forward-line` instead of `vertical-motion`.  This minor-mode will work with your new `post-redisplay-hook` and it also works with the latest example `window_start_end.diff` that I e-mailed last night.  I have included an exception for `mhweel-scroll` so that we can use the mouse wheel to scroll up/down to see how the overlays have been placed.  If we use a large buffer for testing and go to `beginning-of-buffer` or `end-of-buffer` or scroll-up or scroll-down, the l
 ine numbers should be drawn by the time that redisplay finishes.  I have already taken the liberty of adding `ln-draw-numbers` to the `post-redisplay-hook` in anticipation of its future creation.

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

(defvar ln-before-string-list nil
"Doc-string -- `ln-before-string-list`.")
(make-variable-buffer-local 'ln-before-string-list)

(defvar ln-str-list nil
"Doc-string -- `ln-str-list`.")
(make-variable-buffer-local 'ln-str-list)

(defvar ln-this-command nil
"This local variable is set within the `post-command-hook`; and,
is also used by the `window-start-end-hook` hook.")
(make-variable-buffer-local 'ln-this-command)

(defvar ln-overlays nil "Overlays used in this buffer.")
(defvar ln-available nil "Overlays available for reuse.")
(mapc #'make-variable-buffer-local '(ln-overlays ln-available))

(defgroup ln nil
  "Show line numbers in the left margin."
  :group 'convenience)

(defface ln-active-face
  '((t (:background "black" :foreground "#eab700" :weight normal :italic nil
        :underline nil :box nil :overline nil)))
  "Face for `ln-active-face'."
  :group 'ln)

(defface ln-inactive-face
  '((t (:background "black" :foreground "SteelBlue" :weight normal :italic nil
        :underline nil :box nil :overline nil)))
  "Face for `ln-inactive-face'."
  :group 'ln)

(defvar ln-mode nil)

(defun ln-record-this-command ()
  (setq ln-this-command this-command))

(defun ln-draw-numbers (win &optional start end pbol-start peol-end force)
  "Update line numbers for the portion visible in window WIN."
  (message "win: %s | start: %s | end: %s | pbol-start: %s | peol-end: %s"
    win start end pbol-start peol-end)
  (when
      (and
        ln-mode
        (or ln-this-command force)
        (not (eq ln-this-command 'mwheel-scroll))
        (window-live-p win)
        (not (minibufferp))
        (pos-visible-in-window-p nil nil nil) )
    (setq ln-available ln-overlays)
    (setq ln-overlays nil)
    (setq ln-before-string-list nil)
    (setq ln-str-list nil)
    (let* (
        line
        my-initial-line
        (inhibit-point-motion-hooks t)
        (opoint (point))
        (ln-current-line-number (string-to-number (format-mode-line "%l")))
        (window-start (if start start (window-start win)))
        (window-end (if end end (window-end win t)))
        (max-digits-string (number-to-string (length (progn (goto-char (point-max)) (format-mode-line "%l")))))
        (width 0) )
      (goto-char window-start)
      (setq my-initial-line (string-to-number (format-mode-line "%l")))
      (setq line my-initial-line)
      (catch 'done
        (while t
          (when (= (point) (point-at-bol))
            (let* (
                (str
                  (propertize
                    (format (concat "%" max-digits-string "d") line)
                    'face (if (eq line ln-current-line-number) 'ln-active-face 'ln-inactive-face)))
                (ln-before-string
                  (propertize " " 'display `((margin left-margin) ,str)))
                (visited
                  (catch 'visited
                    (dolist (o (overlays-in (point) (point)))
                      (when (equal-including-properties (overlay-get o 'ln-str) str)
                        (unless (memq o ln-overlays)
                          (push o ln-overlays))
                        (setq ln-available (delq o ln-available))
                        (throw 'visited t))))) )
              (push ln-before-string ln-before-string-list)
              (push str ln-str-list)
              (unless visited
                (let ((ov (if (null ln-available)
                            (make-overlay (point) (point))
                          (move-overlay (pop ln-available) (point) (point)))))
                  (push ov ln-overlays)
                  (overlay-put ov 'before-string ln-before-string)
                  (overlay-put ov 'ln-str str)))
              (setq width (max width (length str)))))
            (if (and (not (eobp)) (< (point) window-end))
                (progn
                  (forward-line)
                  (setq line (1+ line)))
              (throw 'done nil))))
      (set-window-margins win width (cdr (window-margins win)))
      (mapc #'delete-overlay ln-available)
      (setq ln-available nil)
      (setq ln-this-command nil)
      (goto-char opoint))))

(defsubst lawlist-remove-overlays (beg end name val)
"Remove the overlays that are `equal-including-properties`.
Includes a unique situation when an overlay with an `'after-string` property
is at the very end of a narrowed-buffer."
  (let* (
      (point-max (point-max))
      (point-min (point-min))
      (narrowed-p (buffer-narrowed-p))
      (beg (if beg beg point-min))
      (end
        (cond
          ((and
              (not narrowed-p)
              end)
            end)
          ((and
              (not narrowed-p)
              (null end))
            point-max)
          ((and
              narrowed-p
              end
              (< end point-max))
            end)
          ((and
              narrowed-p
              end
              (= end point-max))
            (1+ end))
          ((and
              narrowed-p
              (null end))
            (1+ point-max)) ))
      (overlays
        (progn
          (overlay-recenter end)
          (overlays-in beg end))) )
    (when (and beg end name val)
      (dolist (o overlays)
        (cond
          ((and
                (eq name 'face)
                (eq (overlay-get o name) val))
            (if (< (overlay-start o) beg)
                (if (> (overlay-end o) end)
              (progn
                (move-overlay (copy-overlay o)
                  (overlay-start o) beg)
                (move-overlay o end (overlay-end o)))
                  (move-overlay o (overlay-start o) beg))
              (if (> (overlay-end o) end)
                  (move-overlay o end (overlay-end o))
                (delete-overlay o))))
          ((and
                (not (eq name 'face))
                (equal-including-properties (overlay-get o name) val))
            (delete-overlay o)))))))

(define-minor-mode ln-mode
  "A minor-mode for line-numbers in the left-hand margin."
  :init-value nil
  :lighter " #"
  :keymap nil
  :global nil
  :group 'ln
  (cond
    (ln-mode
      (setq window-start-end-var t)
      (add-hook 'pre-command-hook 'ln-record-this-command nil t)
      (add-hook 'window-start-end-hook 'ln-draw-numbers nil t)
      (add-hook 'post-redisplay-hook 'ln-draw-numbers nil t)
      (ln-draw-numbers (selected-window) nil nil nil nil 'force)
      (when (called-interactively-p 'any)
        (message "Turned ON `ln-mode`.")))
    (t
      (remove-hook 'pre-command-hook 'ln-record-this-command t)
      (remove-hook 'window-start-end-hook 'ln-draw-numbers t)
      (remove-hook 'post-redisplay-hook 'ln-draw-numbers t)
      (kill-local-variable 'ln-overlays)
      (kill-local-variable 'ln-available)
      (dolist (val ln-str-list)
        (lawlist-remove-overlays nil nil 'ln-str val))
      (kill-local-variable 'ln-str-list)
      (dolist (val ln-before-string-list)
        (lawlist-remove-overlays nil nil 'before-string val))
      (kill-local-variable 'ln-before-string-list)
      (kill-local-variable 'window-start-end-var)
      (dolist (w (get-buffer-window-list (current-buffer) nil t))
        (set-window-margins w 0 (cdr (window-margins w))))
      (when (called-interactively-p 'any)
        (message "Turned OFF `ln-mode`.")))))

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

bug#22404: 25.1.50; Forcing `window-scroll-functions` to run.

[Keith David Bershatsky]

Thank you, Eli, for the detailed explanation in your most recent e-mail -- greatly appreciated.

Assuming the correct values are available through Lisp when using the `post-redisplay-hook` and assuming that there is no buffer modification (attached to that new hook) that moves point or changes font sizes, I am unaware of any other prospective problem areas.  The minor-mode in my previous e-mail should be sufficient for us to visually see how it works with some basic things like scroll-up/down, beginning/end-of-buffer, next/previous-line when at the window edges.

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

At Tue, 02 Feb 2016 20:43:13 +0200,
Eli Zaretskii wrote:
> 
> 
> I don't understand the "maybe" part.  We are talking about values that
> are easily obtained from any Lisp program, so what are the problems
> you envision?

bug#22404: 25.1.50; Forcing `window-scroll-functions` to run.

[Keith David Bershatsky]

[-- Attachment #1: Type: text/plain, Size: 1370 bytes --]

Attached is the next in the series of draft concept ideas, which ensures the point is fully visible before firing.  Some people will naturally prefer that point be partially visible when the hook is triggered, but I like fully visible instead.

I noticed that redisplay handles a large yank/paste by erroneously letting the user see for a split second that point is partially visible on the bottom of the screen (when `scroll-conservatively` is 101), and then redisplay fixes the display by moving the lines up far enough to bring point back into view.  That behavior appears to be a bug -- i.e., Emacs should internally know that point is partially visible at the bottom of the screen following a large yank and then patiently wait to display the finished product until point is fully visible.  This is most noticeable in my own minor-mode that can take up to .040 seconds to perform its calculations with `vertical-motion'.

The following is the proposed usage based on the attached concept draft patch/diff file:


(defun window-start-end-hook-fn (win start end pbol-start peol-end fully-p)
  (message "win: %s | start: %s | end: %s | pbol-start: %s | peol-end: %s | fully-p: %s"
    win start end pbol-start peol-end fully-p))

(setq window-start-end-hook-var t)

(setq scroll-conservatively 101)

(add-hook 'window-start-end-hook 'window-start-end-hook-fn nil t)



[-- Attachment #2: window_start_end_hook.diff --]
[-- Type: application/diff, Size: 10910 bytes --]

bug#22404: 25.1.50; Forcing `window-scroll-functions` to run.

[Eli Zaretskii]

> Date:  Mon, 08 Feb 2016 00:51:44 -0800
> From:  Keith David Bershatsky <esq@lawlist.com>
> Cc:  22404@debbugs.gnu.org
> 
> I noticed that redisplay handles a large yank/paste by erroneously letting the user see for a split second that point is partially visible on the bottom of the screen (when `scroll-conservatively` is 101), and then redisplay fixes the display by moving the lines up far enough to bring point back into view.  That behavior appears to be a bug -- i.e., Emacs should internally know that point is partially visible at the bottom of the screen following a large yank and then patiently wait to display the finished product until point is fully visible.  This is most noticeable in my own minor-mode that can take up to .040 seconds to perform its calculations with `vertical-motion'.

A reproducible recipe will be appreciated (in a separate bug report,
please).  I'm guessing that some redisplay optimization causes the
slightly inaccurate display, which is thereafter fixed.  If so, it
could be that the conditions for that optimization should be fixed.

> +        CALLN (Frun_hook_with_args, Qwindow_start_end_hook, window,
> +            make_number (startp_integer),
> +            make_number (endp),
> +            make_number (pbol_startp),
> +            make_number (peol_endp),
> +            fully_p ? Qt : Qnil);

When calling Lisp from redisplay, always use safe_call, to be
protected against errors signaled by that Lisp.

> +        w->window_start_end_hook_pending = false;

For the same reason, reset the flag _before_ calling Lisp, not after,
so that if Lisp does signal an error, the flag will be reset, and
won't cause the hook to be called ad nauseam.

bug#22404: 25.1.50; Forcing `window-scroll-functions` to run.

[Keith David Bershatsky]

[-- Attachment #1: Type: text/plain, Size: 1532 bytes --]

Thank you, Eli, for the helpful suggestions mentioned in your last e-mail.  I've incorporated them into the latest draft that is attached, and I've simplified some of the code, and I've added a test to avoid unnecessarily triggering of the `window-scroll-functions` hook.

The issue mentioned in my last email was caused by `font-lock-mode` -- when I turn off `font-lock-mode`, there are a few situations where point remains partially visible at the bottom of the window and redisplay does not catch it -- e.g., when the line that point is on is about 90 percent visible at the bottom of the window.  The test in the `run_window_start_end_hook` realizes that point is partially visible, so the function attached to the `window-start-end-hook` does not fire because point was never fully visible during redisplay.  I'll do some more testing over the next few days to find out exactly where in redisplay the test for partially visible is failing such that redisplay never moves the display up one line to bring it into full view.  And, I'll submit a separate ticket once I've
  tracked it down or once I've come up with a reproducible test.

The proposed usage of this new hook has been simplified to the following:


(defun window-start-end-hook-fn (win start end pbol-start peol-end fully-p)
  (message "win: %s | start: %s | end: %s | pbol-start: %s | peol-end: %s | fully-p: %s"
    win start end pbol-start peol-end fully-p))

(setq scroll-conservatively 101)

(add-hook 'window-start-end-hook 'window-start-end-hook-fn nil t)



[-- Attachment #2: window_start_end_hook.diff --]
[-- Type: application/diff, Size: 12558 bytes --]

bug#22404: 25.1.50; Forcing `window-scroll-functions` to run.

[Eli Zaretskii]

> Date:  Tue, 09 Feb 2016 08:00:33 -0800
> From:  Keith David Bershatsky <esq@lawlist.com>
> Cc:  22404@debbugs.gnu.org
> 
> The issue mentioned in my last email was caused by `font-lock-mode` -- when I turn off `font-lock-mode`, there are a few situations where point remains partially visible at the bottom of the window and redisplay does not catch it -- e.g., when the line that point is on is about 90 percent visible at the bottom of the window.

Did you define faces that change more than just the colors, like use a
different font or bold/italic typefaces?  If so, what you see might
not be a bug: the original display is before JIT font-lock kicks in
and changes the dimensions of the characters on display, so they are
partially visible only after fontifications did their job.

bug#22404: 25.1.50; Forcing `window-scroll-functions` to run.

[Keith David Bershatsky]

Question, please:  Is it a necessary evil that the `timer-idle-list` must always trigger a redisplay?

I discovered today that the mere presence of anything attached to the `timer-idle-list` triggers a redisplay when the clock strikes midnight -- e.g., the following example triggers a redisplay after 10 seconds of inactivity:

    (run-with-idle-timer 10 'repeat 'ignore)

I am working on tracking down the issue of why redisplay doesn't realize that my cursor is below the bottom of the window or partially visible following certain large yanks or custom paragraph-down functions.  `font-lock-mode` was affecting my testing because I had 0.5 second idle timer for `jit-lock-context-fontify`.  For example, my point stayed out of view until the 0.5 second buzzer went off, at which time it magically came back into view.  The same thing will happen if I just have `'ignore` on an idle timer.

I do not yet have enough information to submit a meaningful bug report regarding the cursor staying below the bottom of the screen, but I feel I'm making some progress now that I tracked down the magical appearance of my cursor due to the `timer-idle-list`.

Thanks,

Keith

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

At Tue, 09 Feb 2016 19:48:56 +0200,
Eli Zaretskii wrote:
> 
> > Date:  Tue, 09 Feb 2016 08:00:33 -0800
> > From:  Keith David Bershatsky <esq@lawlist.com>
> > Cc:  22404@debbugs.gnu.org
> > 
> > The issue mentioned in my last email was caused by `font-lock-mode` -- when I turn off `font-lock-mode`, there are a few situations where point remains partially visible at the bottom of the window and redisplay does not catch it -- e.g., when the line that point is on is about 90 percent visible at the bottom of the window.
> 
> Did you define faces that change more than just the colors, like use a
> different font or bold/italic typefaces?  If so, what you see might
> not be a bug: the original display is before JIT font-lock kicks in
> and changes the dimensions of the characters on display, so they are
> partially visible only after fontifications did their job.

bug#22404: 25.1.50; Forcing `window-scroll-functions` to run.

[Eli Zaretskii]

> Date:  Thu, 11 Feb 2016 16:14:44 -0800
> From:  Keith David Bershatsky <esq@lawlist.com>
> Cc:  22404@debbugs.gnu.org
> 
> Question, please:  Is it a necessary evil that the `timer-idle-list` must always trigger a redisplay?

I don't think discussing this belongs to this bug report, so I will
answer on emacs-devel instead.

bug#22404: 25.1.50; Forcing `window-scroll-functions` to run.

[Keith David Bershatsky]

[-- Attachment #1: Type: text/plain, Size: 202 bytes --]

Here is the updated draft of the `window_scroll_functions_hook.diff`.  A copy of this same patch was attached to 22637 today, but I thought it should be included here also with the 22404.

Keith



[-- Attachment #2: window_start_end_hook.diff --]
[-- Type: application/octet-stream, Size: 13754 bytes --]

diff --git a/src/keyboard.c b/src/keyboard.c
index 3431cd8..3d8d54f 100644
--- a/src/keyboard.c
+++ b/src/keyboard.c
@@ -1233,6 +1233,15 @@ static int read_key_sequence (Lisp_Object *, int, Lisp_Object,
                               bool, bool, bool, bool);
 static void adjust_point_for_property (ptrdiff_t, bool);
 
+static void
+set_window_start_end_hook (void)
+{
+  Lisp_Object window = (selected_window);
+  struct window *w = decode_live_window (window);
+  w->window_start_end_hook_force = true;
+  w->window_start_end_hook_pending = true;
+}
+
 Lisp_Object
 command_loop_1 (void)
 {
@@ -1258,6 +1267,8 @@ command_loop_1 (void)
       if (!NILP (Vpost_command_hook) && !NILP (Vrun_hooks))
 	safe_run_hooks (Qpost_command_hook);
 
+      set_window_start_end_hook ();
+
       /* If displaying a message, resize the echo area window to fit
 	 that message's size exactly.  */
       if (!NILP (echo_area_buffer[0]))
@@ -1474,6 +1485,8 @@ command_loop_1 (void)
 
       safe_run_hooks (Qpost_command_hook);
 
+      set_window_start_end_hook ();
+
       /* If displaying a message, resize the echo area window to fit
 	 that message's size exactly.  */
       if (!NILP (echo_area_buffer[0]))
diff --git a/src/lisp.h b/src/lisp.h
index 8aa2861..0962306 100644
--- a/src/lisp.h
+++ b/src/lisp.h
@@ -3913,6 +3913,7 @@ extern void init_eval_once (void);
 extern Lisp_Object safe_call (ptrdiff_t, Lisp_Object, ...);
 extern Lisp_Object safe_call1 (Lisp_Object, Lisp_Object);
 extern Lisp_Object safe_call2 (Lisp_Object, Lisp_Object, Lisp_Object);
+extern Lisp_Object safe_call7 (Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object, Lisp_Object);
 extern void init_eval (void);
 extern void syms_of_eval (void);
 extern void unwind_body (Lisp_Object);
diff --git a/src/window.h b/src/window.h
index c29207d..3156927 100644
--- a/src/window.h
+++ b/src/window.h
@@ -348,6 +348,12 @@ struct window
        Vwindow_scroll_functions; also by Frecenter with argument.  */
     bool_bf optional_new_start : 1;
 
+    /* True means force a call to Vwindow_start_end_hook.  */
+    bool_bf window_start_end_hook_force : 1;
+
+    /* True means the hook has not yet run with point fully visible.  */
+    bool_bf window_start_end_hook_pending : 1;
+
     /* True means the cursor is currently displayed.  This can be
        set to zero by functions overpainting the cursor image.  */
     bool_bf phys_cursor_on_p : 1;
diff --git a/src/xdisp.c b/src/xdisp.c
index fed5879..d3e102d 100644
--- a/src/xdisp.c
+++ b/src/xdisp.c
@@ -2618,6 +2618,13 @@ safe_call2 (Lisp_Object fn, Lisp_Object arg1, Lisp_Object arg2)
   return safe_call (3, fn, arg1, arg2);
 }
 
+/*  This is for the `window_start_end_hook'.  Call function FN with seven arguments.
+    Return the result or nil if something went wrong.  */
+Lisp_Object
+safe_call7 (Lisp_Object fn, Lisp_Object arg1, Lisp_Object arg2, Lisp_Object arg3, Lisp_Object arg4, Lisp_Object arg5, Lisp_Object arg6, Lisp_Object arg7)
+{
+  return safe_call (8, fn, arg1, arg2, arg3, arg4, arg5, arg6, arg7);
+}
 
 \f
 /***********************************************************************
@@ -13417,6 +13424,68 @@ do { if (! polling_stopped_here) stop_polling ();	\
 do { if (polling_stopped_here) start_polling ();	\
        polling_stopped_here = false; } while (false)
 
+static void
+run_window_start_end_hook (struct text_pos startp, struct window *w, Lisp_Object window, struct it it, char *string)
+{
+  void *itdata = NULL;
+  bool fully_p = false;
+  EMACS_INT posint = PT;
+  int x, y, rtop, rbot, rowh, vpos, startp_integer, endp, pbol_startp, peol_endp;
+  startp_integer = CHARPOS (startp);
+  struct buffer *buf;
+  buf = XBUFFER (w->contents);
+  if (!NILP (Flocal_variable_p (Qwindow_start_end_hook, Fwindow_buffer (window)))
+      && w->window_start_end_hook_pending)
+    {
+      if ((posint >= CHARPOS (startp) && posint <= BUF_ZV (buf))
+          && CHARPOS (startp) >= BUF_BEGV (buf)
+          && CHARPOS (startp) <= BUF_ZV (buf)
+          && pos_visible_p (w, posint, &x, &y, &rtop, &rbot, &rowh, &vpos))
+        fully_p = !rtop && !rbot;
+      if (fully_p)
+        {
+          CLIP_TEXT_POS_FROM_MARKER (startp, w->start);
+          itdata = bidi_shelve_cache ();
+          start_display (&it, w, startp);
+          move_it_vertically (&it, window_box_height (w));
+          if (it.current_y < it.last_visible_y)
+            move_it_past_eol (&it);
+          endp = (IT_CHARPOS (it));
+          bidi_unshelve_cache (itdata, false);
+          SET_PT (startp_integer);
+          pbol_startp = XINT (Fline_beginning_position (Qnil));
+          SET_PT (endp);
+          peol_endp = XINT (Fline_end_position (Qnil));
+          SET_PT (posint);
+#ifdef GLYPH_DEBUG
+          /* See the doc-string for `pos-visible-in-window-p'.  */
+          debug_method_add (w, "%s -- x: %d | y: %d | startp: %d | endp: %d | pbol_startp: %d | peol_endp: %d",
+                               string, x, y, startp_integer, endp, pbol_startp, peol_endp);
+#endif
+          /*  Reset the flag _before_ calling Lisp, not after, so that
+              if Lisp does signal an error, the flag will be reset, and
+              won't cause the hook to be called ad nauseam.  */
+          w->window_start_end_hook_pending = false;
+          /*  When calling Lisp from redisplay, always use safe_call, to be
+              protected against errors signaled by that Lisp.  */
+          safe_call7 (Qrun_hook_with_args, Qwindow_start_end_hook,
+              window,
+              make_number (startp_integer),
+              make_number (endp),
+              make_number (pbol_startp),
+              make_number (peol_endp),
+              fully_p ? Qt : Qnil);
+        }
+        else
+          {
+#ifdef GLYPH_DEBUG
+            /* See the doc-string for `pos-visible-in-window-p'.  */
+            debug_method_add (w, "%s -- x: %d | y: %d | rtop: %d | rbot %d | rowh: %d | vpos: %d",
+                                 string, x, y, rtop, rbot, rowh, vpos);
+#endif
+          }
+    }
+}
 
 /* Perhaps in the future avoid recentering windows if it
    is not necessary; currently that causes some problems.  */
@@ -13714,6 +13783,7 @@ redisplay_internal (void)
       && match_p
       && !w->force_start
       && !w->optional_new_start
+      && !w->window_start_end_hook_force
       /* Point must be on the line that we have info recorded about.  */
       && PT >= CHARPOS (tlbufpos)
       && PT <= Z - CHARPOS (tlendpos)
@@ -15418,6 +15488,8 @@ try_scrolling (Lisp_Object window, bool just_this_one_p,
   /* Run window scroll functions.  */
   startp = run_window_scroll_functions (window, startp);
 
+  run_window_start_end_hook (startp, w, window, it, "try_scrolling");
+
   /* Display the window.  Give up if new fonts are loaded, or if point
      doesn't appear.  */
   if (!try_window (window, startp, 0))
@@ -16082,6 +16154,7 @@ redisplay_window (Lisp_Object window, bool just_this_one_p)
   bool last_line_misfit = false;
   ptrdiff_t beg_unchanged, end_unchanged;
   int frame_line_height;
+  bool window_start_end_hook_suppress_wsf = false;
 
   SET_TEXT_POS (lpoint, PT, PT_BYTE);
   opoint = lpoint;
@@ -16272,13 +16345,16 @@ redisplay_window (Lisp_Object window, bool just_this_one_p)
 
   /* If someone specified a new starting point but did not insist,
      check whether it can be used.  */
-  if ((w->optional_new_start || window_frozen_p (w))
+  if ((w->optional_new_start || w->window_start_end_hook_force || window_frozen_p (w))
       && CHARPOS (startp) >= BEGV
       && CHARPOS (startp) <= ZV)
     {
+
+      if (w->window_start_end_hook_force && !w->optional_new_start && !window_frozen_p (w) && !w->force_start)
+        window_start_end_hook_suppress_wsf = true;
+
       ptrdiff_t it_charpos;
 
-      w->optional_new_start = false;
       start_display (&it, w, startp);
       move_it_to (&it, PT, 0, it.last_visible_y, -1,
 		  MOVE_TO_POS | MOVE_TO_X | MOVE_TO_Y);
@@ -16299,14 +16375,28 @@ redisplay_window (Lisp_Object window, bool just_this_one_p)
 	    w->force_start = true;
 #ifdef GLYPH_DEBUG
 	  if (w->force_start)
-	    {
-	      if (window_frozen_p (w))
-		debug_method_add (w, "set force_start from frozen window start");
-	      else
-		debug_method_add (w, "set force_start from optional_new_start");
-	    }
+    {
+      if (window_frozen_p (w))
+        {
+        debug_method_add (w, "set force_start from frozen window start");
+      }
+      else if (w->optional_new_start)
+        {
+        debug_method_add (w, "set force_start from optional_new_start");
+      }
+      else if (w->window_start_end_hook_force)
+        {
+        debug_method_add (w, "set force_start from window_start_end_hook_force");
+      }
+      else
+        {
+        debug_method_add (w, "This situation is not yet contemplated.");
+      }
+    }
 #endif
 	}
+      w->optional_new_start = false;
+      w->window_start_end_hook_force = false;
     }
 
  force_start:
@@ -16334,11 +16424,18 @@ redisplay_window (Lisp_Object window, bool just_this_one_p)
 	 than to get into an infinite loop calling the hook functions
 	 and having them get more errors.  */
       if (!update_mode_line
-	  || ! NILP (Vwindow_scroll_functions))
+	  || ! NILP (Vwindow_scroll_functions)
+	  || ! NILP (Vwindow_start_end_hook))
 	{
 	  update_mode_line = true;
 	  w->update_mode_line = true;
-	  startp = run_window_scroll_functions (window, startp);
+
+  /* Run window scroll functions.  */
+  if (!window_start_end_hook_suppress_wsf)
+    startp = run_window_scroll_functions (window, startp);
+
+  run_window_start_end_hook (startp, w, window, it, "redisplay_window (force_start)");
+
 	}
 
       if (CHARPOS (startp) < BEGV)
@@ -16606,6 +16703,7 @@ redisplay_window (Lisp_Object window, bool just_this_one_p)
 	     because a window scroll function can have changed the
 	     buffer.  */
 	  || !NILP (Vwindow_scroll_functions)
+	  || !NILP (Vwindow_start_end_hook)
 	  || MINI_WINDOW_P (w)
 	  || !(used_current_matrix_p
 	       = try_window_reusing_current_matrix (w)))
@@ -16803,6 +16901,8 @@ redisplay_window (Lisp_Object window, bool just_this_one_p)
   /* Run scroll hooks.  */
   startp = run_window_scroll_functions (window, it.current.pos);
 
+  run_window_start_end_hook (startp, w, window, it, "redisplay_window (recenter)");
+
   /* Redisplay the window.  */
   bool use_desired_matrix = false;
   if (!current_matrix_up_to_date_p
@@ -16811,6 +16911,7 @@ redisplay_window (Lisp_Object window, bool just_this_one_p)
       /* Don't use try_window_reusing_current_matrix in this case
 	 because it can have changed the buffer.  */
       || !NILP (Vwindow_scroll_functions)
+      || !NILP (Vwindow_start_end_hook)
       || !just_this_one_p
       || MINI_WINDOW_P (w)
       || !(used_current_matrix_p
@@ -16931,7 +17032,17 @@ redisplay_window (Lisp_Object window, bool just_this_one_p)
 	  clear_glyph_matrix (w->desired_matrix);
 	  if (1 == try_window (window, it.current.pos,
 			       TRY_WINDOW_CHECK_MARGINS))
-	    goto done;
+    {
+      if (!NILP (Flocal_variable_p (Qwindow_start_end_hook, Fwindow_buffer (window)))
+          && w->window_start_end_hook_pending)
+        {
+          run_window_start_end_hook (it.current.pos, w, window, it, "redisplay_window (post-recenter)");
+          clear_glyph_matrix (w->desired_matrix);
+          try_window (window, it.current.pos, 0);
+        }
+      goto done;
+    }
+
 	}
 
       /* If centering point failed to make the whole line visible,
@@ -23686,6 +23797,20 @@ decode_mode_spec (struct window *w, register int c, int field_width,
 	  return "@";
       }
 
+    case 'w':
+      {
+  ptrdiff_t window_start = marker_position (w->start);
+  pint2str (decode_mode_spec_buf, width, window_start);
+  return decode_mode_spec_buf;
+      }
+
+    case 'W':
+      {
+  ptrdiff_t window_end = BUF_Z (b) - w->window_end_pos;
+  pint2str (decode_mode_spec_buf, width, window_end);
+  return decode_mode_spec_buf;
+      }
+
     case 'z':
       /* coding-system (not including end-of-line format) */
     case 'Z':
@@ -31151,6 +31276,7 @@ They are still logged to the *Messages* buffer.  */);
   DEFSYM (Qoverriding_terminal_local_map, "overriding-terminal-local-map");
   DEFSYM (Qoverriding_local_map, "overriding-local-map");
   DEFSYM (Qwindow_scroll_functions, "window-scroll-functions");
+  DEFSYM (Qwindow_start_end_hook, "window-start-end-hook");
   DEFSYM (Qwindow_text_change_functions, "window-text-change-functions");
   DEFSYM (Qredisplay_end_trigger_functions, "redisplay-end-trigger-functions");
   DEFSYM (Qinhibit_point_motion_hooks, "inhibit-point-motion-hooks");
@@ -31472,6 +31598,13 @@ is scrolled.  It is not designed for that, and such use probably won't
 work.  */);
   Vwindow_scroll_functions = Qnil;
 
+  DEFVAR_LISP ("window-start-end-hook", Vwindow_start_end_hook,
+    doc: /* A bufer-local hook used to obtain new `window-start` and `window-end`
+during redisplay.  The function attached to this hook has the following arguments:
+WINDOW, WINDOW-START, WINDOW-END, PBOL-START, PEOL-END, FULLY-P.  */);
+  Vwindow_start_end_hook = Qnil;
+  Fmake_variable_buffer_local (Qwindow_start_end_hook);
+
   DEFVAR_LISP ("window-text-change-functions",
 	       Vwindow_text_change_functions,
     doc: /* Functions to call in redisplay when text in the window might change.  */);

Re: Why do idle timers trigger redisplay?

[Eli Zaretskii]

> Date:  Thu, 11 Feb 2016 16:14:44 -0800
> From:  Keith David Bershatsky <esq@lawlist.com>
> Cc:  22404@debbugs.gnu.org
> 
> Question, please:  Is it a necessary evil that the `timer-idle-list` must always trigger a redisplay?

Emacs does another redisplay cycle each time there was some kind of
input.  The logic of this is that any input could potentially affect
what is displayed (the job of figuring out what, if anything, does
need to be redrawn is integral part of what redisplay does, and it
might as well conclude that no redisplay is actually needed).

Code run by timers is one such kind of input, since any code run by
timers runs with the purpose of doing something with Emacs data:
buffers, strings, windows, frames, etc.

Other kinds of input include: keyboard input, input from subprocesses,
mouse gestures, events that come from sources like D-bus and file
notifications, etc.  They all cause a cycle of redisplay.

bug#22404: 25.1.50; Forcing `window-scroll-functions` to run.

[Keith David Bershatsky]

[-- Attachment #1: Type: text/plain, Size: 190 bytes --]

Here is the latest draft of the window-start-end-hook.  This is probably about as far as I need to go, unless I discover bugs in the future.  It appears to be working well.

Thanks.

Keith


[-- Attachment #2: window_start_end_hook.diff --]
[-- Type: application/diff, Size: 14323 bytes --]

bug#22404: 25.1.50; Forcing `window-scroll-functions` to run.

[Keith David Bershatsky]

[-- Attachment #1: Type: text/plain, Size: 734 bytes --]

Attached is the next draft of the `window-start-end-hook`, which applies to the master branch as of commit "ea626c72e590aa7a45fd26df42240854e4225cef" on March 10, 2015.

The primary addition is a new location for the hook to be called when scrolling with the mouse-wheel.  A couple of pointers were added to store window/start-end values.  Miscellaneous unrelated new/experimental features have been removed from the patch.  The usage is as follows:


(defun window-start-end-hook-fn (win start end pbol-start peol-end fully-p)
  (message "win: %s | start: %s | end: %s | pbol-start: %s | peol-end: %s | fully-p: %s"
    win start end pbol-start peol-end fully-p))

(add-hook 'window-start-end-hook 'window-start-end-hook-fn nil t)



[-- Attachment #2: window_start_end_hook.diff --]
[-- Type: application/diff, Size: 15202 bytes --]

Re: Why do idle timers trigger redisplay?

[Keith David Bershatsky]

Thank you, Eli, for helping me to better understand why code run by timers is considered input that triggers redisplay (which in turn figures out what to do).

As always, your tutoring is greatly appreciated.

Keith

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

At Fri, 12 Feb 2016 10:25:51 +0200,
Eli Zaretskii wrote:
> 
> > Date:  Thu, 11 Feb 2016 16:14:44 -0800
> > From:  Keith David Bershatsky <esq@lawlist.com>
> > 
> > Question, please:  Is it a necessary evil that the `timer-idle-list` must always trigger a redisplay?
> 
> Emacs does another redisplay cycle each time there was some kind of
> input.  The logic of this is that any input could potentially affect
> what is displayed (the job of figuring out what, if anything, does
> need to be redrawn is integral part of what redisplay does, and it
> might as well conclude that no redisplay is actually needed).
> 
> Code run by timers is one such kind of input, since any code run by
> timers runs with the purpose of doing something with Emacs data:
> buffers, strings, windows, frames, etc.
> 
> Other kinds of input include: keyboard input, input from subprocesses,
> mouse gestures, events that come from sources like D-bus and file
> notifications, etc.  They all cause a cycle of redisplay.

Re: Why do idle timers trigger redisplay?

[Stefan Monnier]

> Thank you, Eli, for helping me to better understand why code run by timers
> is considered input that triggers redisplay (which in turn figures out what
> to do).

Presumably idle timers have effects (their return value is ignored, so
if they have no effect they're useless).  So they may change buffers or
elements that affect the mode-line.  So a redisplay cycle needs to be
triggered after running them.


        Stefan