Can we make set_point_both less expensive?

Can we make set_point_both less expensive?

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 683 bytes --]

Profiling some code I'd written recently, I found that goto-char was
using 90% of CPU time in a large buffer with lots of properties and
overlays. Setting inhibit-point-motion-hooks helped, but it occurs to me
that the operation shouldn't have been expensive in the first place.
Nothing in the buffer had a point motion hook or an intangible property,
though, so the work goto-char was doing was pointless.

In addition to allowing callers to bind inhibit-point-motion-hooks, what
about tracking whether any property or overlay ever inserted into a
buffer uses one of these uncommon facilities? This way, the common case
is fast even without inhibit-point-motion-hooks.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Can we make set_point_both less expensive?

[Stefan Monnier]

> overlays. Setting inhibit-point-motion-hooks helped, but it occurs to me
> that the operation shouldn't have been expensive in the first place.

goto-char should never trigger any hook.  Properties such as
`intangible' are misfeatures, IMO.

> In addition to allowing callers to bind inhibit-point-motion-hooks, what
> about tracking whether any property or overlay ever inserted into a
> buffer uses one of these uncommon facilities? This way, the common case
> is fast even without inhibit-point-motion-hooks.

I think we should make those goto-char hooks conditional on a variable,
and make this variable default to nil.  That's an incompatible change,
of course, but it would solve this issue.  And the var's name should
make it clear it's only present to support some legacy feature that's
being phased out.


        Stefan

Re: Can we make set_point_both less expensive?

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 1017 bytes --]

On 03/15/2015 08:02 PM, Stefan Monnier wrote:
>> overlays. Setting inhibit-point-motion-hooks helped, but it occurs to me
>> that the operation shouldn't have been expensive in the first place.
> 
> goto-char should never trigger any hook.  Properties such as
> `intangible' are misfeatures, IMO.
> 
>> In addition to allowing callers to bind inhibit-point-motion-hooks, what
>> about tracking whether any property or overlay ever inserted into a
>> buffer uses one of these uncommon facilities? This way, the common case
>> is fast even without inhibit-point-motion-hooks.
> 
> I think we should make those goto-char hooks conditional on a variable,
> and make this variable default to nil.  That's an incompatible change,
> of course, but it would solve this issue.  And the var's name should
> make it clear it's only present to support some legacy feature that's
> being phased out.

So you propose effectively making inhibit-point-motion-hooks default to
t? Why not just do precisely that?


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Can we make set_point_both less expensive?

[Stefan Monnier]

> So you propose effectively making inhibit-point-motion-hooks default to
> t? Why not just do precisely that?

Because IIRC inhibit-point-motion-hooks also inhibits cursor-motion
hooks, whereas these should still be fully supported (even more so if
we don't support point-motion hooks any more).


        Stefan

Re: Can we make set_point_both less expensive?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Mon, 16 Mar 2015 08:49:02 -0400
> Cc: Emacs developers <emacs-devel@gnu.org>
> 
> > So you propose effectively making inhibit-point-motion-hooks default to
> > t? Why not just do precisely that?
> 
> Because IIRC inhibit-point-motion-hooks also inhibits cursor-motion
> hooks

Now I'm confused: there's no cursor-motion in Emacs, only
point-motion.  But you already know that, so I don't understand what
are you alluding to here.

Re: Can we make set_point_both less expensive?

[Stefan Monnier]

>> > So you propose effectively making inhibit-point-motion-hooks default to
>> > t? Why not just do precisely that?
>> Because IIRC inhibit-point-motion-hooks also inhibits cursor-motion
>> hooks

Looks like I was mistaken: inhibit-point-motion-hooks only applies to
point-motion and not to cursor-motion (which has instead variables like
disable-point-adjustment and global-disable-point-adjustment).

So, yes, Daniel, I guess I'm just suggesting to make
inhibit-point-motion-hooks default to nil.


        Stefan

Re: Can we make set_point_both less expensive?

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 700 bytes --]

On Mar 16, 2015 4:24 PM, "Eli Zaretskii" <eliz@gnu.org> wrote:
>
> > From: Stefan Monnier <monnier@iro.umontreal.ca>
> > Date: Mon, 16 Mar 2015 08:49:02 -0400
> > Cc: Emacs developers <emacs-devel@gnu.org>
> >
> > > So you propose effectively making inhibit-point-motion-hooks default
to
> > > t? Why not just do precisely that?
> >
> > Because IIRC inhibit-point-motion-hooks also inhibits cursor-motion
> > hooks
>
> Now I'm confused: there's no cursor-motion in Emacs, only
> point-motion.  But you already know that, so I don't understand what
> are you alluding to here.

IIUC, the argument is that these hooks should happen between commands, not
in the middle of elisp code at every goto-char.

[-- Attachment #2: Type: text/html, Size: 1042 bytes --]

Re: Can we make set_point_both less expensive?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Date: Sun, 15 Mar 2015 23:02:49 -0400
> Cc: Emacs developers <emacs-devel@gnu.org>
> 
> > overlays. Setting inhibit-point-motion-hooks helped, but it occurs to me
> > that the operation shouldn't have been expensive in the first place.
> 
> goto-char should never trigger any hook.  Properties such as
> `intangible' are misfeatures, IMO.

Does that include 'point-entered' and 'point-exited' as well?  If so,
can you explain why they are misfeatures?

Re: Can we make set_point_both less expensive?

[Stefan Monnier]

>> goto-char should never trigger any hook.  Properties such as
>> `intangible' are misfeatures, IMO.
> Does that include 'point-entered' and 'point-exited' as well?  If so,
> can you explain why they are misfeatures?

For the same reason `intangible' is a misfeature: it works at the level
of `point' where coders don't expect such weird behavior.  But what
users of those features normally want is to catch movement of the cursor
rather than movement of `point'.


        Stefan

Re: Can we make set_point_both less expensive?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: dancol@dancol.org,  emacs-devel@gnu.org
> Date: Mon, 16 Mar 2015 08:47:47 -0400
> 
> what users of those features normally want is to catch movement of
> the cursor

You mean, we should do this in redisplay?  Wouldn't that be a radical
change in semantics?  In particular, if redisplay doesn't run, neither
will the hooks.

Re: Can we make set_point_both less expensive?

[Stefan Monnier]

>> what users of those features normally want is to catch movement of
>> the cursor
> You mean, we should do this in redisplay?

Probably in pre-redisplay-hook or in pre/post-command-hook, yes.

> Wouldn't that be a radical change in semantics?

Yes.  Which is why we have to phase out the old properties and introduce
new ones instead.  E.g. `cursor-enter' and `cursor-leave'.


        Stefan

Re: Can we make set_point_both less expensive?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Cc: dancol@dancol.org, emacs-devel@gnu.org
> Date: Mon, 16 Mar 2015 13:36:40 -0400
> 
> >> what users of those features normally want is to catch movement of
> >> the cursor
> > You mean, we should do this in redisplay?
> 
> Probably in pre-redisplay-hook or in pre/post-command-hook, yes.

That contradicts the "catch movement of cursor" idea: redisplay could
well move point from where it is found before redisplay, as you know.

Re: Can we make set_point_both less expensive?

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 868 bytes --]

On 03/16/2015 11:05 AM, Eli Zaretskii wrote:
>> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
>> Cc: dancol@dancol.org, emacs-devel@gnu.org
>> Date: Mon, 16 Mar 2015 13:36:40 -0400
>>
>>>> what users of those features normally want is to catch movement of
>>>> the cursor
>>> You mean, we should do this in redisplay?
>>
>> Probably in pre-redisplay-hook or in pre/post-command-hook, yes.
> 
> That contradicts the "catch movement of cursor" idea: redisplay could
> well move point from where it is found before redisplay, as you know.

When does that matter? The intent is to get editing to behave _as if_
invisible regions were intangible, and the existing invisible motion
behavior seems mostly up to the task.

(There's a problem where point remains inside invisible text if text
around point is made newly invisible without moving point.)


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Can we make set_point_both less expensive?

[Eli Zaretskii]

> Date: Mon, 16 Mar 2015 11:18:58 -0700
> From: Daniel Colascione <dancol@dancol.org>
> CC: emacs-devel@gnu.org
> 
> On 03/16/2015 11:05 AM, Eli Zaretskii wrote:
> >> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> >> Cc: dancol@dancol.org, emacs-devel@gnu.org
> >> Date: Mon, 16 Mar 2015 13:36:40 -0400
> >>
> >>>> what users of those features normally want is to catch movement of
> >>>> the cursor
> >>> You mean, we should do this in redisplay?
> >>
> >> Probably in pre-redisplay-hook or in pre/post-command-hook, yes.
> > 
> > That contradicts the "catch movement of cursor" idea: redisplay could
> > well move point from where it is found before redisplay, as you know.
> 
> When does that matter? The intent is to get editing to behave _as if_
> invisible regions were intangible, and the existing invisible motion
> behavior seems mostly up to the task.

I wasn't talking about intangible, I was talking about the
point-entered and point-exited properties.

Stefan argues that these should actually be cursor-entered/exited,
i.e. we should run the hooks when we know that point has the value
that will be used to set the cursor there.  But doing that before
redisplay doesn't guarantee that, since redisplay sometimes moves
point to bring it into view.  So in that case, the hooks might run
when they shouldn't have, or vice versa.

Re: Can we make set_point_both less expensive?

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 1877 bytes --]

On 03/16/2015 11:27 AM, Eli Zaretskii wrote:
>> Date: Mon, 16 Mar 2015 11:18:58 -0700
>> From: Daniel Colascione <dancol@dancol.org>
>> CC: emacs-devel@gnu.org
>>
>> On 03/16/2015 11:05 AM, Eli Zaretskii wrote:
>>>> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
>>>> Cc: dancol@dancol.org, emacs-devel@gnu.org
>>>> Date: Mon, 16 Mar 2015 13:36:40 -0400
>>>>
>>>>>> what users of those features normally want is to catch movement of
>>>>>> the cursor
>>>>> You mean, we should do this in redisplay?
>>>>
>>>> Probably in pre-redisplay-hook or in pre/post-command-hook, yes.
>>>
>>> That contradicts the "catch movement of cursor" idea: redisplay could
>>> well move point from where it is found before redisplay, as you know.
>>
>> When does that matter? The intent is to get editing to behave _as if_
>> invisible regions were intangible, and the existing invisible motion
>> behavior seems mostly up to the task.
> 
> I wasn't talking about intangible, I was talking about the
> point-entered and point-exited properties.
> 
> Stefan argues that these should actually be cursor-entered/exited,
> i.e. we should run the hooks when we know that point has the value
> that will be used to set the cursor there.  But doing that before
> redisplay doesn't guarantee that, since redisplay sometimes moves
> point to bring it into view.  So in that case, the hooks might run
> when they shouldn't have, or vice versa.

Sure, but intangible is the most useful facility we're talking about
disabling by default. Aren't point-entered and point-exited even less
useful? Looking through the source, ERC uses these hooks to echo
timestamps, which it could do just as well with a post-command-hook.
Gnus does something similar to update its toolbar. table also uses these
hooks to refresh its menu bar.

Are there any other important use cases?


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Can we make set_point_both less expensive?

[Eli Zaretskii]

> Date: Mon, 16 Mar 2015 11:31:57 -0700
> From: Daniel Colascione <dancol@dancol.org>
> CC: monnier@IRO.UMontreal.CA, emacs-devel@gnu.org
> 
> > I wasn't talking about intangible, I was talking about the
> > point-entered and point-exited properties.
> > 
> > Stefan argues that these should actually be cursor-entered/exited,
> > i.e. we should run the hooks when we know that point has the value
> > that will be used to set the cursor there.  But doing that before
> > redisplay doesn't guarantee that, since redisplay sometimes moves
> > point to bring it into view.  So in that case, the hooks might run
> > when they shouldn't have, or vice versa.
> 
> Sure, but intangible is the most useful facility we're talking about
> disabling by default. Aren't point-entered and point-exited even less
> useful? Looking through the source, ERC uses these hooks to echo
> timestamps, which it could do just as well with a post-command-hook.
> Gnus does something similar to update its toolbar. table also uses these
> hooks to refresh its menu bar.

My understanding was that Stefan wants these hooks replaced with
different ones, not throw them away.  So I was talking about those
different hooks.

Re: Can we make set_point_both less expensive?

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 1648 bytes --]

On 03/16/2015 12:42 PM, Eli Zaretskii wrote:
>> Date: Mon, 16 Mar 2015 11:31:57 -0700
>> From: Daniel Colascione <dancol@dancol.org>
>> CC: monnier@IRO.UMontreal.CA, emacs-devel@gnu.org
>>
>>> I wasn't talking about intangible, I was talking about the
>>> point-entered and point-exited properties.
>>>
>>> Stefan argues that these should actually be cursor-entered/exited,
>>> i.e. we should run the hooks when we know that point has the value
>>> that will be used to set the cursor there.  But doing that before
>>> redisplay doesn't guarantee that, since redisplay sometimes moves
>>> point to bring it into view.  So in that case, the hooks might run
>>> when they shouldn't have, or vice versa.
>>
>> Sure, but intangible is the most useful facility we're talking about
>> disabling by default. Aren't point-entered and point-exited even less
>> useful? Looking through the source, ERC uses these hooks to echo
>> timestamps, which it could do just as well with a post-command-hook.
>> Gnus does something similar to update its toolbar. table also uses these
>> hooks to refresh its menu bar.
> 
> My understanding was that Stefan wants these hooks replaced with
> different ones, not throw them away.  So I was talking about those
> different hooks.

I don't see a good use case for either the old or the new hook. Instead
of making point motion updates "edge triggered" (i.e., run a function
when point enters or exits a certain region), modes should be using a
"level triggered" approach, where they inspect the current location of
point and update whatever state they need based on that current location.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Can we make set_point_both less expensive?

[Lennart Borgman]

On Mon, Mar 16, 2015 at 9:43 PM, Daniel Colascione <dancol@dancol.org> wrote:
>
>
> modes should be using a
> "level triggered" approach, where they inspect the current location of
> point and update whatever state they need based on that current location.

Don't they need the previous location sometimes too? (Entering
direction in invisible regions, for example.)

Re: Can we make set_point_both less expensive?

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 620 bytes --]

On 03/16/2015 01:48 PM, Lennart Borgman wrote:
> On Mon, Mar 16, 2015 at 9:43 PM, Daniel Colascione <dancol@dancol.org> wrote:
>>
>>
>> modes should be using a
>> "level triggered" approach, where they inspect the current location of
>> point and update whatever state they need based on that current location.
> 
> Don't they need the previous location sometimes too? (Entering
> direction in invisible regions, for example.)

We already have code to adjust point out of invisible regions. The
command loop takes care of that. AFAICT, the actual existing uses of
point motion hooks don't require state.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Can we make set_point_both less expensive?

[Stefan Monnier]

> We already have code to adjust point out of invisible regions.
> The command loop takes care of that.

Indeed, it remembers the previous position in a global variable.


        Stefan

Re: Can we make set_point_both less expensive?

[Eli Zaretskii]

> Date: Mon, 16 Mar 2015 13:43:57 -0700
> From: Daniel Colascione <dancol@dancol.org>
> CC: monnier@IRO.UMontreal.CA, emacs-devel@gnu.org
> 
> > My understanding was that Stefan wants these hooks replaced with
> > different ones, not throw them away.  So I was talking about those
> > different hooks.
> 
> I don't see a good use case for either the old or the new hook. Instead
> of making point motion updates "edge triggered" (i.e., run a function
> when point enters or exits a certain region), modes should be using a
> "level triggered" approach, where they inspect the current location of
> point and update whatever state they need based on that current location.

There are a lot of features in Emacs that might make little sense to
me or you or someone else.  Removing features just because of that, or
because we think they don't have use cases that are valid in our eyes,
is not a practice I'd recommend.  At least, we never did that in
Emacs, AFAIR.  Features might be deprecated because they cause real
damage to Emacs development, but even then we do that very cautiously
and slowly.

My $0.02, anyway.

RE: Can we make set_point_both less expensive?

[Drew Adams]

> There are a lot of features in Emacs that might make little sense to
> me or you or someone else.  Removing features just because of that, or
> because we think they don't have use cases that are valid in our eyes,
> is not a practice I'd recommend.  At least, we never did that in
> Emacs, AFAIR.  Features might be deprecated because they cause real
> damage to Emacs development, but even then we do that very cautiously
> and slowly.
> 
> My $0.02, anyway.

Please add my 2c as well.

(Independent of any consideration of this particular feature.)

Re: Can we make set_point_both less expensive?

[Stefan Monnier]

>> >> what users of those features normally want is to catch movement of
>> >> the cursor
>> > You mean, we should do this in redisplay?
>> Probably in pre-redisplay-hook or in pre/post-command-hook, yes.
> That contradicts the "catch movement of cursor" idea: redisplay could
> well move point from where it is found before redisplay, as you know.

Indeed, that's a bug that we need to fix: after moving point, redisplay should
re-run pre-redisplay-hook; same thing after scrolling.


        Stefan

Re: Can we make set_point_both less expensive?

[martin rudalics]

 > Indeed, that's a bug that we need to fix: after moving point, redisplay should
 > re-run pre-redisplay-hook; same thing after scrolling.

Couldn't pre-redisplay-hook move `point' off the window?  Then redisplay
would have to move `point' back to the window.

martin

Re: Can we make set_point_both less expensive?

[Stefan Monnier]

>> Indeed, that's a bug that we need to fix: after moving point,
>> redisplay should re-run pre-redisplay-hook; same thing
>> after scrolling.
> Couldn't pre-redisplay-hook move `point' off the window?

Yes.  Or it could delete the buffer and all my files.

> Then redisplay would have to move `point' back to the window.

Indeed.


        Stefan

Re: Can we make set_point_both less expensive?

[martin rudalics]

 >> Couldn't pre-redisplay-hook move `point' off the window?
 >
 > Yes.  Or it could delete the buffer and all my files.

Hmm... moving `point' off the window could be a side-effect of
fontifying.

 >> Then redisplay would have to move `point' back to the window.
 >
 > Indeed.

And would run pre-redisplay-hook again.

martin

Re: Can we make set_point_both less expensive?

[Stefan Monnier]

>>> Couldn't pre-redisplay-hook move `point' off the window?
>> Yes.  Or it could delete the buffer and all my files.
> Hmm... moving `point' off the window could be a side-effect of
> fontifying.

>>> Then redisplay would have to move `point' back to the window.
>> Indeed.
> And would run pre-redisplay-hook again.

That's right.


        Stefan

Re: Can we make set_point_both less expensive?

[martin rudalics]

 >>>> Then redisplay would have to move `point' back to the window.
 >>> Indeed.
 >> And would run pre-redisplay-hook again.
 >
 > That's right.

Then why would we OT1H care about removing a function from
`post-command-hook' whenever it causes an error and OTOH not care about
running into an endless loop with pre-redisplay-hook?

martin

Re: Can we make set_point_both less expensive?

[Stefan Monnier]

>>>>> Then redisplay would have to move `point' back to the window.
>>>> Indeed.
>>> And would run pre-redisplay-hook again.
>> That's right.
> Then why would we OT1H care about removing a function from
> `post-command-hook' whenever it causes an error and OTOH not care about
> running into an endless loop with pre-redisplay-hook?

We don't have to have an endless loop.  We can decide to stop the loop
at any point.   Since such inf-loop would be an error in any case it's
perfectly OK to end up with an "incorrect" display.

Currently we decide to stop the loop before it can even start, so we end
up with an incorrect display even if there would not be any such
inf-looping.


        Stefan

Re: Can we make set_point_both less expensive?

[martin rudalics]

 > We don't have to have an endless loop.  We can decide to stop the loop
 > at any point.   Since such inf-loop would be an error in any case it's
 > perfectly OK to end up with an "incorrect" display.

The loop wouldn't be "an error in any case".  It would be the result of
an interaction of the function on the hook and the display engine.

 > Currently we decide to stop the loop before it can even start, so we end
 > up with an incorrect display even if there would not be any such
 > inf-looping.

I'd be among the first to appreciate a hook that is run once after the
display engine has established new window-start/-end/-point positions.
But having the display engine potentially call that hook again and again
would scare me.

martin

Re: Can we make set_point_both less expensive?

[Stefan Monnier]

>> We don't have to have an endless loop.  We can decide to stop the loop
>> at any point.   Since such inf-loop would be an error in any case it's
>> perfectly OK to end up with an "incorrect" display.
> The loop wouldn't be "an error in any case".  It would be the result of
> an interaction of the function on the hook and the display engine.

Right, but such an interaction is undesirable, so it would reflect
a misbehavior, i.e. an error.

>> Currently we decide to stop the loop before it can even start, so we end
>> up with an incorrect display even if there would not be any such
>> inf-looping.
> I'd be among the first to appreciate a hook that is run once after the
> display engine has established new window-start/-end/-point positions.
> But having the display engine potentially call that hook again and again
> would scare me.

I don't want an inf-loop either.
Basically, I'd want something like:
1- pre-redisplay-function
2- do redisplay
3- point is out of the window, so move point
4- pre-redisplay-function again
5- do redisplay again
6- point is out of the window again, so move point again but without
   going through a new redisplay loop (might want to record the problem
   in *Messages* instead)

tho the exact behavior of 6 is not very important to me (as long as
it doesn't loop indefinitely).

Also, step (4) will often do nothing in which case we should be able to
skip 5 and 6.  For that we might need to slightly change
pre-redisplay-function by making it return whether something was changed
or not.


        Stefan

Re: Can we make set_point_both less expensive?

[martin rudalics]

 >> The loop wouldn't be "an error in any case".  It would be the result of
 >> an interaction of the function on the hook and the display engine.
 >
 > Right, but such an interaction is undesirable, so it would reflect
 > a misbehavior, i.e. an error.

In my experience errors in hooks are a pain to detect and debug.  So I'd
rather try to inhibit potential misbehavior than to fix it.

 > Basically, I'd want something like:
 > 1- pre-redisplay-function
 > 2- do redisplay
 > 3- point is out of the window, so move point
 > 4- pre-redisplay-function again
 > 5- do redisplay again
 > 6- point is out of the window again, so move point again but without
 >     going through a new redisplay loop (might want to record the problem
 >     in *Messages* instead)
 >
 > tho the exact behavior of 6 is not very important to me (as long as
 > it doesn't loop indefinitely).
 >
 > Also, step (4) will often do nothing in which case we should be able to
 > skip 5 and 6.  For that we might need to slightly change
 > pre-redisplay-function by making it return whether something was changed
 > or not.

I would have "4" (which presumably would happen around the time
mark_window_display_accurate is run) call a different function so that
the callee is aware of (1) whether the window's point/start/end
positions are valid and (2) the fact that redisplay might not call it
again before the end of the present redisplay cycle.

martin

Re: Can we make set_point_both less expensive?

[Stefan Monnier]

>>> The loop wouldn't be "an error in any case".  It would be the result of
>>> an interaction of the function on the hook and the display engine.
>> Right, but such an interaction is undesirable, so it would reflect
>> a misbehavior, i.e. an error.
> In my experience errors in hooks are a pain to detect and debug.

Agreed.

> So I'd rather try to inhibit potential misbehavior than to fix it.

Not running the hook a second time (like we do now) doesn't inhibit this
bad interaction.
And I'm not sure how we could actually inhibit this bad interaction.

> I would have "4" (which presumably would happen around the time
> mark_window_display_accurate is run) call a different function so that
> the callee is aware of (1) whether the window's point/start/end
> positions are valid and (2) the fact that redisplay might not call it
> again before the end of the present redisplay cycle.

Sounds fine.  It would also solve the issue of changing the API so the
function can indicate whether a full redisplay is needed.

Tho, I'd expect that this function will need/want to do exactly the same
as pre-redisplay-function, so maybe keeping a single function (but
providing it with some extra argument/data to distinguish the two cases
if/when needed) would work better.

And it still wouldn't prevent bad interactions (this new hook could
still move point out of the window), of course.


        Stefan

Re: Can we make set_point_both less expensive?

[martin rudalics]

 >> So I'd rather try to inhibit potential misbehavior than to fix it.
 >
 > Not running the hook a second time (like we do now) doesn't inhibit this
 > bad interaction.
 > And I'm not sure how we could actually inhibit this bad interaction.

By not running the same hook twice (AFAIAC a bad interaction is only one
where the function and redisplay try forever to annihilate each other's
actions).  Although one problem of `pre-redisplay-function' is that for
my taste it already runs a bit too often, for example, whenever we blink
the cursor.

 > Tho, I'd expect that this function will need/want to do exactly the same
 > as pre-redisplay-function, so maybe keeping a single function (but
 > providing it with some extra argument/data to distinguish the two cases
 > if/when needed) would work better.

But wouldn't we then have to change the signature of functions on
`pre-redisplay-function'?  In any case, I don't know yet about the major
use cases you have in mind.  `redisplay--update-region-highlights' is
not very overhead-intensive and thus probably not very representative
for what may follow.  With `scroll-restore-mode' I'm only interested in
calls where redisplay has already calculated the prospective position of
`window-point'.  The current version of `pre-redisplay-function' is not
useful for that.

 > And it still wouldn't prevent bad interactions (this new hook could
 > still move point out of the window), of course.

Yes.  And redisplay would move it back (or maybe not if we decide to
allow such behavior in general and not for horizontal scrolling alone).

martin

Re: Can we make set_point_both less expensive?

[Stefan Monnier]

> By not running the same hook twice (AFAIAC a bad interaction is only one
> where the function and redisplay try forever to annihilate each other's
> actions).

For me, a bad interaction is one where p-r-f followed by redisplay
followed by p-r-f followed by redisplay doesn't give the right answer.

But with something like `cursor-intangible', there will surely be cases
where that can happen.  E.g. imagine you scroll to a part of the buffer
that's got a cursor-intangible element: redisplay will move point to
a visible (and intangible) part of the buffer, then run p-r-f which
moves point outside of the window, after which redisplay has two
choices: either it decides to scroll to where point is (which would be
wrong, since we've just scrolled so we should move point rather than
scroll), or it moves point back into the intangible element.

> Although one problem of `pre-redisplay-function' is that for my taste
> it already runs a bit too often, for example, whenever we blink
> the cursor.

I agree that running it for cursor blinking is a misfeature.

> But wouldn't we then have to change the signature of functions on
> `pre-redisplay-function'?

Yup.

> In any case, I don't know yet about the major
> use cases you have in mind.

Implement a new `cursor-intangible' property, by having p-r-f move point
away from text with that property.


        Stefan

Re: Can we make set_point_both less expensive?

[Lennart Borgman]

On Fri, Mar 20, 2015 at 3:14 PM, Stefan Monnier
<monnier@iro.umontreal.ca> wrote:
>
> For me, a bad interaction is one where p-r-f followed by redisplay
> followed by p-r-f followed by redisplay doesn't give the right answer.
>
> But with something like `cursor-intangible', there will surely be cases
> where that can happen.  E.g. imagine you scroll to a part of the buffer
> that's got a cursor-intangible element: redisplay will move point to
> a visible (and intangible) part of the buffer, then run p-r-f which
> moves point outside of the window, after which redisplay has two
> choices: either it decides to scroll to where point is (which would be
> wrong, since we've just scrolled so we should move point rather than
> scroll), or it moves point back into the intangible element.

Unless the functions in the hooks continues to change the display this
can't loop, or can it?

Re: Can we make set_point_both less expensive?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Date: Fri, 20 Mar 2015 10:14:11 -0400
> Cc: Eli Zaretskii <eliz@gnu.org>, dancol@dancol.org, emacs-devel@gnu.org
> 
> Implement a new `cursor-intangible' property, by having p-r-f move point
> away from text with that property.

The specification should be much more detailed.  E.g., right now the
cursor is set by a backend-specific function after the redisplay cycle
is already done.

Re: Can we make set_point_both less expensive?

[martin rudalics]

 > For me, a bad interaction is one where p-r-f followed by redisplay
 > followed by p-r-f followed by redisplay doesn't give the right answer.

OK.

 > But with something like `cursor-intangible', there will surely be cases
 > where that can happen.  E.g. imagine you scroll to a part of the buffer
 > that's got a cursor-intangible element: redisplay will move point to
 > a visible (and intangible) part of the buffer, then run p-r-f which
 > moves point outside of the window, after which redisplay has two
 > choices: either it decides to scroll to where point is (which would be
 > wrong, since we've just scrolled so we should move point rather than
 > scroll), or it moves point back into the intangible element.

I suppose we agree that when the entire window text is intangible, no
"classic" solution to this problem exists.  So a solution will exist iff
the intangible area leaves some tangible text above or below.  Handling
the intangible property makes IMO sense only after redisplay has
determined the start/point/end positions of the window.  Then the second
incarnation of p-r-f will have to move the window's point to the start
or end of the intangible area, whichever is nearer to the window point
established by redisplay.  Or do I miss something?

 > Implement a new `cursor-intangible' property, by having p-r-f move point
 > away from text with that property.

Strictly spoken I see no reason why the display engine should not handle
that property by itself.  Someone would have to handle the scroll
margins too when determining on which side of the area to move point.
And someone would have to remember the old window start position to
ensure that scrolling actually makes some progress.  So in the worst
case we would probably have to come up with a "non-classic" solution
anyway where point ends up outside the window.

martin

Re: Can we make set_point_both less expensive?

[Eli Zaretskii]

> Date: Fri, 20 Mar 2015 15:48:21 +0100
> From: martin rudalics <rudalics@gmx.at>
> CC: Eli Zaretskii <eliz@gnu.org>, dancol@dancol.org, emacs-devel@gnu.org
> 
>  > Implement a new `cursor-intangible' property, by having p-r-f move point
>  > away from text with that property.
> 
> Strictly spoken I see no reason why the display engine should not handle
> that property by itself.

Redisplay never moves point, except when it was forced to keep a
specific window-start position.  In all other cases, it scrolls the
display, but leave point intact.

And cursor is always set where point is, we have no other strategy for
positioning the cursor.  So I don't see how handling intangible could
be the job of redisplay, without significant changes.

> Someone would have to handle the scroll margins too when determining
> on which side of the area to move point.

Scroll margins don't cause point movement; they cause movement of text
wrt the window.

Re: Can we make set_point_both less expensive?

[Stefan Monnier]

> Redisplay never moves point, except when it was forced to keep a
> specific window-start position.

This is the case under discussion.


        Stefan

Re: Can we make set_point_both less expensive?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Cc: martin rudalics <rudalics@gmx.at>, dancol@dancol.org, emacs-devel@gnu.org
> Date: Fri, 20 Mar 2015 13:57:57 -0400
> 
> > Redisplay never moves point, except when it was forced to keep a
> > specific window-start position.
> 
> This is the case under discussion.

That's not my understanding.

Re: Can we make set_point_both less expensive?

[Stefan Monnier]

>> > Redisplay never moves point, except when it was forced to keep a
>> > specific window-start position.
>> This is the case under discussion.
> That's not my understanding.

I know, but I assure you that it is.  It started with your comment:

   That contradicts the "catch movement of cursor" idea: redisplay could
   well move point from where it is found before redisplay, as you know.


-- Stefan

Re: Can we make set_point_both less expensive?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Cc: rudalics@gmx.at, dancol@dancol.org, emacs-devel@gnu.org
> Date: Fri, 20 Mar 2015 17:50:25 -0400
> 
> >> > Redisplay never moves point, except when it was forced to keep a
> >> > specific window-start position.
> >> This is the case under discussion.
> > That's not my understanding.
> 
> I know, but I assure you that it is.  It started with your comment:
> 
>    That contradicts the "catch movement of cursor" idea: redisplay could
>    well move point from where it is found before redisplay, as you know.

I know, but it sounds like you are talking about much broader issues
now than just that contradiction.

In any case, the issue of where the cursor position which will be
shown to user is known still stands.

Re: Can we make set_point_both less expensive?

[martin rudalics]

 > In any case, the issue of where the cursor position which will be
 > shown to user is known still stands.

Now it's my turn to not understand what you're saying ;-)

martin

Re: Can we make set_point_both less expensive?

[Eli Zaretskii]

> Date: Sat, 21 Mar 2015 10:58:34 +0100
> From: martin rudalics <rudalics@gmx.at>
> CC: dancol@dancol.org, emacs-devel@gnu.org
> 
>  > In any case, the issue of where the cursor position which will be
>  > shown to user is known still stands.
> 
> Now it's my turn to not understand what you're saying ;-)

Let me rephrase:

What code knows where is the cursor position that will actually be
shown?

Re: Can we make set_point_both less expensive?

[martin rudalics]

 > What code knows where is the cursor position that will actually be
 > shown?

The redisplay code, hopefully.  And no-one should interfere with it,
hopefully.

martin

Re: Can we make set_point_both less expensive?

[Eli Zaretskii]

> Date: Sat, 21 Mar 2015 13:30:55 +0100
> From: martin rudalics <rudalics@gmx.at>
> CC: dancol@dancol.org, monnier@IRO.UMontreal.CA, emacs-devel@gnu.org
> 
>  > What code knows where is the cursor position that will actually be
>  > shown?
> 
> The redisplay code, hopefully.

Yes, but which code in redisplay, exactly?

Re: Can we make set_point_both less expensive?

[martin rudalics]

 >> Strictly spoken I see no reason why the display engine should not handle
 >> that property by itself.
 >
 > Redisplay never moves point, except when it was forced to keep a
 > specific window-start position.  In all other cases, it scrolls the
 > display, but leave point intact.

But when redisplay moves point (usually after scrolling the window) it
could check whether it would end up in an intangible region and adjust
point accordingly.  Why would this be such a big deal?

 > And cursor is always set where point is, we have no other strategy for
 > positioning the cursor.

After `scroll-left'?

 > So I don't see how handling intangible could
 > be the job of redisplay, without significant changes.

Isn't adjust_point_for_property some sort of redisplay revisited?

 >> Someone would have to handle the scroll margins too when determining
 >> on which side of the area to move point.
 >
 > Scroll margins don't cause point movement; they cause movement of text
 > wrt the window.

They would determine the position of the cursor when an intangible area
starts or ends within them.

martin

Re: Can we make set_point_both less expensive?

[Eli Zaretskii]

> Date: Fri, 20 Mar 2015 20:53:53 +0100
> From: martin rudalics <rudalics@gmx.at>
> CC: dancol@dancol.org, monnier@IRO.UMontreal.CA, emacs-devel@gnu.org
> 
>  >> Strictly spoken I see no reason why the display engine should not handle
>  >> that property by itself.
>  >
>  > Redisplay never moves point, except when it was forced to keep a
>  > specific window-start position.  In all other cases, it scrolls the
>  > display, but leave point intact.
> 
> But when redisplay moves point (usually after scrolling the window) it
> could check whether it would end up in an intangible region and adjust
> point accordingly.  Why would this be such a big deal?

It could, but AFAIU, this is not the only situation that is being
discussed.  Stefan wants redisplay to move point also in the case
where some Lisp moved point and entered a region of buffer positions
that has a certain text property.  That's not a scrolling scenario you
describe.

>  > And cursor is always set where point is, we have no other strategy for
>  > positioning the cursor.
> 
> After `scroll-left'?

Yes, after scroll-left, too.  We simply put the cursor as close to
point as possible.

>  > So I don't see how handling intangible could
>  > be the job of redisplay, without significant changes.
> 
> Isn't adjust_point_for_property some sort of redisplay revisited?

No.  It does nothing that is related to display.

>  >> Someone would have to handle the scroll margins too when determining
>  >> on which side of the area to move point.
>  >
>  > Scroll margins don't cause point movement; they cause movement of text
>  > wrt the window.
> 
> They would determine the position of the cursor when an intangible area
> starts or ends within them.

Sorry, too much "they" in this sentence, I don't really understand
what you are sayin, or how it is relevant to the issue at hand.

Re: Can we make set_point_both less expensive?

[Stefan Monnier]

> discussed.  Stefan wants redisplay to move point also in the case
> where some Lisp moved point and entered a region of buffer positions
> that has a certain text property.

Yes and no: I want pre-redisplay-function to do that.  Whether you
consider it as "redisplay" or not is another question, but when I say
"redisplay" in this thread I mean the C part that fills & uses the
glyph matrices (i.e. it does not include pre-redisplay-function).


        Stefan

Re: Can we make set_point_both less expensive?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Cc: martin rudalics <rudalics@gmx.at>, dancol@dancol.org, emacs-devel@gnu.org
> Date: Fri, 20 Mar 2015 17:53:22 -0400
> 
> > discussed.  Stefan wants redisplay to move point also in the case
> > where some Lisp moved point and entered a region of buffer positions
> > that has a certain text property.
> 
> Yes and no: I want pre-redisplay-function to do that.

This will only work if pre-redisplay-function is called in a way that
guarantees it will catch the position of point after the previous
command.  Another, related, requirement is that we never call
pre-redisplay-function when point is in a position that is not
supposed to be visible by the user.

Overall, it sounds like a non-trivial deviation from the original
purpose of pre-redisplay-function, as I understand it.

> Whether you consider it as "redisplay" or not is another question,

I don't consider pre-redisplay-function part of redisplay.  Martin was
clearly talking about redisplay, so I answered in kind.

What I _am_ saying is that only redisplay, in its last part, knows
where the cursor will be positioned.

> but when I say "redisplay" in this thread I mean the C part that
> fills & uses the glyph matrices

That part of redisplay doesn't yet know where the cursor will be.
Cursor positioning happens after that.

Re: Can we make set_point_both less expensive?

[martin rudalics]

 > This will only work if pre-redisplay-function is called in a way that
 > guarantees it will catch the position of point after the previous
 > command.

This command, I presume.

 > Another, related, requirement is that we never call
 > pre-redisplay-function when point is in a position that is not
 > supposed to be visible by the user.

Do you mean invisible text here?

 > Overall, it sounds like a non-trivial deviation from the original
 > purpose of pre-redisplay-function, as I understand it.
 >
 >> Whether you consider it as "redisplay" or not is another question,
 >
 > I don't consider pre-redisplay-function part of redisplay.  Martin was
 > clearly talking about redisplay, so I answered in kind.

Redisplay happens in cycles where a subsequent cycle (and maybe also
some input that arrived in between) can invalidate what happened before.
And IMO certain things that happen at top level like making the selected
window's buffer current or adjusting point in adjust_point_for_property
are part of redisplay too.  Here we disagree, obviously.

 > What I _am_ saying is that only redisplay, in its last part, knows
 > where the cursor will be positioned.

Precisely; provided it runs to completion.

 >> but when I say "redisplay" in this thread I mean the C part that
 >> fills & uses the glyph matrices
 >
 > That part of redisplay doesn't yet know where the cursor will be.
 > Cursor positioning happens after that.

And sometimes even after a first redisplay, if I can trust my eyes.

martin

Re: Can we make set_point_both less expensive?

[Eli Zaretskii]

> Date: Sat, 21 Mar 2015 10:59:28 +0100
> From: martin rudalics <rudalics@gmx.at>
> CC: dancol@dancol.org, emacs-devel@gnu.org
> 
>  > This will only work if pre-redisplay-function is called in a way that
>  > guarantees it will catch the position of point after the previous
>  > command.
> 
> This command, I presume.

The command that preceded redisplay.  "This command" is ambiguous.

>  > Another, related, requirement is that we never call
>  > pre-redisplay-function when point is in a position that is not
>  > supposed to be visible by the user.
> 
> Do you mean invisible text here?

No, I mean positions traversed while "this command" is executed.  The
position shown to the user is the last one of those, but how do we
ensure that pre-redisplay-function is never run for those?

Re: Can we make set_point_both less expensive?

[Eli Zaretskii]

> Date: Sat, 21 Mar 2015 12:46:32 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: dancol@dancol.org, monnier@IRO.UMontreal.CA, emacs-devel@gnu.org
> 
> No, I mean positions traversed while "this command" is executed.  The
> position shown to the user is the last one of those, but how do we
> ensure that pre-redisplay-function is never run for those?
                                                  ^^^^^^^^^
"for others"

Re: Can we make set_point_both less expensive?

[martin rudalics]

 >>   > This will only work if pre-redisplay-function is called in a way that
 >>   > guarantees it will catch the position of point after the previous
 >>   > command.
 >>
 >> This command, I presume.
 >
 > The command that preceded redisplay.  "This command" is ambiguous.

Then the value of `this-command' provided it's non-nil at the time
`pre-redisplay-function' is called.  What is the position of point when
calling `pre-redisplay-function' after `scroll-bar-toolkit-scroll'?  The
one before `scroll-bar-toolkit-scroll' was called, I presume.

 >>   > Another, related, requirement is that we never call
 >>   > pre-redisplay-function when point is in a position that is not
 >>   > supposed to be visible by the user.
 >>
 >> Do you mean invisible text here?
 >
 > No, I mean positions traversed while "this command" is executed.  The
 > position shown to the user is the last one of those, but how do we
 > ensure that pre-redisplay-function is never run for those?

We won't I presume.  My understanding is that C code calls and will
continue to call `pre-redisplay-function' regardless of whether "point
is in a position that is not supposed to be visible by the user".  It
will simply try to come up with the best position of point it can
provide at that moment.

martin

Re: Can we make set_point_both less expensive?

[Stefan Monnier]

>> > Another, related, requirement is that we never call
>> > pre-redisplay-function when point is in a position that is not
>> > supposed to be visible by the user.
>> Do you mean invisible text here?
> No, I mean positions traversed while "this command" is executed.  The
> position shown to the user is the last one of those, but how do we
> ensure that pre-redisplay-function is never run for those?

Ah, I see what you mean now: the command ensures this by not causing
redisplay during its execution.

Since pre-redisplay-function is run only at the beginning of redisplay,
it is only run when we're about to make the current state "visible by
the user".


        Stefan

Re: Can we make set_point_both less expensive?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Cc: martin rudalics <rudalics@gmx.at>, dancol@dancol.org, emacs-devel@gnu.org
> Date: Sat, 21 Mar 2015 10:03:53 -0400
> 
> >> > Another, related, requirement is that we never call
> >> > pre-redisplay-function when point is in a position that is not
> >> > supposed to be visible by the user.
> >> Do you mean invisible text here?
> > No, I mean positions traversed while "this command" is executed.  The
> > position shown to the user is the last one of those, but how do we
> > ensure that pre-redisplay-function is never run for those?
> 
> Ah, I see what you mean now: the command ensures this by not causing
> redisplay during its execution.

Commands don't have full enough control to ensure that, I think.
Various features and customizations could trigger redisplay behind
their back.

Re: Can we make set_point_both less expensive?

[Stefan Monnier]

>> >> > Another, related, requirement is that we never call
>> >> > pre-redisplay-function when point is in a position that is not
>> >> > supposed to be visible by the user.
>> >> Do you mean invisible text here?
>> > No, I mean positions traversed while "this command" is executed.  The
>> > position shown to the user is the last one of those, but how do we
>> > ensure that pre-redisplay-function is never run for those?
>> Ah, I see what you mean now: the command ensures this by not causing
>> redisplay during its execution.
> Commands don't have full enough control to ensure that, I think.
> Various features and customizations could trigger redisplay behind
> their back.

Of course, but if redisplay is triggered, then the state is
made visible.
So if the state has "a position that is not supposed to be visible by
the user" it's an error for the code to cause redisplay.
The existence and use of pre-redisplay-function doesn't affect this.


        Stefan

Re: Can we make set_point_both less expensive?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Date: Sat, 21 Mar 2015 13:28:18 -0400
> Cc: rudalics@gmx.at, dancol@dancol.org, emacs-devel@gnu.org
> 
> >> >> > Another, related, requirement is that we never call
> >> >> > pre-redisplay-function when point is in a position that is not
> >> >> > supposed to be visible by the user.
> >> >> Do you mean invisible text here?
> >> > No, I mean positions traversed while "this command" is executed.  The
> >> > position shown to the user is the last one of those, but how do we
> >> > ensure that pre-redisplay-function is never run for those?
> >> Ah, I see what you mean now: the command ensures this by not causing
> >> redisplay during its execution.
> > Commands don't have full enough control to ensure that, I think.
> > Various features and customizations could trigger redisplay behind
> > their back.
> 
> Of course, but if redisplay is triggered, then the state is
> made visible.
> So if the state has "a position that is not supposed to be visible by
> the user" it's an error for the code to cause redisplay.
> The existence and use of pre-redisplay-function doesn't affect this.

I understand the theory, but I think in practice doing that in
pre-redisplay-function will not do a perfect job, perhaps not even a
good enough job.  The position of point/cursor shown to the user is
known _at_the_end_ of redisplay, not at its beginning.  the latter can
approximate the former, but only approximate.

Re: Can we make set_point_both less expensive?

[Stefan Monnier]

> good enough job.  The position of point/cursor shown to the user is
> known _at_the_end_ of redisplay, not at its beginning.  the latter can
> approximate the former, but only approximate.

It's known well enough in pre-redisplay-function that we can use this
information to setup an overlay to highlight the region.  And we haven't
had bug-reports yet about this "approximation" being imprecise (tho
I admittedly fixed just such a bug between 24.4 and 25.1).


        Stefan

Re: Can we make set_point_both less expensive?

[martin rudalics]

 > It's known well enough in pre-redisplay-function that we can use this
 > information to setup an overlay to highlight the region.  And we haven't
 > had bug-reports yet about this "approximation" being imprecise (tho
 > I admittedly fixed just such a bug between 24.4 and 25.1).

One thing that's troubling me is that in pre-redisplay-function
this-command is apparently always nil.  For example:

(defun show-this-command (&windows)
   "..."
   (message "This command is %s" this-command))

(add-function :after pre-redisplay-function #'show-this-command)

martin

Re: Can we make set_point_both less expensive?

[Stefan Monnier]

> One thing that's troubling me is that in pre-redisplay-function
> this-command is apparently always nil.

Not always.  If run during `sit-for' it will have a non-nil
this-command, but in the "usual" case, indeed it's run between two
commands, so this-command is nil.


        Stefan

Re: Can we make set_point_both less expensive?

[Stefan Monnier]

>> > discussed.  Stefan wants redisplay to move point also in the case
>> > where some Lisp moved point and entered a region of buffer positions
>> > that has a certain text property.
>> Yes and no: I want pre-redisplay-function to do that.
> This will only work if pre-redisplay-function is called in a way that
> guarantees it will catch the position of point after the previous
> command.

I think I don't understand what you're saying:
pre-redisplay-function is called at the beginning of redisplay, so of
course it's called "after the previous command".

> Another, related, requirement is that we never call
> pre-redisplay-function when point is in a position that is not
> supposed to be visible by the user.

Again I don't understand.  pre-redisplay-function will be called at the
beginning of every redisplay, so it can't assume anything about where
point is.  On the contrary, it's pre-redisplay-function which would be
in charge of changing point (if needed) to take `cursor-intangible'
into account.

> Overall, it sounds like a non-trivial deviation from the original
> purpose of pre-redisplay-function, as I understand it.

I assure you it isn't: `cursor-intangible' was very much in my mind
while I was implementing this new feature.

> What I _am_ saying is that only redisplay, in its last part, knows
> where the cursor will be positioned.

Of course: pre-redisplay-function only has control of `point', not over
the cursor.  So it should move point away from the `cursor-intangible'
property.  The difference between point and the place where the cursor
is drawn on screen is not really relevant to `cursor-intangible'.


        Stefan

Re: Can we make set_point_both less expensive?

[Stefan Monnier]

BTW, I now remember that re-running pre-redisplay-function (and then
redisplay) is already implemented in xdisp.c.  The part that's not yet
implemented is to detect inf-loops and break them (i.e. to not run
pre-redisplay-function when we're doing the second redisplay).

The relevant code is

	  /* Re-run pre-redisplay-function so it can update the region
	     according to the new position of point.  */
	  /* Other than the cursor, w's redisplay is done so we can set its
	     redisplay to false.  Also the buffer's redisplay can be set to
	     false, since propagate_buffer_redisplay should have already
	     propagated its info to `w' anyway.  */
	  w->redisplay = false;
	  XBUFFER (w->contents)->text->redisplay = false;
	  safe__call1 (true, Vpre_redisplay_function, Fcons (window, Qnil));

	  if (w->redisplay || XBUFFER (w->contents)->text->redisplay)
	    {
	      /* pre-redisplay-function made changes (e.g. move the region)
		 that require another round of redisplay.  */
	      clear_glyph_matrix (w->desired_matrix);
	      if (!try_window (window, startp, 0))
		goto need_larger_matrices;
	    }

This was needed to handle correctly the following recipe:

   emacs -Q lisp/subr.el \
         --eval "(global-set-key [?\\M-n] \
                                 (lambda () \
                                   (interactive) \
                                   (set-window-start \
                                    nil (+ (window-start) 1000))))" \
         -f blink-cursor-mode
   C-SPC M-n M-n M-n

This potential inf-loop is new in Emacs-25 (instead, in Emacs-24.4, the
recipe misbehaves).


        Stefan

Re: Can we make set_point_both less expensive?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Cc: rudalics@gmx.at, dancol@dancol.org, emacs-devel@gnu.org
> Date: Sat, 21 Mar 2015 10:35:10 -0400
> 
> BTW, I now remember that re-running pre-redisplay-function (and then
> redisplay) is already implemented in xdisp.c.

So do you still claim that pre-redisplay-function is only run "before
redisplay", where "we can be sure the value of point is the one that
will be shown to the user"?  Or do you now understand the nature of my
concerns?

IOW, do you still think pre-redisplay-function is the place to put
this logic of "cursor-intangible", or maybe there's a better place?

Re: Can we make set_point_both less expensive?

[Stefan Monnier]

> So do you still claim that pre-redisplay-function is only run "before
> redisplay", where "we can be sure the value of point is the one that
> will be shown to the user"?

Yes.

> Or do you now understand the nature of my concerns?

No, I still don't.

> IOW, do you still think pre-redisplay-function is the place to put
> this logic of "cursor-intangible",

Yes, I think it's a good place.

> or maybe there's a better place?

Like where?


        Stefan

Re: Can we make set_point_both less expensive?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Cc: rudalics@gmx.at, dancol@dancol.org, emacs-devel@gnu.org
> Date: Sat, 21 Mar 2015 13:10:45 -0400
> 
> > So do you still claim that pre-redisplay-function is only run "before
> > redisplay", where "we can be sure the value of point is the one that
> > will be shown to the user"?
> 
> Yes.

I don't see how you can say that, when you just said it will be run
elsewhere.

> > or maybe there's a better place?
> 
> Like where?

Where we set the cursor, of course.

Re: Can we make set_point_both less expensive?

[Stefan Monnier]

>> > So do you still claim that pre-redisplay-function is only run "before
>> > redisplay", where "we can be sure the value of point is the one that
>> > will be shown to the user"?
>> Yes.
> I don't see how you can say that, when you just said it will be run
> elsewhere.

AFAIK this "elsewhere" is still "before redisplay".

>> > or maybe there's a better place?
>> Like where?
> Where we set the cursor, of course.

What if it moves point outside the window?


        Stefan

Re: Can we make set_point_both less expensive?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Cc: rudalics@gmx.at, dancol@dancol.org, emacs-devel@gnu.org
> Date: Sat, 21 Mar 2015 18:12:08 -0400
> 
> >> > So do you still claim that pre-redisplay-function is only run "before
> >> > redisplay", where "we can be sure the value of point is the one that
> >> > will be shown to the user"?
> >> Yes.
> > I don't see how you can say that, when you just said it will be run
> > elsewhere.
> 
> AFAIK this "elsewhere" is still "before redisplay".

No, it's "during redisplay".

> >> > or maybe there's a better place?
> >> Like where?
> > Where we set the cursor, of course.
> 
> What if it moves point outside the window?

It can't.

Re: Can we make set_point_both less expensive?

[Stefan Monnier]

>> AFAIK this "elsewhere" is still "before redisplay".
> No, it's "during redisplay".

It's right after an incomplete redisplay, and right before the
next redisplay.  I count this as still "before redisplay" since it's
immediately followed by redisplay.

>> >> > or maybe there's a better place?
>> >> Like where?
>> > Where we set the cursor, of course.
>> What if it moves point outside the window?
> It can't.

That's a misfeature: when using cursor-intangible to force going from
field to field, it may very well be that C-n makes you jump to the next
field which is past window-end (even though C-n itself just jumped to
the next char which is still before window-end), so it should be
possible for the handling of cursor-intangible to move point outside of
the window.


        Stefan

Re: Can we make set_point_both less expensive?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Cc: rudalics@gmx.at, dancol@dancol.org, emacs-devel@gnu.org
> Date: Sun, 22 Mar 2015 09:58:02 -0400
> 
> >> AFAIK this "elsewhere" is still "before redisplay".
> > No, it's "during redisplay".
> 
> It's right after an incomplete redisplay, and right before the
> next redisplay.  I count this as still "before redisplay" since it's
> immediately followed by redisplay.

It is "during redisplay" in the sense that the position of cursor was
not yet finalized, and you already move it, perhaps for no good
reason.

> >> >> > or maybe there's a better place?
> >> >> Like where?
> >> > Where we set the cursor, of course.
> >> What if it moves point outside the window?
> > It can't.
> 
> That's a misfeature: when using cursor-intangible to force going from
> field to field, it may very well be that C-n makes you jump to the next
> field which is past window-end (even though C-n itself just jumped to
> the next char which is still before window-end), so it should be
> possible for the handling of cursor-intangible to move point outside of
> the window.

Ah, I misunderstood your question.  The answer for your question is
"abort redisplay cycle after moving point out of intangible".

Re: Can we make set_point_both less expensive?

[Stefan Monnier]

>> It's right after an incomplete redisplay, and right before the
>> next redisplay.  I count this as still "before redisplay" since it's
>> immediately followed by redisplay.
> It is "during redisplay" in the sense that the position of cursor was
> not yet finalized, and you already move it, perhaps for no good
> reason.

I don't deny it can be described differently.  I.e. sometimes it
happens during redisplay, sometimes it happens in a sit-for, sometimes
it happens at the end of a command.  But always it happens right
before a redisplay.

> Ah, I misunderstood your question.  The answer for your question is
> "abort redisplay cycle after moving point out of intangible".

In which case you risk inf-looping (and hence need to detect/prevent
it), same as with pre-redisplay-function.


        Stefan

Re: Can we make set_point_both less expensive?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: rudalics@gmx.at,  dancol@dancol.org,  emacs-devel@gnu.org
> Date: Sun, 22 Mar 2015 18:23:24 -0400
> 
> >> It's right after an incomplete redisplay, and right before the
> >> next redisplay.  I count this as still "before redisplay" since it's
> >> immediately followed by redisplay.
> > It is "during redisplay" in the sense that the position of cursor was
> > not yet finalized, and you already move it, perhaps for no good
> > reason.
> 
> I don't deny it can be described differently.  I.e. sometimes it
> happens during redisplay, sometimes it happens in a sit-for, sometimes
> it happens at the end of a command.  But always it happens right
> before a redisplay.

I don't understand what you are trying to say.  Every single moment in
Emacs is before _some_ redisplay, that's true, but I don't believe you
meant that.

Otherwise, running the function before redisplay finished its job and
returned after marking the window up-to-day is bound to catch
incorrect position of point/cursor sometimes.

> > Ah, I misunderstood your question.  The answer for your question is
> > "abort redisplay cycle after moving point out of intangible".
> 
> In which case you risk inf-looping (and hence need to detect/prevent
> it), same as with pre-redisplay-function.

No, because the code to abort redisplay was written long ago and is
well tested, by time if nothing else.

Re: Can we make set_point_both less expensive?

[Stefan Monnier]

>> it happens at the end of a command.  But always it happens right
>> before a redisplay.
> I don't understand what you are trying to say.  Every single moment in
> Emacs is before _some_ redisplay, that's true, but I don't believe you
> meant that.

There's "before" and then there's "right before".

>> In which case you risk inf-looping (and hence need to detect/prevent
>> it), same as with pre-redisplay-function.
> No, because the code to abort redisplay was written long ago and is
> well tested, by time if nothing else.

Then, maybe this redisplay-abort thingy already works for
pre-redisplay-function as well.


        Stefan

Re: Can we make set_point_both less expensive?

[Stefan Monnier]

> Otherwise, running the function before redisplay finished its job and
> returned after marking the window up-to-day is bound to catch
> incorrect position of point/cursor sometimes.

I'm completely lost.  What kind of "incorrect position" and what kind of
"catching" are you worried about?


        Stefan

Re: Can we make set_point_both less expensive?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: rudalics@gmx.at,  dancol@dancol.org,  emacs-devel@gnu.org
> Date: Mon, 23 Mar 2015 15:28:53 -0400
> 
> > Otherwise, running the function before redisplay finished its job and
> > returned after marking the window up-to-day is bound to catch
> > incorrect position of point/cursor sometimes.
> 
> I'm completely lost.  What kind of "incorrect position" and what kind of
> "catching" are you worried about?

"Incorrect position" == position that will never be shown to the user.

"Catching" == using an incorrect position for the purposes discussed
in this thread.

Re: Can we make set_point_both less expensive?

[Stefan Monnier]

>> I'm completely lost.  What kind of "incorrect position" and what kind of
>> "catching" are you worried about?
> "Incorrect position" == position that will never be shown to the user.
> "Catching" == using an incorrect position for the purposes discussed
> in this thread.

I still have no idea what scenario you're worried about.


        Stefan

Re: Can we make set_point_both less expensive?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Cc: rudalics@gmx.at, dancol@dancol.org, emacs-devel@gnu.org
> Date: Sat, 21 Mar 2015 09:59:35 -0400
> 
> >> > discussed.  Stefan wants redisplay to move point also in the case
> >> > where some Lisp moved point and entered a region of buffer positions
> >> > that has a certain text property.
> >> Yes and no: I want pre-redisplay-function to do that.
> > This will only work if pre-redisplay-function is called in a way that
> > guarantees it will catch the position of point after the previous
> > command.
> 
> I think I don't understand what you're saying:
> pre-redisplay-function is called at the beginning of redisplay, so of
> course it's called "after the previous command".

But in the course of this discussion, you suggested it to be called in
other places as well.

> > Another, related, requirement is that we never call
> > pre-redisplay-function when point is in a position that is not
> > supposed to be visible by the user.
> 
> Again I don't understand.

My concern is again for your suggestions to call it under additional
circumstances.

> > Overall, it sounds like a non-trivial deviation from the original
> > purpose of pre-redisplay-function, as I understand it.
> 
> I assure you it isn't: `cursor-intangible' was very much in my mind
> while I was implementing this new feature.

Maybe so, but are you sure it is used for purposes that are similar to
to what you envisioned?  I am not, based on discussions since its
introduction where it was suggested to use it.

> > What I _am_ saying is that only redisplay, in its last part, knows
> > where the cursor will be positioned.
> 
> Of course: pre-redisplay-function only has control of `point', not over
> the cursor.  So it should move point away from the `cursor-intangible'
> property.  The difference between point and the place where the cursor
> is drawn on screen is not really relevant to `cursor-intangible'.

If I now mention, again, that redisplay can move point, we would be
going in circles.  Is that what you want?

Re: Can we make set_point_both less expensive?

[martin rudalics]

 >> discussed.  Stefan wants redisplay to move point also in the case
 >> where some Lisp moved point and entered a region of buffer positions
 >> that has a certain text property.
 >
 > Yes and no: I want pre-redisplay-function to do that.

Do what?

martin

Re: Can we make set_point_both less expensive?

[Stefan Monnier]

>>> discussed.  Stefan wants redisplay to move point also in the case
>>> where some Lisp moved point and entered a region of buffer positions
>>> that has a certain text property.
>> Yes and no: I want pre-redisplay-function to do that.
> Do what?

"move point also in the case where some Lisp moved point and entered
a region of buffer positions that has a certain text property"


        Stefan

Re: Can we make set_point_both less expensive?

[martin rudalics]

 > It could, but AFAIU, this is not the only situation that is being
 > discussed.  Stefan wants redisplay to move point also in the case
 > where some Lisp moved point and entered a region of buffer positions
 > that has a certain text property.  That's not a scrolling scenario you
 > describe.

Stefan will tell us whether he has any additional issues in mind.  Here
I'm only concerned about the scrolling scenario.

 >>   > And cursor is always set where point is, we have no other strategy for
 >>   > positioning the cursor.
 >>
 >> After `scroll-left'?
 >
 > Yes, after scroll-left, too.  We simply put the cursor as close to
 > point as possible.

But not necessarily _where point is_.

 >> Isn't adjust_point_for_property some sort of redisplay revisited?
 >
 > No.  It does nothing that is related to display.

Can't it trigger a new redisplay?

 >> They would determine the position of the cursor when an intangible area
 >> starts or ends within them.
 >
 > Sorry, too much "they" in this sentence, I don't really understand
 > what you are sayin, or how it is relevant to the issue at hand.

Consider the following scenario:

(1) Using the scrollbar thumb, the user scrolls window text thus
     implicitly establishing a new window start position WS.

(2) Respecting WS, redisplay puts window point at some position WP.

(3) A function on `pre-redisplay-function' decides that WP would end up
     within intangible text and moves WP to a position WT.

(4) Redisplay notices that WT would end up in the scroll margin and
     scrolls the window invalidating WS.  As a result windw text would
     not scroll as intended by the user in (1).

But if redisplay were aware of intangible text, it would handle both -
the intangible property and the scroll margin - in (2) and there would
be no need to enter the (3)-(4) rigmarole.

Note that (3) means we have to search all text/overlay properties of
every window that gets redisplayed for the sole purpose of detecting the
presence of intangible text.  In my estimate 99.99% of our windows don't
contain such text.

martin

Re: Can we make set_point_both less expensive?

[Eli Zaretskii]

> Date: Sat, 21 Mar 2015 10:58:01 +0100
> From: martin rudalics <rudalics@gmx.at>
> CC: dancol@dancol.org, monnier@IRO.UMontreal.CA, emacs-devel@gnu.org
> 
>  >>   > And cursor is always set where point is, we have no other strategy for
>  >>   > positioning the cursor.
>  >>
>  >> After `scroll-left'?
>  >
>  > Yes, after scroll-left, too.  We simply put the cursor as close to
>  > point as possible.
> 
> But not necessarily _where point is_.

I was talking about the implementation.  It looks for the glyph that
is the closest to point.  It never consciously moves the cursor to any
other place.

>  >> Isn't adjust_point_for_property some sort of redisplay revisited?
>  >
>  > No.  It does nothing that is related to display.
> 
> Can't it trigger a new redisplay?

Yes, it can.  But I'm not sure it is needed.

>  >> They would determine the position of the cursor when an intangible area
>  >> starts or ends within them.
>  >
>  > Sorry, too much "they" in this sentence, I don't really understand
>  > what you are sayin, or how it is relevant to the issue at hand.
> 
> Consider the following scenario:
> 
> (1) Using the scrollbar thumb, the user scrolls window text thus
>      implicitly establishing a new window start position WS.
> 
> (2) Respecting WS, redisplay puts window point at some position WP.
> 
> (3) A function on `pre-redisplay-function' decides that WP would end up
>      within intangible text and moves WP to a position WT.
> 
> (4) Redisplay notices that WT would end up in the scroll margin and
>      scrolls the window invalidating WS.  As a result windw text would
>      not scroll as intended by the user in (1).

If the flag that forced redisplay to honor WS is still set, then (4)
won't happen, I think.  Instead, redisplay will recenter point in the
window.

> But if redisplay were aware of intangible text, it would handle both -
> the intangible property and the scroll margin - in (2) and there would
> be no need to enter the (3)-(4) rigmarole.

Good luck implementing that!  We already have a thick forest of
conflicting goals in that area, due to requirements for scroll-margin,
scroll-*-aggressively, scroll-conservatively, and scroll-step.

> Note that (3) means we have to search all text/overlay properties of
> every window that gets redisplayed for the sole purpose of detecting the
> presence of intangible text.  In my estimate 99.99% of our windows don't
> contain such text.

We already do precisely that, just in another place, where we move
point.  That was the issue that triggered this thread in the first
place.  Stefan's intent is to lower the number of such searches, by
only doing that for position of point that will be actually shown to
the user, whereas today we do that for interim positions that will
never be displayed.

Re: Can we make set_point_both less expensive?

[martin rudalics]

 > If the flag that forced redisplay to honor WS is still set, then (4)
 > won't happen, I think.  Instead, redisplay will recenter point in the
 > window.

Recentering point during scrolling constitutes the worst of all worlds
in my experience.

 > Good luck implementing that!  We already have a thick forest of
 > conflicting goals in that area,

We can always close our eyes and hope that moving further goals to
`pre-redisplay-function' won't thicken that forest even more so.

 > due to requirements for scroll-margin,
 > scroll-*-aggressively, scroll-conservatively, and scroll-step.

I've never been able to understand these anyway.  Here, I only want to
keep point from recentering.  One option for that would be all I need.

 >> Note that (3) means we have to search all text/overlay properties of
 >> every window that gets redisplayed for the sole purpose of detecting the
 >> presence of intangible text.  In my estimate 99.99% of our windows don't
 >> contain such text.
 >
 > We already do precisely that, just in another place, where we move
 > point.

In adjust_point_for_property?

 > That was the issue that triggered this thread in the first
 > place.  Stefan's intent is to lower the number of such searches, by
 > only doing that for position of point that will be actually shown to
 > the user, whereas today we do that for interim positions that will
 > never be displayed.

The display engine knows what will be displayed where.  Elisp never
will.  I see little reason to delegate to Elisp the task of finding a
position where to show the cursor.

martin

Re: Can we make set_point_both less expensive?

[Eli Zaretskii]

> Date: Sat, 21 Mar 2015 13:30:46 +0100
> From: martin rudalics <rudalics@gmx.at>
> CC: dancol@dancol.org, monnier@IRO.UMontreal.CA, emacs-devel@gnu.org
> 
>  >> Note that (3) means we have to search all text/overlay properties of
>  >> every window that gets redisplayed for the sole purpose of detecting the
>  >> presence of intangible text.  In my estimate 99.99% of our windows don't
>  >> contain such text.
>  >
>  > We already do precisely that, just in another place, where we move
>  > point.
> 
> In adjust_point_for_property?

No, in set_point_both.

Re: Can we make set_point_both less expensive?

[Stefan Monnier]

> I suppose we agree that when the entire window text is intangible, no
> "classic" solution to this problem exists.

Indeed.

> Then the second incarnation of p-r-f will have to move the window's
> point to the start or end of the intangible area, whichever is nearer
> to the window point established by redisplay.  Or do I miss something?

More important than "nearer" would be "after window-start and before
window-end" to avoid inf-looping.

>> Implement a new `cursor-intangible' property, by having p-r-f move point
>> away from text with that property.
> Strictly spoken I see no reason why the display engine should not handle
> that property by itself.

It's complicated enough as it is.


        Stefan

Re: Can we make set_point_both less expensive?

[martin rudalics]

 >> Then the second incarnation of p-r-f will have to move the window's
 >> point to the start or end of the intangible area, whichever is nearer
 >> to the window point established by redisplay.  Or do I miss something?
 >
 > More important than "nearer" would be "after window-start and before
 > window-end" to avoid inf-looping.

Yes.  As I said before: "Handling the intangible property makes IMO
sense only after redisplay has determined the start/point/end positions
of the window."
 >
 >> Strictly spoken I see no reason why the display engine should not handle
 >> that property by itself.
 >
 > It's complicated enough as it is.

Maybe you're right.

martin

Re: Can we make set_point_both less expensive?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

The reason I implemented 'intangible' was for the sake of editing
forms.  A form has fields to be edited, and fixed text that shouldn't
be changed.  But there is no need to put point within the fixed text;
allowing that seems ugly.

'intangible' did not turn out to be a very good implementation
of this.  Can anyone do a better job?

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! See stallman.org/skype.html.

Re: Can we make set_point_both less expensive?

[Yuri Khan]

On Tue, Mar 17, 2015 at 8:29 PM, Richard Stallman <rms@gnu.org> wrote:

> The reason I implemented 'intangible' was for the sake of editing
> forms.  A form has fields to be edited, and fixed text that shouldn't
> be changed.  But there is no need to put point within the fixed text;
> allowing that seems ugly.

There are in fact plenty of reasons to be able to move point all over
the form. Off the top of my head:

* The user might want to mark and copy some part of the form text.
* The user might want to look up some words of the form text in a dictionary.
* The user might want parts of the form text spoken aloud or displayed
on a Braille terminal.
* Moving the point only through the form fields feels unpredictable;
the user feels out of control.

There is nothing wrong in preventing modification of the form text,
though. There can be convenience bindings such as TAB and S-TAB to
move between fields, SPC and S-SPC to scroll by page, RET to activate
hyperlinks, and SPC and RET to activate buttons.

> 'intangible' did not turn out to be a very good implementation
> of this.  Can anyone do a better job?

The Customize system looks like a working real life example of a form,
but it does not prevent moving the point outside the fields. Does that
mean the original problem 'intangible' was intended to solve is
already solved in a better way?

Re: Can we make set_point_both less expensive?

[Eli Zaretskii]

> From: Yuri Khan <yuri.v.khan@gmail.com>
> Date: Tue, 17 Mar 2015 21:15:46 +0600
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>, Eli Zaretskii <eliz@gnu.org>, 
> 	Daniel Colascione <dancol@dancol.org>, Emacs developers <emacs-devel@gnu.org>
> 
> On Tue, Mar 17, 2015 at 8:29 PM, Richard Stallman <rms@gnu.org> wrote:
> 
> > The reason I implemented 'intangible' was for the sake of editing
> > forms.  A form has fields to be edited, and fixed text that shouldn't
> > be changed.  But there is no need to put point within the fixed text;
> > allowing that seems ugly.
> 
> There are in fact plenty of reasons to be able to move point all over
> the form.

You can accomplish that by let-binding inhibit-point-motion-hooks.

Re: Can we make set_point_both less expensive?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > There are in fact plenty of reasons to be able to move point all over
  > the form. Off the top of my head:

  > * The user might want to mark and copy some part of the form text.
  > * The user might want to look up some words of the form text in a dictionary.
  > * The user might want parts of the form text spoken aloud or displayed
  > on a Braille terminal.

Some people occasionally want to do these things, but most of the time
it feels strange that point can move into the fixed text.  In all the
other form filling programs the cursor always in some field.

  > * Moving the point only through the form fields feels unpredictable;
  > the user feels out of control.

On the contrary, that is precisely what all users are accustomed to.
Seeing point move into the labels is what will feel out of control.

  > The Customize system looks like a working real life example of a form,
  > but it does not prevent moving the point outside the fields. Does that
  > mean the original problem 'intangible' was intended to solve is
  > already solved in a better way?

Users seem not to be very happy with Customize.
Perhaps this is one of the reasons of why.

Here's an idea.  There should be a minor mode which controls whether point
can move into the fixed text between the forms, which is enabled by default
when you're filling out a form, but you can turn it off if you want to
do one of those unusual things.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! See stallman.org/skype.html.