invisible text and point

invisible text and point

[Luc Teirlinck]

There has been a relatively recent change in the treatment of point
near invisible text.  That change can be very confusing to the user
and apparently also to the Emacs code.  I noticed several bugs which
seem to be related to the change in question.  I have the impression
that they all could be undone by reverting the change.

Do the following both in Emacs-21.3 and Emacs-21.3.50.

C-h i m cvs RETURN

Put point on the * of

* CVS commands::                CVS commands share some things

C-u C-x = shows that in Emacs-21.3 we are where we think we are, on
the *, but in 21.3.50, we are on an invisible -.  The consequences
are:

try m

In Emacs-21.3, our default is "CVS commands" (what else), but in
21.3.50, it is Special Files.

C-g.  Now try RETURN.

In Emacs-21.3, we go to the expected place, but in 21.3.50, we get an
error message.

Reverting the change would automatically get rid of these bugs.  It
would also be easy to fix these bugs without reverting the change.
However, I have the impression that if one went on a bug hunt one
could find tons of similar bugs.  For instance, with point in
invisible text, one could easily get extremely strange defaults for
C-h f, C-h v, M-x man and so on...

There are worse things going on which only happen in the presence of
invisible text and only in 21.3.50, which I strongly suspect (even
though I am not 100% sure) to be related to the same change.

Do, say C-u C-x =, which pops up a new window.

Move that window upward with mouse-1.  No problem until we hit that
same spot with the invisible text.  There Emacs goes in an infinite
loop, apparently in the C code.  C-g is no help.

Start a new Emacs.

C-h i g (bash)Redirections RETURN

Just hit the down arrow key.  No problem until we hit some invisible
text after "Redirecting Output".  Infinite loop. no response to C-g.

Try everything over, but first do:

M-: (setq buffer-invisibility-spec nil)

No infinite loops, no problems.
No infinite loops, no problems in 21.3 either, regardless of
buffer-invisibility-spec.

I realize that putting point inside that invisible text has some
advantages in terms of stickiness.  However, in user produced buffers,
a user sophisticated enough to be playing around with invisibility
properties is probably sophisticated enough to handle stickiness in
the way that is best for him/her.  The Emacs produced buffers we are
talking about are all read-only to begin with, but to avoid confusing
users who, for some reason make them writable, we could give
text-property-default-nonsticky a buffer-local value and make the
invisibility property non-sticky in those particular buffers.

All of this is, of course just a suggestion.  It is possible to fix
all above bugs without reverting the change.  In the case of m and
RETURN in info, that would actually be easy, fixing the infinite loops
might be more complex.  However, as already mentioned, I believe that
the above bugs are just a small tip of a much bigger iceberg and that
there must be plenty more bugs related to the change.  The change in
question does not appear in any released version of Emacs, only in
21.3.50, so only people using the CVS would have gotten a shot at
encountering these bugs.  I encountered three in a few hours.  That
probably is because I was using info extensively, which contains
invisible text.
  
Sincerely,

Luc.

Re: invisible text and point

[Luc Teirlinck]

>From my previous message:

   Start a new Emacs.

   C-h i g (bash)Redirections RETURN

   Just hit the down arrow key.  No problem until we hit some invisible
   text after "Redirecting Output".  Infinite loop. no response to C-g.

Sorry, I forgot something important.  To reproduce this particular
bug, you first have to do:
(setq scroll-conservatively 1000)

Sincerely,

Luc.

Re: invisible text and point

[Luc Teirlinck]

>From my earlier message:

   There are worse things going on which only happen in the presence of
   invisible text and only in 21.3.50, which I strongly suspect (even
   though I am not 100% sure) to be related to the same change.

   Do, say C-u C-x =, which pops up a new window.

   Move that window upward with mouse-1.  No problem until we hit that
   same spot with the invisible text.  There Emacs goes in an infinite
   loop, apparently in the C code.  C-g is no help.

This bug also requires (setq scroll-conservatively 1000).  Both
infinite loops depend on that. Again, sorry for forgetting to mention
that.  Also make sure that point in the info buffer is on that
particular "*".

Sincerely,

Luc.

Re: invisible text and point

[Luc Teirlinck]

The one thing I should have said at the very beginning but somehow
forgot:

Do:

emacs-21.3.50 -q --eval "(blink-cursor-mode 0)" \
 --eval "(setq scroll-conservatively 1000)" &

Then proceed as described.

Sincerely,

Luc.

Re: invisible text and point

[Luc Teirlinck]

In this thread I failed to remember what the exact pre-change
behavior was.  Here is the change mentioned in the NEWS:

    ** At the end of a command, point moves out from within invisible
    text, in the same way it moves out from within text covered by an
    image or composition property.

    This makes it generally unnecessary to mark invisible text as
    intangible.
    This is particularly good because the intangible property often has
    unexpected side-effects since the property applies to everything
    (including `goto-char', ...) whereas this new code is only run after
    post-command-hook and thus does not care about intermediate states.

Thus the change was not motivated by problems with stickiness, as I
originally thought.  I confused with the next NEWS entry, which I
remembered from something else.  I do not copy it entirely, since it
is long:

  ** Only one of the beginning or end of an invisible, intangible region
     is considered an acceptable value for point; which one is
     determined by examining how the invisible/intangible properties are
     inherited when new text is inserted adjacent to them.  (The
     `front-sticky' and `rear-sticky' properties control this.)

So there are three solutions:

Really revert to the emacs-21.3 behavior and use intangible properties
in these situations again, leave things as they are now and deal with
the bugs, or (what I really was suggesting yesterday, when I
misinterpreted the actual emacs-21.3 behavior) make the code that runs
after post-command-hook really mimic the old behavior by never placing
point inside an invisible region.  (And thus not confusing commands
like m and RETURN in info, C-h f, C-h v, M-x man and countless others
that act upon, or choose a default based on, the position of point.)
At first view, I might prefer the latter solution.

Sincerely,

Luc.

Re: invisible text and point

[Stefan Monnier]

> There has been a relatively recent change in the treatment of point
> near invisible text.  That change can be very confusing to the user
> and apparently also to the Emacs code.  I noticed several bugs which
> seem to be related to the change in question.  I have the impression
> that they all could be undone by reverting the change.

Well, yes, but there's a good reason for the change.
I'll look at the bugs and see how to fix them.


	Stefan

Re: invisible text and point

[Luc Teirlinck]

I posted a follow-up to my original message from yesterday evening
this morning.  I no longer propose "reverting" to the emacs-21.3
behavior (actually, I only did this because I misinterpreted it
yesterday, as I noticed this morning), but "mimicking it better", by
making sure that (point) returns the position of a visible character.

Since my conjecture that the infinite loops were also related to the
problem was apparently a premature conclusion based on the fact that
the bugs did not occur in 21.3 and happened at similar places as the
other bug, we are left with the m and RETURN bugs in info.  Just
fixing those individual bugs would not really seem to really address
the problem, which is more general.  Namely a huge fraction of Emacs
commands act based on the return value of (point).  Since point is not
where the user believes it is, this is bound to confuse the user.  I
do not know whether it would be feasible to make all interactive
commands act based on the "user-visible" displayed position of point
instead.  That would really eliminate the need for any change.

Stefan Monnier wrote:

   Point is not put inside the invisible text.  It's put at one of the
   two ends.  In this case it happens to be "just before" rather than
   "just after" because the stickiness is the default one so that inserting
   a char at the end would lead to a hidden char whereas inserting a char
   at the beginning would lead to a visible char: the choice is to prefer
   putting point at a position such that inserted text is visible.

   I believe that the `intangible' change mentioned underneath in the NEWS
   file did the same for intangible&invisible text.

Yes, but semantics aside, if (point) returns the position of an
invisible character, we have the kind of problems of which the m and
RETURN behaviors in info are symptomatic.

   Or rather make it "front-sticky" rather than "rear-sticky", such
   that when the user sees point in front of letter "a", `char-after'
   will indeed show letter ?a.  Right now, it's the opposite: if point
   is right after letter "a" `char-before' will indeed return ?a.

That would work for those buffers.  I would guess that there probably
are good reasons to keep invisible rear-sticky in general.  In that
case, we would be left with the problem of making sure that the user
would understand in which buffers invisibility is front-sticky and in
which it is rear-sticky.

Sincerely,

Luc.

Re: invisible text and point

[Stefan Monnier]

> I posted a follow-up to my original message from yesterday evening
> this morning.  I no longer propose "reverting" to the emacs-21.3
> behavior (actually, I only did this because I misinterpreted it
> yesterday, as I noticed this morning), but "mimicking it better", by
> making sure that (point) returns the position of a visible character.

Semantic detail (relevant in this context): `point' is always between
two characters, which is why there is no (char-of POS) but there are
(char-before POS) and (char-after POS).

> Since my conjecture that the infinite loops were also related to the
> problem was apparently a premature conclusion based on the fact that
> the bugs did not occur in 21.3 and happened at similar places as the
> other bug, we are left with the m and RETURN bugs in info.  Just

Yes, we need to address those.  Making invisible front-sticky and
not rear-sticky (in the info buffer) is easy enough.

I guess the question now is: should we give preference to the desire
to ensure that "self-insert-command will insert a visible character"
or to "`char-after' is what the user sees (i.e. is visible)".


	Stefan

Re: invisible text and point

[Luc Teirlinck]

Stefan Monnier wrote:

   Yes, we need to address those.  Making invisible front-sticky and
   not rear-sticky (in the info buffer) is easy enough.

Yes although, as you pointed out, that might in certain situations
produce non-intuitive results for commands that check char-before (or
both char-before and char-after) for what to do.  Seems that we can
not win either way.  The change would automatically fix the bugs with
m and RETURN in info, although it would, of course, also be possible
to fix these without any change in the treatment of invisible text.

To make things worse, the invisibility property is not the only text
property that appears to give problems of the "things are not always
what they really seem" type.  The display property can be just as bad.

Indeed, do C-h i again and put point near "* echo".  Suppose that for
some reason we might want to take a look at the man page for echo,
maybe to see how it differs from the info we just come to read (or
whatever).

M-x man RETURN.

Result:

Manual entry (default echo                     ):

Lots of useless whitespace at the end, but that is perfectly OK, one
can do M-x man echo and add tons of trailing whitespace, it gets
disregarded anyway.  So: RETURN.

Result:

error in process sentinel: Can't find the echo: manpage

In other words, in spite of not only what seemed to be in the buffer,
but even in spite of what the prompt seemed to say, the default was
not really "echo" but "echo:".  Who would have guessed?

Actually, the text on the line is:

echo: (sh-utils)echo invocation.              Print a line of text.

The user does not see the:

: (sh-utils)echo invocation.

But to play semantic games (that have some relevance), that text is
completely "visible" (line-move-invisible returns nil for all involved
positions and there are no invisibility properties around). It just
has the display property.

All of this would seem to be very confusing to the user.

Sincerely,

Luc.

Re: invisible text and point

[Luc Teirlinck]

Maybe extremely naive question, but what would be the harm in actually
showing all that invisible and visible-but-you-just-can-not-see-it
text to the user to begin with?  It would take care of all these
problems.  (Except for the infinite loops which are a different
problem and need to be addressed regardless.)  Another confusing thing
about the display property is that the involved string of text remains
impossible to see (to avoid the term invisible) when yanked inside
emacs, but all of a sudden appears out of nowhere when yanked into,
say, an xterm?  On the other hand the "really invisible" text becomes
also visible (by default) when yanked inside emacs, adding to the
confusion.

Some of the hidden text is actually informative.

Sincerely,

Luc.

Re: invisible text and point

[David Kastrup]

Luc Teirlinck <teirllm@dms.auburn.edu> writes:

> Stefan Monnier wrote:
> 
>    Yes, we need to address those.  Making invisible front-sticky and
>    not rear-sticky (in the info buffer) is easy enough.
> 
> Yes although, as you pointed out, that might in certain situations
> produce non-intuitive results for commands that check char-before
> (or both char-before and char-after) for what to do.

Of course.  The purpose of the invisible property is to actually have
the characters in question appear in the buffer.  If you wanted to
attach some information to text that would _not_ influence editing,
you would put it into text properties, for example.

> To make things worse, the invisibility property is not the only text
> property that appears to give problems of the "things are not always
> what they really seem" type.  The display property can be just as
> bad.

Again, the purpose of this property is to make things different from
what they really seem, so it is hardly surprising that it succeeds.

> echo: (sh-utils)echo invocation.              Print a line of text.
> 
> The user does not see the:
> 
> : (sh-utils)echo invocation.
> 
> But to play semantic games (that have some relevance), that text is
> completely "visible" (line-move-invisible returns nil for all involved
> positions and there are no invisibility properties around). It just
> has the display property.
> 
> All of this would seem to be very confusing to the user.

Which is the reason one would have to fix this right there in the code
at the place of usage.  One can't cook up semantics for invisible and
display that will be perfect in all situations, so one has to cater
for those situations explicitly where it turns out the chosen
alternative does not fit.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: invisible text and point

[Richard Stallman]

    C-u C-x = shows that in Emacs-21.3 we are where we think we are, on
    the *, but in 21.3.50, we are on an invisible -.

I think the problem is that point is before the invisible text
rather than on the first visible character where it ought to be.

    Move that window upward with mouse-1.  No problem until we hit that
    same spot with the invisible text.  There Emacs goes in an infinite
    loop, apparently in the C code.  C-g is no help.

Someone needs to debug it.  etc/DEBUG gives advice on how to debug
a case of looping usefully.