last character on line different when behind invis text?

last character on line different when behind invis text?

[Damien Elmes]

Hey folks,

I'm the new maintainer of emacs-wiki as of recently. Someone let me know about
some differing behavior which occurs in the current emacs CVS (I'm still on
21.3.50.1 here, and the same thing is happening).

The problem manifested itself in end-of-visible-line's behavior changing so
that it jumps over all lines ending with invisible text. When sitting at the
end of a normal line, hitting c-x = says that the character at point is a C-j.
This occurs on all emacs releases.

What appears to have changed in CVS is that c-x = on the end of a line with
invisible characters in it, actually reveals the last invisible character,
instead of the new line.

Was there any reason for this change? I can't find it in the NEWS file, and it
breaks emacs-wiki (since kill-line uses end-of-visible-line, killing an
[[ExtendedWikiLink]] actually keeps killing lines until it encounters a line
not ending with invisible text. The only way I can see around this at the
moment is adding a non-invisible character to the end of the line, which leads
me to think it's a bug - but it's quite possible i'm being naive.

Any thoughts would be appreciated!

Regards,

-- 
Damien Elmes

Re: last character on line different when behind invis text?

[Richard Stallman]

    What appears to have changed in CVS is that c-x = on the end of a line with
    invisible characters in it, actually reveals the last invisible character,
    instead of the new line.

Can you provide a precise test case?  It is not feasible to think
about what's going on here without one.

Re: last character on line different when behind invis text?

[Sacha Chua]

Richard Stallman <rms@gnu.org> writes:

> What appears to have changed in CVS is that c-x = on the end of a line with
> invisible characters in it, actually reveals the last invisible character,
> instead of the new line.
> 
> Can you provide a precise test case?  It is not feasible to think
> about what's going on here without one.

Related test case:

Killing a line that ends with invisible text results in killing the
next line as well, even if it has visible text that is clearly not
part of the first line. To wit:

After an emacs -q, eval this in a scratch buffer:

(insert-string
   (concat
      "Line 1 with " (propertize "invisible text." 'invisible \t 'intangible t)
    "\nLine 2 with " (propertize "invisible text." 'invisible t 'intangible t)
    "\nLine 3 with visible text.\n"))

Output:
Line 1 with 
Line 2 with 
Line 3 with visible text.

Position point on line 1 (say, on the n) and kill the line.

I expect output to be:
Li
Line 2 with
Line 3 with visible text.

But output is:
Li

Strange. Please explain behavior of end-of-visible-line? I've been
staring at this piece of code and I know what it does, but not why it
makes sense to skip over visible characters that are part of a different
line.

-- 
Sacha Chua <sacha@free.net.ph> - 4 BS CS Ateneo geekette
interests: emacs, gnu/linux, wearables, teaching compsci

Re: last character on line different when behind invis text?

[Damien Elmes]

Sacha Chua <sacha@free.net.ph> writes:

> (insert-string
>    (concat
>       "Line 1 with " (propertize "invisible text." 'invisible \t 'intangible t)
>     "\nLine 2 with " (propertize "invisible text." 'invisible t 'intangible t)
>     "\nLine 3 with visible text.\n"))

After that text is inserted into the buffer in CVS, you can also see that (c-x
=) doesn't identify the last char on a line as a C-j as in previous emacsen.

My guess is that end-of-visible-line is trying to skip over invisible
newlines? And because the newlines aren't appearing at the end of the line
anymore (even though they're not invisible), the semantics of its operation
have changed.

Regards,

-- 
Damien Elmes

Re: last character on line different when behind invis text?

[Damien Elmes]

Damien Elmes <emacs-devel@repose.cx> writes:

> Sacha Chua <sacha@free.net.ph> writes:
>
>> (insert-string
>>    (concat
>>       "Line 1 with " (propertize "invisible text." 'invisible \t 'intangible t)
>>     "\nLine 2 with " (propertize "invisible text." 'invisible t 'intangible t)
>>     "\nLine 3 with visible text.\n"))
>
> After that text is inserted into the buffer in CVS, you can also see that (c-x
> =) doesn't identify the last char on a line as a C-j as in previous emacsen.

Okay, I've dug through intervals.c in an effort to find out what's going on.
The change appears to be related to adjust_for_invis_intang() which Miles B
added on 14/3.

I've found that I can return to the old behavior of previous emacsen (ie, c-k
on the first line will only kill to the end of the line), by turning off
rear-sticky, as adjust_for_invis_intang checks to see if the text properties
of the invis+intang text are sticky or not. So a line like so works:

"Line 1 with " (propertize "invisible text." 'invisible t 'intangible t
rear-nonsticky t)

It looks like the code I inherited was using some magic which has broken with
the new way invis+intangible text is handled. I'll look into that and see if I
can figure out what's going on before bothering you guys again.

Regards,

-- 
Damien Elmes

Re: last character on line different when behind invis text?

[Richard Stallman]

    After that text is inserted into the buffer in CVS, you can also see that (c-x
    =) doesn't identify the last char on a line as a C-j as in previous emacsen.

This is actually a feature.  When text is invisible and intangible,
point can't be positioned after it, only before it.  This avoids the
anomalous behavior that there are two possible point values with apparently
no text between them.

It is true that some code may need to be changed to cope with the
feature.  I will fix C-k.

Re: last character on line different when behind invis text?

[Damien Elmes]

Richard Stallman <rms@gnu.org> writes:

>     After that text is inserted into the buffer in CVS, you can also see that (c-x
>     =) doesn't identify the last char on a line as a C-j as in previous emacsen.
>
> This is actually a feature.  When text is invisible and intangible,
> point can't be positioned after it, only before it.  This avoids the
> anomalous behavior that there are two possible point values with apparently
> no text between them.
>
> It is true that some code may need to be changed to cope with the
> feature.  I will fix C-k.

Thanks for putting in that fix, Richard. 

Is the following code a problem, however?

(insert-string
   (concat
      "\n\n"
      (propertize "invisible text." 'invisible t 'intangible t)
      "on a line"))

Eval the following code, then move to the blank line. Hit C-k. Instead of the
blank line being deleted, it is deleted along with the line with invisible
text below - infact all following lines that start with invis+intang text are
deleted.

Is this a feature? If it is, how can I work around it? I'd imagine this
behavior very confusing to users in most circumstances.

This seems to happen on both CVS and 21.2.

Regards,

-- 
Damien Elmes

Re: last character on line different when behind invis text?

[Richard Stallman]

Please try this replacement function.  The code does what the doc strings
says.  Whether that makes for correct behavior I am not sure.



(defun forward-visible-line (arg)
  "Move forward by ARG lines, ignoring currently invisible newlines only.
If ARG is negative, move backward -ARG lines.
If ARG is zero, move to the beginning of the current line."
  (condition-case nil
      (if (> arg 0)
	  (progn
	    (while (> arg 0)
	      (or (zerop (forward-line 1))
		  (signal 'end-of-buffer nil))
	      ;; If the newline we just skipped is invisible,
	      ;; don't count it.
	      (let ((prop
		     (get-char-property (1- (point)) 'invisible)))
		(if (if (eq buffer-invisibility-spec t)
			prop
		      (or (memq prop buffer-invisibility-spec)
			  (assq prop buffer-invisibility-spec)))
		    (setq arg (1+ arg))))
	      (setq arg (1- arg)))
	    ;; If invisible text follows, and it is a number of complete lines,
	    ;; skip it.
	    (let ((opoint (point)))
	      (while (and (not (eobp))
			  (let ((prop
				 (get-char-property (point) 'invisible)))
			    (if (eq buffer-invisibility-spec t)
				prop
			      (or (memq prop buffer-invisibility-spec)
				  (assq prop buffer-invisibility-spec)))))
		(goto-char
		 (if (get-text-property (point) 'invisible)
		     (or (next-single-property-change (point) 'invisible)
			 (point-max))
		   (next-overlay-change (point)))))
	      (unless (bolp)
		(goto-char opoint))))
	(let ((first t))
	  (while (or first (< arg 0))
	    (if (zerop arg)
		(beginning-of-line)
	      (or (zerop (forward-line -1))
		  (signal 'beginning-of-buffer nil)))
	    ;; If the newline we just moved to is invisible,
	    ;; don't count it.
	    (unless (bobp)
	      (let ((prop
		     (get-char-property (1- (point)) 'invisible)))
		(if (if (eq buffer-invisibility-spec t)
			prop
		      (or (memq prop buffer-invisibility-spec)
			  (assq prop buffer-invisibility-spec)))
		    (setq arg (1+ arg)))))
	    (setq first nil)
	    (setq arg (1+ arg)))
	  ;; If invisible text follows, and it is a number of complete lines,
	  ;; skip it.
	  (let ((opoint (point)))
	    (while (and (not (bobp))
			(let ((prop
			       (get-char-property (1- (point)) 'invisible)))
			  (if (eq buffer-invisibility-spec t)
			      prop
			    (or (memq prop buffer-invisibility-spec)
				(assq prop buffer-invisibility-spec)))))
	      (goto-char
	       (if (get-text-property (1- (point)) 'invisible)
		   (or (previous-single-property-change (point) 'invisible)
		       (point-min))
		 (previous-overlay-change (point)))))
	    (unless (bolp)
	      (goto-char opoint)))))
    ((beginning-of-buffer end-of-buffer)
     nil)))

Re: last character on line different when behind invis text?

[Damien Elmes]

Richard Stallman <rms@gnu.org> writes:

> Please try this replacement function.  The code does what the doc strings
> says.  Whether that makes for correct behavior I am not sure.

This fixes the problem I mentioned! I had a look through the CVS logs for
forward-visible-line, but I couldn't see any change which may have caused this
behavior. The new behavior is certainly preferable - if there is any visible
text on the line, it shouldn't be skipped over.

Does anyone see a reason why this change shouldn't be commited?

Regards,

-- 
Damien Elmes