I can see invisible characters

I can see invisible characters

[Werner LEMBERG]

[-- Attachment #1: Type: Text/Plain, Size: 640 bytes --]

[Emacs CVS 2009-05-13]

0. Start `emacs -Q'.

1. Do `M-x load-file line-invisible.el' (this assigns
   `make-lines-invisible' to `C-c z').

2. Load file `w' with `C-x C-f w'.

3. Do `C-c z ·' (this is, make all lines invisible which contain a
   middle dot).

4. Do `C-u C-x =': The first visible character in the buffer has set
   the `invisible' property.  It seems to be a bug that I can see that
   character.  It has also set the `intangible' property, and I can
   move the cursor on top of it (with the arrow keys, but not with
   C-a).  This doesn't look right too...

   An off-by-one error?


     Werner


[-- Attachment #2: w --]
[-- Type: Application/Octet-Stream, Size: 390 bytes --]

Aachen;Aa·chen
Aachener;Aa·che·ner
Aachenerin;Aa·che·ne·rin
Aachenern;Aa·che·nern
Aacheners;Aa·che·ners
Aachens;Aa·chens
Aadorf;Aa·dorf
Aalbauer;Aal=bau-er
Aalbeck;Aal·beck
Aalbestand;Aal=be-stand
Aalbestände;Aal=be-stän-de
Aalborg;Aal-borg
Aalborger;Aal=bor-ger
Aalders;Aal·ders
Aale;Aa-le
Aalen;Aa-len
Aalener;Aa-le-ner
Aalenern;Aa-le-nern
Aaleners;Aa-le-ners
Aalens;Aa-lens
Aaler;Aa-ler

[-- Attachment #3: line-invisible.el --]
[-- Type: Text/Plain, Size: 3818 bytes --]

;;; line-invisible.el
;;;
;;; Werner Lemberg, April 2009


;; from font-lock.el
(defmacro save-buffer-state (varlist &rest body)
  "Bind variables according to VARLIST and eval BODY restoring
buffer state."
  (declare (indent 1) (debug let))
  (let ((modified (make-symbol "modified")))
    `(let* ,(append varlist
		    `((,modified (buffer-modified-p))
		      (buffer-undo-list t)
		      (inhibit-read-only t)
		      (inhibit-point-motion-hooks t)
		      (inhibit-modification-hooks t)
		      deactivate-mark
		      buffer-file-name
		      buffer-file-truename))
       (unwind-protect
	   (progn
	     ,@body)
	 (unless ,modified
	   (restore-buffer-modified-p nil))))))

(defun make-lines-invisible (regexp &optional arg)
  "Make all lines matching a regexp invisible and intangible.
With a prefix arg, make them visible again.  It is not necessary
that REGEXP matches the whole line; if a hit is found, the
affected line gets automatically selected.

This function always applies to the whole buffer.

Note that this function modifies the `invisible' and `intangible'
text properties; it may thus interfere with modes which use them.
Due to implementation restrictions in current Emacs versions it
is not possible to use overlays -- which would avoid text
property modifications -- without becoming unbearably slow for
large buffers with many matches."
  (interactive "MLine matching regexp: \nP")
  (save-excursion
    (cond
     (arg
      (save-buffer-state ((next-pos (point-min)))
	(while (setq next-pos
		     (text-property-any next-pos
					(point-max)
					'make-lines-invisible
					t))
	  (goto-char next-pos)
	  (setq next-pos (or (next-single-property-change
			      (point)
			      'make-lines-invisible)
			     (point-max)))
	  (remove-list-of-text-properties (point)
					  next-pos
					  '(make-lines-invisible
					    invisible
					    intangible)))))
    (t
     (save-buffer-state ((start 0)
			 (end 0))
       (goto-char (point-min))
       (while (re-search-forward regexp nil t)
	 (setq start (match-beginning 0))
	 (goto-char start)
	 (setq start (line-beginning-position))
	 (setq end (match-end 0))
	 (goto-char end)
	 (setq end (1+ (line-end-position))) ; handle \n
	 (add-text-properties start
			      end
			      '(make-lines-invisible t
				invisible t
				intangible t))
	 (goto-char (line-end-position))))))))

(defun make-other-lines-invisible (regexp)
  "Make all lines not matching a regexp invisible and intangible.
It is not necessary that REGEXP matches the whole line; if a hit
is found, the affected line gets automatically unselected.

This function always applies to the whole buffer.

Note that this function modifies the `invisible' and `intangible'
text properties; it may thus interfere with modes which use them.
Due to implementation restrictions in current Emacs versions it
is not possible to use overlays -- which would avoid text
property modifications -- without becoming unbearably slow for
large buffers with many matches."
  (interactive "MLine matching regexp: \n")
  (save-excursion
    (save-buffer-state ((start 1)
			(end 1))
      (goto-char (point-min))
      (while (re-search-forward regexp nil t)
	(setq end (match-beginning 0))
	(goto-char end)
	(setq end (1- (line-beginning-position))) ; handle \n
	(if (<= start end)
	    (add-text-properties start
				 end
				 '(make-lines-invisible t
				   invisible t
				   intangible t)))
	(setq start (match-end 0))
	(goto-char start)
	(setq start (line-end-position))
	(goto-char (line-end-position)))
      (setq end (point-max))
      (if (< start end)
	  (add-text-properties start
			       end
			       '(make-lines-invisible t
				 invisible t
				 intangible t))))))

(global-set-key "\C-cz" 'make-lines-invisible)
(global-set-key "\C-cZ" 'make-other-lines-invisible)

;; eof

Re: I can see invisible characters

[Stefan Monnier]

> 4. Do `C-u C-x =': The first visible character in the buffer has set
>    the `invisible' property.  It seems to be a bug that I can see that
>    character.

I don't know about you, but I can't see it: it's not displayed.

>    It has also set the `intangible' property, and I can
>    move the cursor on top of it (with the arrow keys, but not with
>    C-a).  This doesn't look right too...

>    An off-by-one error?

I think that BOB and EOB are exceptions.
But note that if you set the front-sticky property, it's more difficult
to get to BOB.


        Stefan

Re: I can see invisible characters

[Werner LEMBERG]

[-- Attachment #1: Type: Text/Plain, Size: 558 bytes --]


>> 4. Do `C-u C-x =': The first visible character in the buffer has
>>    set the `invisible' property.  It seems to be a bug that I can
>>    see that character.
>
> I don't know about you, but I can't see it: it's not displayed.

Attached is an image which shows what I get.

>>    It has also set the `intangible' property, and I can move the
>>    cursor on top of it (with the arrow keys, but not with C-a).
>>    This doesn't look right too...
>>
>>    An off-by-one error?
>
> I think that BOB and EOB are exceptions.

Documented where?


    Werner

[-- Attachment #2: emacs-invisible.png --]
[-- Type: Image/Png, Size: 130969 bytes --]

Re: I can see invisible characters

[Stefan Monnier]

>>> 4. Do `C-u C-x =': The first visible character in the buffer has
>>> set the `invisible' property.  It seems to be a bug that I can
>>> see that character.
>> I don't know about you, but I can't see it: it's not displayed.
> Attached is an image which shows what I get.

I see the same thing: the char is not displayed.

>>> It has also set the `intangible' property, and I can move the
>>> cursor on top of it (with the arrow keys, but not with C-a).
>>> This doesn't look right too...
>>> An off-by-one error?
>> I think that BOB and EOB are exceptions.
> Documented where?

Nowhere.  But note that the relevant info is the stickiness: if 3 chars
are marked as `intangible', point cannot be in the middle of them, and
depending on the front&rear-stickiness of the `intangible' property
point may or may not be allowed at either of the ends.
In your case, the property is not front-sticky, so point can be at
beginning of the intangible span, i.e. at BOB.  This is actually not an
exceptional case and should behave identically if it's not at BOB.


        Stefan

Re: I can see invisible characters

[grischka]

Stefan Monnier wrote:
 > In your case, the property is not front-sticky, so point can be at
 > beginning of the intangible span, i.e. at BOB.

But note that even both "front sticky" and "intangible" set on
the topmost area don't prevent point from going to BOB.

--- grischka

Re: I can see invisible characters

[Stefan Monnier]

>> In your case, the property is not front-sticky, so point can be at
>> beginning of the intangible span, i.e. at BOB.
> But note that even both "front sticky" and "intangible" set on
> the topmost area don't prevent point from going to BOB.

If the span is rear-sticky as well, that's perfectly normal, I think.
If the span is rear-nonsticky, I guess this may also be the case and
would then be a BOB exception.


        Stefan

Re: I can see invisible characters

[Werner LEMBERG]

>>>> 4. Do `C-u C-x =': The first visible character in the buffer has
>>>> set the `invisible' property.  It seems to be a bug that I can
>>>> see that character.
>>>
>>> I don't know about you, but I can't see it: it's not displayed.
>>
>> Attached is an image which shows what I get.
>
> I see the same thing: the char is not displayed.

I'm confused.  The first lines are

  1  Aachen;Aa·chen
 16  Aachener;Aa·che·ner
 36  Aachenerin;Aa·che·ne·rin
 61  Aachenern;Aa·che·nern
 83  Aacheners;Aa·che·ners
105  Aachens;Aa·chens
122  Aadorf;Aa·dorf
137  Aalbauer;Aal=bau-er

My code calls `add-text-properties' with the following start and end
values:

   1  16
  16  36
  ...
 122 137

As you can see, the positions overlap.  If I don't do this, I get
empty lines.  However, I really don't understand the whole issue!  The
last character in the first line has position 14, thus the end-of-line
character has position 15 -- why do I need to use position 16 to make
the first line completely disappear?

Note that, according to the info of `C-x =', I can't access position
138, the first visible character after applying my code.  This *can't*
be right IMHO.


    Werner

Re: I can see invisible characters

[Stefan Monnier]

>> I see the same thing: the char is not displayed.

> I'm confused.  The first lines are

>   1  Aachen;Aa·chen
>  16  Aachener;Aa·che·ner
>  36  Aachenerin;Aa·che·ne·rin
>  61  Aachenern;Aa·che·nern
>  83  Aacheners;Aa·che·ners
> 105  Aachens;Aa·chens
> 122  Aadorf;Aa·dorf
> 137  Aalbauer;Aal=bau-er

> My code calls `add-text-properties' with the following start and end
> values:

>    1  16
>   16  36
>   ...
>  122 137

> As you can see, the positions overlap.

The positions are *between* chars, so the positions themselves have
a width of 0, so the interval 1-16 and 16-36 do not overlap.

> If I don't do this, I get empty lines.

Of course, because otherwise they don't cover the newlines.

> However, I really don't understand the whole issue!  The
> last character in the first line has position 14,

14 is the position before the last char.

> thus the end-of-line character has position 15

Indeed, 15 is the position after the last char, aka before the newline.

> -- why do I need to use position 16 to make
> the first line completely disappear?

Because 15-16 is the interval that covers the newline.

> Note that, according to the info of `C-x =', I can't access position
> 138, the first visible character after applying my code.  This *can't*
> be right IMHO.

That doesn't sound right: the first visible position is the char on
interval 137-138.  So you should not be able to access position 137
(because the invisible&intangible interval 1-137 is rear-sticky, so you
can place point at its beginning but not at its end), but you should be
able to go to position 138 (between the A and the a of Aalbauer).


        Stefan

Re: I can see invisible characters

[grischka]

Stefan Monnier wrote:
>>> In your case, the property is not front-sticky, so point can be at
>>> beginning of the intangible span, i.e. at BOB.
>> But note that even both "front sticky" and "intangible" set on
>> the topmost area don't prevent point from going to BOB.
> 
> If the span is rear-sticky as well, that's perfectly normal, I think.
> If the span is rear-nonsticky, I guess this may also be the case and
> would then be a BOB exception.

I'm pretty sure that front-sticky intangible doesn't work right.
And that not only in the exceptional but also in the normal case.

--- grischka

Re: I can see invisible characters

[Werner LEMBERG]

>> As you can see, the positions overlap.
> 
> The positions are *between* chars, so the positions themselves have
> a width of 0, so the interval 1-16 and 16-36 do not overlap.

OK.  Silly me.  I somehow got the impression that `point' and
`positions' are different things.

>> Note that, according to the info of `C-x =', I can't access
>> position 138, the first visible character after applying my code.
>> This *can't* be right IMHO.
> 
> That doesn't sound right: the first visible position is the char on
> interval 137-138.  So you should not be able to access position 137
> (because the invisible&intangible interval 1-137 is rear-sticky, so
> you can place point at its beginning but not at its end), but you
> should be able to go to position 138 (between the A and the a of
> Aalbauer).

This relieves me :-)  At least I'm only partially dumb.


    Werner

Re: I can see invisible characters

[Stefan Monnier]

> I'm pretty sure that front-sticky intangible doesn't work right.

Can you make a bug report with a recipe to reproduce the problem?


        Stefan

Re: I can see invisible characters

[Werner LEMBERG]

>> Note that, according to the info of `C-x =', I can't access
>> position 138, the first visible character after applying my code.
>> This *can't* be right IMHO.
> 
> That doesn't sound right:

Shall I make a bug report?


    Werner

Re: I can see invisible characters

[grischka]

Stefan Monnier wrote:
>> I'm pretty sure that front-sticky intangible doesn't work right.
> 
> Can you make a bug report with a recipe to reproduce the problem? 

Can you give a recipe to reproduce the normal behavior that you
described?

--- grischka