Todays exercise of sanity (or does "see" really match "not"?)

Todays exercise of sanity (or does "see" really match "not"?)

[Lennart Borgman]

Place yourself at the top of the node "Special Characters in Regular
Expressions" in Info. The press C-s for I-search and type "not" and press
C-s a couple of time. Does this stop on "see" for you?

Excuse me if you have already discussed something similar. I know I have
skipped quite a few messages about search. And I currently is not quite
uptodate with CVS.

Re: Todays exercise of sanity (or does "see" really match "not"?)

[Luc Teirlinck]

Lennart Borgman wrote:

   Place yourself at the top of the node "Special Characters in Regular
   Expressions" in Info. The press C-s for I-search and type "not" and press
   C-s a couple of time. Does this stop on "see" for you?

If you do `c' in an Info buffer, you copy the node name
`(elisp)Regexp Special' to the kill ring.  If you include that in your
message, it makes it easier to find for people.

I guess that you have Info-hide-note-references enabled.  The reason
why it stops on "see" is that the actual text in the buffer is:
(*note Character Sets::). and note matches "not".

Sincerely,

Luc.

Re: Todays exercise of sanity (or does "see" really match "not"?)

[Lennart Borgman]

----- Original Message ----- 
From: "Luc Teirlinck" <teirllm@dms.auburn.edu>

> If you do `c' in an Info buffer, you copy the node name
> `(elisp)Regexp Special' to the kill ring.  If you include that in your
> message, it makes it easier to find for people.
> 
> I guess that you have Info-hide-note-references enabled.  The reason
> why it stops on "see" is that the actual text in the buffer is:
> (*note Character Sets::). and note matches "not".

Thanks. I am feeling better.

Re: Todays exercise of sanity (or does "see" really match "not"?)

[Juri Linkov]

"Lennart Borgman" <lennart.borgman.073@student.lu.se> writes:
> From: "Luc Teirlinck" <teirllm@dms.auburn.edu>
>> I guess that you have Info-hide-note-references enabled.  The reason
>> why it stops on "see" is that the actual text in the buffer is:
>> (*note Character Sets::). and note matches "not".
>
> Thanks. I am feeling better.

The text "*note " has the `display' text property with the string "see "
which is displayed instead of the original text.  The function
`isearch-range-invisible' (which is used by Info search) doesn't pay
attention to the `display' property.  It processes only the
`invisible' property.  It works the same in all Emacs buffers with the
`display' property (in `image-mode', `w3m' mode, etc.) where it finds
the text not visible on the screen.

How to handle this situation is not clear.  Should isearch find
the displayed text as a contiguous part of the search space?

The simplest solution is to treat the `display' property like `invisible'
property and to skip the text displayed over the original text.

-- 
Juri Linkov
http://www.jurta.org/emacs/

Re: Todays exercise of sanity (or does "see" really match "not"?)

[David Kastrup]

Juri Linkov <juri@jurta.org> writes:

> "Lennart Borgman" <lennart.borgman.073@student.lu.se> writes:
>> From: "Luc Teirlinck" <teirllm@dms.auburn.edu>
>>> I guess that you have Info-hide-note-references enabled.  The reason
>>> why it stops on "see" is that the actual text in the buffer is:
>>> (*note Character Sets::). and note matches "not".
>>
>> Thanks. I am feeling better.
>
> The text "*note " has the `display' text property with the string
> "see " which is displayed instead of the original text.  The
> function `isearch-range-invisible' (which is used by Info search)
> doesn't pay attention to the `display' property.  It processes only
> the `invisible' property.  It works the same in all Emacs buffers
> with the `display' property (in `image-mode', `w3m' mode, etc.)
> where it finds the text not visible on the screen.
>
> How to handle this situation is not clear.  Should isearch find the
> displayed text as a contiguous part of the search space?
>
> The simplest solution is to treat the `display' property like
> `invisible' property and to skip the text displayed over the
> original text.

No, no, no.  preview-latex is one such application, and it is
essential that the text remains searchable (it will get autoopened if
point is inside of it).  It's overlays it uses instead of text
properties, but that does not change the underlying basics.  If we
want some text to be impervious to search, you can
a) make it a single character, and so point will never be inside
b) declare a special property that makes this character not match for
searches.

But that will also break things like ^.*$, so it would probably be
better to use some otherwise unused character and adorn it with proper
character class and syntax.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: Todays exercise of sanity (or does "see" really match "not"?)

[Richard Stallman]

    The simplest solution is to treat the `display' property like `invisible'
    property and to skip the text displayed over the original text.

I think that is the right thing to do.

Re: Todays exercise of sanity (or does "see" really match "not"?)

[David Kastrup]

Richard Stallman <rms@gnu.org> writes:

>     The simplest solution is to treat the `display' property like
>     `invisible' property and to skip the text displayed over the
>     original text.
>
> I think that is the right thing to do.

It would be wrong for preview-latex, probably one of the most
important applications extensively using the display property.  It is
very important that the underlying buffer text remains both searchable
and replaceable, and preview-latex uses isearch-hooks and similar to
make this as natural and convenient as possible.

It has been doing this for years, probably like all applications using
the display property on non-trivial texts.

Doing an incompatible interface change breaking such packages at the
current point of time would be misguided.  Even if it weren't, it is
wrong to do such a large change in the course of the 22.1 release.
You can't expect all package authors to have adapted to such an
incompatible change as fast as that.

Please don't.  I can't see any advantage of breaking working packages
right now for no currently apparent significant practical advantage.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: Todays exercise of sanity (or does "see" really match "not"?)

[Juri Linkov]

David Kastrup <dak@gnu.org> writes:
> Please don't.  I can't see any advantage of breaking working packages
> right now for no currently apparent significant practical advantage.

I agree that handling the `display' property in a way similar to the
`invisible' property could break existing packages.  The rules for
handling invisible text are quite complex and include invisibility
specifications in `buffer-invisibility-spec'.  Changing them to
support the `display' property would be too drastic change.

With the current code one of the following methods can be used
to make isearch to skip text under the `display' property:

1. Set both `invisible' and `display' properties.  It seems the
   display engine ignores the `invisible' property when the `display'
   property is present.  But isearch still respects it.

2. Override the default `isearch-range-invisible' function with a new
   proposed variable `isearch-success-function'.

In the patch below the second method is used.  The default
`Info-search-success-function' skips "*note" text under point.
Everyone who wants to include "*note" to search results can override
`isearch-success-function'.

There are also two new variables `isearch-message-prefix-add' and
`isearch-message-suffix-add' to allow adding arbitrary text
to the isearch message's prefix and suffix (e.g. "X-Node").

Lazy highlighting code ignores both `invisible' and `display'
properties, since I see no reason to lazy-highlight invisible text.

Index: lisp/isearch.el
===================================================================
RCS file: /cvsroot/emacs/emacs/lisp/isearch.el,v
retrieving revision 1.261
diff -u -r1.261 isearch.el
--- lisp/isearch.el	18 Mar 2005 09:59:31 -0000	1.261
+++ lisp/isearch.el	22 Mar 2005 17:20:14 -0000
@@ -165,6 +165,11 @@
   "Function to save a function restoring the mode-specific isearch state
 to the search status stack.")
 
+(defvar isearch-success-function nil
+  "Function to report whether the new search match is considered successful.
+The function has two arguments: the positions of start and end of text
+matched by search.")
+
 ;; Search ring.
 
 (defvar search-ring nil
@@ -445,6 +450,9 @@
 (defvar isearch-just-started nil)
 (defvar isearch-start-hscroll 0)	; hscroll when starting the search.
 
+(defvar isearch-message-prefix-add nil)
+(defvar isearch-message-suffix-add nil)
+
 ; case-fold-search while searching.
 ;   either nil, t, or 'yes.  'yes means the same as t except that mixed
 ;   case in the search string is ignored.
@@ -640,6 +648,8 @@
 	isearch-adjusted nil
 	isearch-yank-flag nil
 	isearch-error nil
+	isearch-message-prefix-add nil
+	isearch-message-suffix-add nil
 	isearch-slow-terminal-mode (and (<= baud-rate search-slow-speed)
 					(> (window-height)
 					   (* 4
@@ -1993,6 +2003,7 @@
 			      (< (point) isearch-opoint)))
 		       "over")
 		   (if isearch-wrapped "wrapped ")
+		   (or isearch-message-prefix-add "")
 		   (if isearch-word "word " "")
 		   (if isearch-regexp "regexp " "")
 		   (if nonincremental "search" "I-search")
@@ -2008,7 +2019,8 @@
   (concat (if c-q-hack "^Q" "")
 	  (if isearch-error
 	      (concat " [" isearch-error "]")
-	    "")))
+	    "")
+	  (or isearch-message-suffix-add "")))
 
 \f
 ;; Searching
@@ -2039,7 +2051,8 @@
       (setq isearch-case-fold-search
 	    (isearch-no-upper-case-p isearch-string isearch-regexp)))
   (condition-case lossage
-      (let ((inhibit-point-motion-hooks search-invisible)
+      (let ((inhibit-point-motion-hooks
+	     (and (not isearch-success-function) search-invisible))
 	    (inhibit-quit nil)
 	    (case-fold-search isearch-case-fold-search)
 	    (search-spaces-regexp search-whitespace-regexp)
@@ -2052,12 +2065,15 @@
 		 isearch-string nil t))
 	  ;; Clear RETRY unless we matched some invisible text
 	  ;; and we aren't supposed to do that.
-	  (if (or (eq search-invisible t)
-		  (not isearch-success)
+	  (if (or (not isearch-success)
 		  (bobp) (eobp)
 		  (= (match-beginning 0) (match-end 0))
-		  (not (isearch-range-invisible
-			(match-beginning 0) (match-end 0))))
+		  (if isearch-success-function
+		      (funcall isearch-success-function
+			       (match-beginning 0) (match-end 0))
+		    (or (eq search-invisible t)
+			(not (isearch-range-invisible
+			      (match-beginning 0) (match-end 0))))))
 	      (setq retry nil)))
 	(setq isearch-just-started nil)
 	(if isearch-success
@@ -2393,18 +2409,35 @@
 	(isearch-regexp isearch-lazy-highlight-regexp)
 	(search-spaces-regexp search-whitespace-regexp))
     (condition-case nil
-	(funcall (isearch-search-fun)
-		 isearch-lazy-highlight-last-string
-		 (if isearch-forward
-		     (min (or isearch-lazy-highlight-end-limit (point-max))
-			  (if isearch-lazy-highlight-wrapped
-			      isearch-lazy-highlight-start
-			    (window-end)))
-		   (max (or isearch-lazy-highlight-start-limit (point-min))
-			(if isearch-lazy-highlight-wrapped
-			    isearch-lazy-highlight-end
-			  (window-start))))
-		 t)
+	(let ((retry t)
+	      (success nil)
+	      (bound (if isearch-forward
+			 (min (or isearch-lazy-highlight-end-limit (point-max))
+			      (if isearch-lazy-highlight-wrapped
+				  isearch-lazy-highlight-start
+				(window-end)))
+		       (max (or isearch-lazy-highlight-start-limit (point-min))
+			    (if isearch-lazy-highlight-wrapped
+				isearch-lazy-highlight-end
+			      (window-start))))))
+	  (while retry
+	    (setq success
+		  (funcall (isearch-search-fun)
+			   isearch-lazy-highlight-last-string
+			   bound t))
+	    (if (or (not success)
+		    (eq (point) bound)
+		    (if isearch-success-function
+			(funcall isearch-success-function
+				 (match-beginning 0) (match-end 0))
+		      (and (text-property-any
+			    (match-beginning 0) (match-end 0)
+			    'invisible nil)
+			   (text-property-any
+			    (match-beginning 0) (match-end 0)
+			    'display nil))))
+		(setq retry nil)))
+	  success)
       (error nil))))
 
 (defun isearch-lazy-highlight-update ()

Index: lisp/info.el
===================================================================
RCS file: /cvsroot/emacs/emacs/lisp/info.el,v
retrieving revision 1.421
diff -u -r1.421 info.el
--- lisp/info.el	16 Mar 2005 07:32:43 -0000	1.421
+++ lisp/info.el	22 Mar 2005 17:22:26 -0000
@@ -1503,19 +1503,10 @@
 				(1- (point)))
 			      (point-max)))
 	  (while (and (not give-up)
-		      (save-match-data
-			(or (null found)
-			    (if backward
-				(isearch-range-invisible found beg-found)
-			      (isearch-range-invisible beg-found found))
-			    ;; Skip node header line
-			    (and (save-excursion (forward-line -1)
-						 (looking-at "\^_"))
-				 (forward-line (if backward -1 1)))
-			    ;; Skip Tag Table node
-			    (save-excursion
-			      (and (search-backward "\^_" nil t)
-				   (looking-at "\^_\nTag Table"))))))
+		      (or (null found)
+			  (not (if isearch-success-function
+				   (funcall isearch-success-function beg-found found)
+				 (Info-search-success-function beg-found found)))))
 	    (let ((search-spaces-regexp Info-search-whitespace-regexp))
 	      (if (if backward
 		      (re-search-backward regexp bound t)
@@ -1529,7 +1520,7 @@
 		 (not bound)
 		 (or give-up (and found (not (and (> found opoint-min)
 						  (< found opoint-max))))))
-	(signal 'search-failed (list regexp "initial node")))
+	(signal 'search-failed (list regexp)))
 
       ;; If no subfiles, give error now.
       (if give-up
@@ -1591,19 +1582,10 @@
 		(setq list (cdr list))
 		(setq give-up nil found nil)
 		(while (and (not give-up)
-			    (save-match-data
-			      (or (null found)
-				  (if backward
-				      (isearch-range-invisible found beg-found)
-				    (isearch-range-invisible beg-found found))
-				  ;; Skip node header line
-				  (and (save-excursion (forward-line -1)
-						       (looking-at "\^_"))
-				       (forward-line (if backward -1 1)))
-				  ;; Skip Tag Table node
-				  (save-excursion
-				    (and (search-backward "\^_" nil t)
-					 (looking-at "\^_\nTag Table"))))))
+			    (or (null found)
+				(not (if isearch-success-function
+					 (funcall isearch-success-function beg-found found)
+				       (Info-search-success-function beg-found found)))))
 		  (let ((search-spaces-regexp Info-search-whitespace-regexp))
 		    (if (if backward
 			    (re-search-backward regexp nil t)
@@ -1667,9 +1649,27 @@
 		      nil 'Info-search-history)))
   (Info-search regexp bound noerror count 'backward))
 
+(defun Info-search-success-function (beg-found found)
+  (save-match-data
+    (let ((backward (< found beg-found)))
+      (not
+       (or
+	;; Skip `*Note'
+	(Info-get-token (point) "\\*note[ \n\t]+" "\\(\\*note[ \n\t]+\\)")
+	;; Skip node header line
+	(and (save-excursion (forward-line -1)
+			     (looking-at "\^_"))
+	     (forward-line (if backward -1 1)))
+	;; Skip Tag Table node
+	(save-excursion
+	  (and (search-backward "\^_" nil t)
+	       (looking-at "\^_\nTag Table"))))))))
+
 (defun Info-isearch-search ()
   (if Info-isearch-search
       (lambda (string &optional bound noerror count)
+	(setq isearch-message-prefix-add
+	      (if Info-isearch-initial-node "X-node " ""))
 	(if isearch-word
 	    (Info-search (concat "\\b" (replace-regexp-in-string
 					"\\W+" "\\\\W+"

-- 
Juri Linkov
http://www.jurta.org/emacs/

Re: Todays exercise of sanity (or does "see" really match "not"?)

[David Kastrup]

Juri Linkov <juri@jurta.org> writes:

> David Kastrup <dak@gnu.org> writes:
>> Please don't.  I can't see any advantage of breaking working packages
>> right now for no currently apparent significant practical advantage.
>
> I agree that handling the `display' property in a way similar to the
> `invisible' property could break existing packages.  The rules for
> handling invisible text are quite complex and include invisibility
> specifications in `buffer-invisibility-spec'.  Changing them to
> support the `display' property would be too drastic change.
>
> With the current code one of the following methods can be used
> to make isearch to skip text under the `display' property:
>
> 1. Set both `invisible' and `display' properties.  It seems the
>    display engine ignores the `invisible' property when the `display'
>    property is present.  But isearch still respects it.

This is definitely the correct solution.  If one really wants to have
the display property itself invisible (for whatever obscure reason),
then one should set the invisibility properties on the display
property's string.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: Todays exercise of sanity (or does "see" really match "not"?)

[Richard Stallman]

    1. Set both `invisible' and `display' properties.  It seems the
       display engine ignores the `invisible' property when the `display'
       property is present.  But isearch still respects it.

I agree that that way is the best--if we decide that the current
behavior when invisible and display are both present is the correct
one.  Maybe it is the right behavior; I see no reason to think it
should be changed.  But it would be nice to be really sure.

Re: Todays exercise of sanity (or does "see" really match "not"?)

[Juri Linkov]

Richard Stallman <rms@gnu.org> writes:
>     1. Set both `invisible' and `display' properties.  It seems the
>        display engine ignores the `invisible' property when the `display'
>        property is present.  But isearch still respects it.
>
> I agree that that way is the best--if we decide that the current
> behavior when invisible and display are both present is the correct
> one.  Maybe it is the right behavior; I see no reason to think it
> should be changed.  But it would be nice to be really sure.

Even though adding the `invisible' property will allow isearch to skip
the text under the `display' property for Emacs packages where this is
desirable, it will not solve the original problem in Info reported
in the initial message of this thread.

Info-search operates on plain text Info files without text properties
set by Info-fontify.  So it should rely on pure syntactical elements
like `*Note ' tags.  This is implemented in my latest patch.

-- 
Juri Linkov
http://www.jurta.org/emacs/

Re: Todays exercise of sanity (or does "see" really match "not"?)

[Luc Teirlinck]

Juri Linkov wrote:

   Info-search operates on plain text Info files without text properties
   set by Info-fontify.  So it should rely on pure syntactical elements
   like `*Note ' tags.  This is implemented in my latest patch.

Just to make sure: did you check that your patch works correctly when
Info-hide-note-references is nil, where `not' _should_ match `*Note'?
Personally, I have not had the time to follow this thread completely,
but at first view, nothing in this seem like a bug in any search
function to me.  To me, it seems like an obvious bug in
Info-hide-note-references; it abuses the display property for editing
purposes.

Sincerely,

Luc.

Re: Todays exercise of sanity (or does "see" really match "not"?)

[Juri Linkov]

Luc Teirlinck <teirllm@dms.auburn.edu> writes:
> Just to make sure: did you check that your patch works correctly when
> Info-hide-note-references is nil, where `not' _should_ match `*Note'?
> Personally, I have not had the time to follow this thread completely,
> but at first view, nothing in this seem like a bug in any search
> function to me.  To me, it seems like an obvious bug in
> Info-hide-note-references; it abuses the display property for editing
> purposes.

It is easy to add a test for Info-hide-note-references, but I think it
would be more convenient to skip the `*Note ' tag even when it is visible.
Searching for "not" is too tiresome when search stops at every cross
reference.

-- 
Juri Linkov
http://www.jurta.org/emacs/

Re: Todays exercise of sanity (or does "see" really match "not"?)

[Luc Teirlinck]

Yuri Linkov wrote:

   It is easy to add a test for Info-hide-note-references, but I think it
   would be more convenient to skip the `*Note ' tag even when it is visible.
   Searching for "not" is too tiresome when search stops at every cross
   reference.

A search for `not' is going to stop at many places.  Searching should
be predictable.  Please do not start adding tons of exceptions and
"exceptions to exceptions" to searching by trying to double guess the
user.

A case-insensitive non-word search for `not' _should_ match `*Note'.

Sincerely,

Luc.

Re: Todays exercise of sanity (or does "see" really match "not"?)

[Juri Linkov]

>    It is easy to add a test for Info-hide-note-references, but I think it
>    would be more convenient to skip the `*Note ' tag even when it is visible.
>    Searching for "not" is too tiresome when search stops at every cross
>    reference.
>
> A search for `not' is going to stop at many places.  Searching should
> be predictable.  Please do not start adding tons of exceptions and
> "exceptions to exceptions" to searching by trying to double guess the
> user.
>
> A case-insensitive non-word search for `not' _should_ match `*Note'.

`*Note' is a special tag, not part of the text.  For instance,
Web browsers don't display tags, and don't allow them to interfere
while searching.  Some browsers provide two search modes: search for
text including link text, and separate mode to search only link text.
And nobody asks for an ability to search for HTML tags like `<a href='.
So I'm sure many people will like to skip `*Note' in Info too.
With my patch this is easily configurable with only one function
which can be redefined to match `*Note', or to skip it, or to search
only inside cross-reference text, or anything else.

Since Emacs Info browser is too controversial piece of software,
let's leave the current default with only one change to skip "see ".

-- 
Juri Linkov
http://www.jurta.org/emacs/

Re: Todays exercise of sanity (or does "see" really match "not"?)

[Stefan Monnier]

> `*Note' is a special tag, not part of the text.  For instance,
> Web browsers don't display tags, and don't allow them to interfere
> while searching.  Some browsers provide two search modes: search for
> text including link text, and separate mode to search only link text.
> And nobody asks for an ability to search for HTML tags like `<a href='.

The `<a href=' stuff is not displayed, so it's no surprise people don't
expect search to find those things.

> So I'm sure many people will like to skip `*Note' in Info too.

That's possible, but it won't surprise them if it matches them, so long as
they're visible.


        Stefan

Re: Todays exercise of sanity (or does "see" really match "not"?)

[Luc Teirlinck]

Yuri Linkov wrote:

   `*Note' is a special tag, not part of the text.

`*Note' _is_ a part of the text.  It occurs in the output produced by
`makeinfo --plaintext', which, since it is entirely plain text, does
not have _any_ special markers.  Anyway, rather than to argue over
terminology, the important thing is: the user sees it, search has to
find it.  Note that both `s' and `C-s' in standalone Info correctly
match `*Note'.  If `not' would not match `*Note' in Emacs Info with
Info-hide-note-references set to t, then this would be very confusing.
It would make people start to doubt the reliability of search in
Emacs.  (What _other_ stuff does search in Emacs ignore?)  It would be
an outright bug.

Sincerely,

Luc.

Re: Todays exercise of sanity (or does "see" really match "not"?)

[Luc Teirlinck]

Actually, the fact that search does not match `see' when
Info-hide-note-references is enabled is also confusing, unless one
knows the internals of Info-hide-note-references.  However, it would
be hard to do anything about that one.  As I already said, to me the
_real_ bug is that the Info-hide-note-references implementation abuses
the display property.  The point of setting
`Info-hide-note-references' to nil is to avoid all bugs that
inevitably come with a non-nil setting, including this one.

Sincerely,

Luc.

Re: Todays exercise of sanity (or does "see" really match "not"?)

[Kim F. Storm]

Luc Teirlinck <teirllm@dms.auburn.edu> writes:

> Actually, the fact that search does not match `see' when
> Info-hide-note-references is enabled is also confusing, unless one
> knows the internals of Info-hide-note-references.  However, it would
> be hard to do anything about that one.  As I already said, to me the
> _real_ bug is that the Info-hide-note-references implementation abuses
> the display property.  

I agree it needs fixing, but not that it is urgent.

I suggest you put it on the TODO list for after(!!) the release.

-- 
Kim F. Storm <storm@cua.dk> http://www.cua.dk

Re: Todays exercise of sanity (or does "see" really match "not"?)

[Luc Teirlinck]

Kim Storm wrote:

   Luc Teirlinck <teirllm@dms.auburn.edu> writes:

   > Actually, the fact that search does not match `see' when
   > Info-hide-note-references is enabled is also confusing, unless one
   > knows the internals of Info-hide-note-references.  However, it would
   > be hard to do anything about that one.  As I already said, to me the
   > _real_ bug is that the Info-hide-note-references implementation abuses
   > the display property.  

   I agree it needs fixing, but not that it is urgent.

I did not claim that it was urgent.  I claimed that it was confusing
and I objected against a similar confusion being regressively
introduced in the case where Info-hide-note-references was nil.

Sincerely,

Luc.

Re: Todays exercise of sanity (or does "see" really match "not"?)

[Richard Stallman]

    > Actually, the fact that search does not match `see' when
    > Info-hide-note-references is enabled is also confusing, unless one
    > knows the internals of Info-hide-note-references.

    I suggest you put it on the TODO list for after(!!) the release.

Please don't.  I think this would be very hard and is not the cost in
code complexity.

RE: Todays exercise of sanity (or does "see" really match "not"?)

[Drew Adams]

       > > `*Note' is a special tag, not part of the text.

    > `*Note' _is_ a part of the text.

There are obviously different ways of looking at this, and the issue is not
just one of terminology (what we decide to call "text" or "content").

Perhaps we can agree that some people and some programs will likely want to
treat such stuff as part of the "text/content/data" and others will want to
ignore it as part of the "text/content" and treat it instead as
"tags"/"metadata". It will always be true that one person's data is another
person's metadata.

The analogy with markup is useful, even if it is true that 1) the mechanism
used in Info is not exactly markup and 2) at least some of these "tags"
(e.g. "Note") are visible in the Info text.

Perhaps we can also agree that we should not try to change anything about
this now, at least not in any fundamental way.

And perhaps we can also agree that it might be good to look, later, at the
possibility of letting users/programs have their cake and eat it too: be
able to treat such "tags" as data or metadata, au choix.

Re: Todays exercise of sanity (or does "see" really match "not"?)

[Juri Linkov]

> `*Note' _is_ a part of the text.  It occurs in the output produced by
> `makeinfo --plaintext', which, since it is entirely plain text, does
> not have _any_ special markers.  Anyway, rather than to argue over
> terminology, the important thing is: the user sees it, search has to
> find it.  Note that both `s' and `C-s' in standalone Info correctly
> match `*Note'.  If `not' would not match `*Note' in Emacs Info with
> Info-hide-note-references set to t, then this would be very confusing.
                                ====
Did you mean "Info-hide-note-references set to nil"?

> It would make people start to doubt the reliability of search in
> Emacs.  (What _other_ stuff does search in Emacs ignore?)  It would be
> an outright bug.

Isearch ignores text with `invisible' text properties and hidden
overlays according to `buffer-invisibility-spec'.

Info-search ignores Info files headers, header lines and Tag Table
nodes.

> Sorry for misspelling your name once more in my previous reply.
> I do not know why I keep doing this.

I guess it's because you manually type a citation line every time
you write a response.  To insert it automatically you can use message.el
and set `message-citation-line-function' to `message-insert-citation-line'.

-- 
Juri Linkov
http://www.jurta.org/emacs/

Re: Todays exercise of sanity (or does "see" really match "not"?)

[Luc Teirlinck]

Juri Linkov wrote:

   Did you mean "Info-hide-note-references set to nil"?

Yes.

   Isearch ignores text with `invisible' text properties and hidden
   overlays according to `buffer-invisibility-spec'.

   Info-search ignores Info files headers, header lines and Tag Table
   nodes.

None of which apply to `*Note'.

   > It would make people start to doubt the reliability of search in
   > Emacs.  (What _other_ stuff does search in Emacs ignore?)  It would be
   > an outright bug.

What I meant with the parenthetical remark is: If Isearch starts
ignoring things like `*Note', then people could not safely rely on it
anymore, because they would always have to ask themselves: "What
_other_ stuff does search in Emacs ignore?".  There could always be
_some_ reason why Isearch decided that they were actually not
interested in finding what they were trying to find and that showing
them the match would only "inconvenience" them.

Sincerely,

Luc.

Re: Todays exercise of sanity (or does "see" really match "not"?)

[Richard Stallman]

      Anyway, rather than to argue over
    terminology, the important thing is: the user sees it, search has to
    find it.

I wouldn't put it quite so strongly, but this is basically right.
Search should find all sorts of markup that are in the text.

Please, everyone, consider this question settled and stop arguing!

Re: Todays exercise of sanity (or does "see" really match "not"?)

[Luc Teirlinck]

Sorry for misspelling your name once more in my previous reply.  I do
not know why I keep doing this.

Sincerely,

Luc.

Re: Todays exercise of sanity (or does "see" really match "not"?)

[Richard Stallman]

    A case-insensitive non-word search for `not' _should_ match `*Note'.

I agree.

Re: Todays exercise of sanity (or does "see" really match "not"?)

[Richard Stallman]

    Info-search operates on plain text Info files without text properties
    set by Info-fontify.

How does that happen?  Does Info-fontify sometimes fail to be called?

Anyway, when there are no such text properties, as far as I can see,
there is no problem and no need for any change.  Search will find
the text that's in the buffer, which is also what the user sees.
Is there something here I am missing?

Re: Todays exercise of sanity (or does "see" really match "not"?)

[Juri Linkov]

>     Info-search operates on plain text Info files without text properties
>     set by Info-fontify.
>
> How does that happen?  Does Info-fontify sometimes fail to be called?

Info-search searches the text on plain unfontified Info files (sometimes
fontification remains on parts of the Info file with visited nodes where
fontification was done, but search should work for general case with
unfontified Info files) and when it finds it narrows the Info file buffer
to the found node and calls `Info-fontify' on that node.

Fontifying the whole Info file before starting the search only for the
sake of putting text properties to help Info-search to find the text
that will be invisible after fontification is too costly operation.

Info-search should anticipate what text will be invisible (or will be
replaced by the `display' property).  To do this it should use rules
similar to rules used by `Info-fontify' and skip the text that
`Info-fontify' will make invisible.

-- 
Juri Linkov
http://www.jurta.org/emacs/

Re: Todays exercise of sanity (or does "see" really match "not"?)

[Richard Stallman]

    Info-search should anticipate what text will be invisible (or will be
    replaced by the `display' property).

It should fontify the text around the potential match and recheck it.
That would be cleaner, simpler, and more general.

Re: Todays exercise of sanity (or does "see" really match "not"?)

[Richard Stallman]

    It would be wrong for preview-latex, probably one of the most
    important applications extensively using the display property.

With all due respect, I have not been shown a reason to consider that
package tremendously important.  All user program are important to
some extent, but I don't see why this one should be overridingly so.

Re: Todays exercise of sanity (or does "see" really match "not"?)

[David Kastrup]

Richard Stallman <rms@gnu.org> writes:

>     It would be wrong for preview-latex, probably one of the most
>     important applications extensively using the display property.
>
> With all due respect, I have not been shown a reason to consider
> that package tremendously important.  All user program are important
> to some extent, but I don't see why this one should be overridingly
> so.

Oh, I was not as immodest to claim one of the most important
applications whatsoever, but only among a particular subset.  It
actually happens to be rather easy to be "one of the most important
applications [among those that are] _extensively_ using the display
property", since there are not many of them.  The only other one I can
think of right now would be w3, and development of that appears pretty
much stagnant.

Just ask Gerd what application produced the most bug reports in the
betatest of Emacs-21.1 concerning the display property...

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: Todays exercise of sanity (or does "see" really match "not"?)

[Kim F. Storm]

David Kastrup <dak@gnu.org> writes:

> Richard Stallman <rms@gnu.org> writes:
>
>>     It would be wrong for preview-latex, probably one of the most
>>     important applications extensively using the display property.
>>
>> With all due respect, I have not been shown a reason to consider
>> that package tremendously important.  All user program are important
>> to some extent, but I don't see why this one should be overridingly
>> so.
>
> Oh, I was not as immodest to claim one of the most important
> applications whatsoever, but only among a particular subset.  It
> actually happens to be rather easy to be "one of the most important
> applications [among those that are] _extensively_ using the display
> property", since there are not many of them.  The only other one I can
> think of right now would be w3, and development of that appears pretty
> much stagnant.

I fully agree with David that we should not make such changes to the
behaviour of the display property that would break _the_ major package
actually using it for anything non-trivial.

Instead we can explicitly put an inivisible property together with the
display property on a priece of "hidden" text which should be ignored
by searches.  That need a little work to check all uses of display
properties, but it doesn't break any old code relying on the current
behaviour.

I don't know whether it actually works to put both invisible and
display on some text (with the effect of showing the image and
ignoring the under-laying text).  Would someone make some examples
to check it?

-- 
Kim F. Storm <storm@cua.dk> http://www.cua.dk

Re: Todays exercise of sanity (or does "see" really match "not"?)

[David Kastrup]

[-- Attachment #1: Type: text/plain, Size: 1673 bytes --]

storm@cua.dk (Kim F. Storm) writes:

> Instead we can explicitly put an inivisible property together with
> the display property on a priece of "hidden" text which should be
> ignored by searches.  That need a little work to check all uses of
> display properties, but it doesn't break any old code relying on the
> current behaviour.
>
> I don't know whether it actually works to put both invisible and
> display on some text (with the effect of showing the image and
> ignoring the under-laying text).

It did in the past.  I remember that effect from when we were
desperately trying to find a combination of text properties that would
work with both Emacs and XEmacs.

> Would someone make some examples to check it?

I just patched up preview-latex to place the invisible property on the
overlays it creates (obviously, not the desired default behavior).
Now this is using overlays, not text properties, but the results
should be pretty much the same.

And the results are that stuff is not found indeed when using isearch.
query-replace-regexp, however, still finds it.  Non-interactive search
doesn't find it.

I also remember that in the Emacs-21.1 code base, placing the
"invisible" property on displayed stuff had the effect that scrolling
did no longer account for the amount of vertical space taken by the
image.  However, with the current HEAD, this effect is not to be seen.

In case that you want to experiment with it, here is the patch I
used.  Unfortunately, the main work is done in a macro, so you need
something like

(let ((load-path (cons "." load-path))) (load "preview.el"))

to let it take effect unless you recompile and reinstall
preview-latex.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 937 bytes --]

--- prv-emacs.el	18 Mar 2005 22:36:24 +0100	1.63
+++ prv-emacs.el	23 Mar 2005 11:48:08 +0100	
@@ -168,6 +168,7 @@
 	  `(propertize
 	    "x"
 	    'display ,glyph
+	    'invisible t
 	    'mouse-face 'highlight
 	    'help-echo
 	    ,(if (stringp helpstring)
@@ -347,12 +348,12 @@
 	    (overlay-put ov 'category 'preview-overlay)
 	    (if (eq (overlay-start ov) (overlay-end ov))
 		(overlay-put ov 'before-string (car strings))
-	      (dolist (prop '(display keymap mouse-face help-echo))
+	      (dolist (prop '(invisible display keymap mouse-face help-echo))
 		(overlay-put ov prop
 			     (get-text-property 0 prop (car strings))))
 	      (overlay-put ov 'before-string nil))
 	    (overlay-put ov 'face nil))
-	(dolist (prop '(display keymap mouse-face help-echo))
+	(dolist (prop '(invisible display keymap mouse-face help-echo))
 	  (overlay-put ov prop nil))
 	(overlay-put ov 'face 'preview-face)
 	(unless (cdr strings)

[-- Attachment #3: Type: text/plain, Size: 52 bytes --]



-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

[-- Attachment #4: Type: text/plain, Size: 142 bytes --]

_______________________________________________
Emacs-devel mailing list
Emacs-devel@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-devel

Re: Todays exercise of sanity (or does "see" really match "not"?)

[Kim F. Storm]

David Kastrup <dak@gnu.org> writes:

> I just patched up preview-latex to place the invisible property on the
> overlays it creates (obviously, not the desired default behavior).
> Now this is using overlays, not text properties, but the results
> should be pretty much the same.
>
> And the results are that stuff is not found indeed when using isearch.
> query-replace-regexp, however, still finds it.  Non-interactive search
> doesn't find it.

I think that's ok.

Would someone write some advise to put into the lispref section on
the display property.

-- 
Kim F. Storm <storm@cua.dk> http://www.cua.dk

Re: Todays exercise of sanity (or does "see" really match "not"?)

[Kim F. Storm]

David Kastrup <dak@gnu.org> writes:

In my previous mail I intended to say:

> And the results are that stuff is not found indeed when using isearch.

Good!

> query-replace-regexp, however, still finds it.

Seems like a general problem with invisible text and replace.
Should it be fixed?

>  Non-interactive search doesn't find it.

That's ok I think.

-- 
Kim F. Storm <storm@cua.dk> http://www.cua.dk

Re: Todays exercise of sanity (or does "see" really match "not"?)

[David Kastrup]

storm@cua.dk (Kim F. Storm) writes:

> David Kastrup <dak@gnu.org> writes:
>
> In my previous mail I intended to say:
>
>> And the results are that stuff is not found indeed when using
>> isearch.
>
> Good!
>
>> query-replace-regexp, however, still finds it.
>
> Seems like a general problem with invisible text and replace.
> Should it be fixed?

I think that all user-level search-type commands should behave
consistently.  query-replace (and its cousins) are implemented quite
differently from search and isearch.  Fixing this seems desirable in
the long run, but completely infeasible now.

Getting this consistent is a _lot_ of work.  For example, commands
like zap-to-char, forward-sexp and so on also are user-level commands,
as are fill-region and indent: how to deal with invisibility there
(actually a topic that AUCTeX&preview-latex has mostly evaded
addressing properly up to now): does it make sense to tamper with
material that actually cannot contribute to the screen effect the
commands are supposed to achieve?

It would appear sensible to draw the line somewhere.  The current line
between search on the one hand and search&replace on the other seems
particularly arbitrary, but I don't see that we should even think
about it before the release.

>>  Non-interactive search doesn't find it.
>
> That's ok I think.

Yes (I am not talking about the Lisp-level command, but about C-s RET).

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: Todays exercise of sanity (or does "see" really match "not"?)

[Richard Stallman]

    > query-replace-regexp, however, still finds it.

    Seems like a general problem with invisible text and replace.
    Should it be fixed?

I think this would open a can of worms and we should leave it alone,
at least for now.

Re: Todays exercise of sanity (or does "see" really match "not"?)

[Juri Linkov]

storm@cua.dk (Kim F. Storm) writes:
> Instead we can explicitly put an inivisible property together with the
> display property on a priece of "hidden" text which should be ignored
> by searches.  That need a little work to check all uses of display
> properties, but it doesn't break any old code relying on the current
> behaviour.

Yes, there are existing packages where finding the underlying text
might be useful.  For example, in w3m images are placed over the text
extracted from the ALT="" image HTML attribute.  Finding it with isearch
seems useful.

> I don't know whether it actually works to put both invisible and
> display on some text (with the effect of showing the image and
> ignoring the under-laying text).  Would someone make some examples
> to check it?

The `display' property takes precedence over the `invisible' property:

(progn
  (goto-char (point-min))
  (insert "abcdefghi\n")
  (add-text-properties (- (point) 9) (- (point) 2)
   `(display ,(create-image (expand-file-name "gnu.xpm" data-directory) 'xpm)
     invisible t))
  (goto-char (point-min))
  (search-forward "def"))

And the image is visible.  Incremental search skips the underlying text,
but low-level search functions find it.

-- 
Juri Linkov
http://www.jurta.org/emacs/

RE: Todays exercise of sanity (or does "see" really match "not"?)

[Drew Adams]

Just a reminder, I guess, with another example of why this should eventually
be fixed.

I'm coming back to this thread from 2005/03/21. I think we all agreed to
worry about this later, and that 1) programs should somehow be able to
search for and see metadata (such as Note: and Menu: tags), but that 2)
users searching Info should not find such tags (they can always find them
using the appropriate Info commands, such as Info-next-reference).

I was just trying to find information on menus in the Elisp manual, and I
was reminded about how important #2 is above. The information on menus is in
several different places in the manual. It is impractical to search for
"menu", however, because you hit all of the Menu: tags. It would really be
helpful to have this fixed somehow, after the release.

Re: Todays exercise of sanity (or does "see" really match "not"?)

[Eli Zaretskii]

> From: "Drew Adams" <drew.adams@oracle.com>
> Date: Tue, 9 Aug 2005 17:19:33 -0700
> 
> I was just trying to find information on menus in the Elisp manual, and I
> was reminded about how important #2 is above. The information on menus is in
> several different places in the manual. It is impractical to search for
> "menu", however, because you hit all of the Menu: tags.

You shouldn't search with the `s' command, except as the last resort.
You should search with the `i' command, which does not have this
deficiency.

RE: Todays exercise of sanity (or does "see" really match "not"?)

[Drew Adams]

    You shouldn't search with the `s' command, except as the last resort.
    You should search with the `i' command, which does not have this
    deficiency.

That helps me find many (but not all) occurrences of "menu". Thanks.

It would still be useful to fix the deficiency in `s' search, however.

Re: Todays exercise of sanity (or does "see" really match "not"?)

[Juri Linkov]

> Just a reminder, I guess, with another example of why this should
> eventually be fixed.
>
> I'm coming back to this thread from 2005/03/21. I think we all agreed to
> worry about this later, and that 1) programs should somehow be able to
> search for and see metadata (such as Note: and Menu: tags), but that 2)
> users searching Info should not find such tags (they can always find them
> using the appropriate Info commands, such as Info-next-reference).

I have a small uninstalled patch that fixes this problem trivially.

Is it ok to install it?

-- 
Juri Linkov
http://www.jurta.org/emacs/

RE: Todays exercise of sanity (or does "see" really match "not"?)

[Drew Adams]

    I guess that you have Info-hide-note-references enabled.  The reason
    why it stops on "see" is that the actual text in the buffer is:
    (*note Character Sets::). and note matches "not".

I filed a bug report on this behavior on Oct 4. Juri fixed it on Oct 31 with
respect to stuff in the header (File, Next etc.), but there was no fix AFAIK
for Note and Menu.

In discussion of the bug report, I stated that "we should be able somehow to
separate _content_ in a manual from _structural and navigational artifacts_
that support that content."

I still think that should be the goal: users should use TAB to go to the
next cross reference, instead of searching for "Note". The text "Note" for a
cross reference is not content; it is metacontent. If a user searches for
"Note", he should find only occurrences of the actual text "Note" in the
manual, not cross references.

In practice, however, it might not be that easy to distinguish content from
meta-information in the case of meta-info that is not in the header. At
least that's what I understood from the discussion at the time.

Here is part of Juri's explanation from that discussion:

    `Info-search' searches for a string on raw text of Info files.
    Some parts of Info files where Info nodes were already visited are
    fontified and have `invisible' property.  But fontifying and adding
    `invisible' or other properties to hide the tags from search
    to the whole Info file before search is too inefficient, so
    `Info-search' can't rely on text properties.  Instead of that,
    it should rely on the known syntax of Info files such as that Info nodes
    begin with a ^_ character.  I think skipping the header is sufficient
    to ignore irrelevant parts of Info nodes, and I see no problem with
    other special Info tags like Note and Menu.

In reply to David's point about preview-latex needing to search and find the
meta-text: I agree that programs need to be able to find meta-text. The
point is to separate such meta-text from the content - or at least be _able_
to separate/distinguish it. In most modes, at least, Info-search should not
find strings that are in meta-text (if this is not to difficult to fix).

Re: Todays exercise of sanity (or does "see" really match "not"?)

[Miles Bader]

On Mon, 21 Mar 2005 15:20:34 -0800, Drew Adams <drew.adams@oracle.com> wrote:
> In reply to David's point about preview-latex needing to search and find the
> meta-text: I agree that programs need to be able to find meta-text. The
> point is to separate such meta-text from the content - or at least be _able_
> to separate/distinguish it. In most modes, at least, Info-search should not
> find strings that are in meta-text (if this is not to difficult to fix).

How about a text/overlay property that says what text to use for
i-searching instead of the underlying text?  Then the mode itself (or
whoever added the corresponding display/invisible property) could make
exactly the right decision -- e.g. David would add the underlying
text, but info mode could add the text shown by the display property.

Something like:

   (overlay-put ov 'isearch-text "This is the search text")
   (overlay-put ov 'display (some-bit-map))

[I assume an implementation of this would be rather inefficient,
because the text wouldn't be in the buffer and the regexp search
primitives work directly on the buffer-gap data structures -- but I
assume these cases are relatively rare, and doesn't the
isearch-invisible hack have to deal with similar cases anyway?]

-Miles
-- 
Do not taunt Happy Fun Ball.

Re: Todays exercise of sanity (or does "see" really match "not"?)

[David Kastrup]

Miles Bader <snogglethorpe@gmail.com> writes:

> On Mon, 21 Mar 2005 15:20:34 -0800, Drew Adams <drew.adams@oracle.com> wrote:
>> In reply to David's point about preview-latex needing to search and find the
>> meta-text: I agree that programs need to be able to find meta-text. The
>> point is to separate such meta-text from the content - or at least be _able_
>> to separate/distinguish it. In most modes, at least, Info-search should not
>> find strings that are in meta-text (if this is not to difficult to fix).
>
> How about a text/overlay property that says what text to use for
> i-searching instead of the underlying text?  Then the mode itself
> (or whoever added the corresponding display/invisible property)
> could make exactly the right decision -- e.g. David would add the
> underlying text,

David would do no such thing because searches have to end up at the
correct corresponding buffer position, not inside of some overlay
property.  Which is not even possible if I do a search and replace on
a string bracketing an overlay boundary.

> but info mode could add the text shown by the display property.
>
> Something like:
>
>    (overlay-put ov 'isearch-text "This is the search text")
>    (overlay-put ov 'display (some-bit-map))

I think it is a bad idea.  What about matches partly in the text,
partly out of it?  What should get displayed?

The underlying basic unit of a buffer is text.  If you don't want it
manipulated as text, then don't use text in the first place.  Of
course, this means that we might have to rethink using "x" as a
character to carry images, and use something non-printable.  Or it
might be a good idea to have a property to mark text non-searchable
completely for this case.  But we should not build a completely
separate reality just for searching.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum