etags.el tags-search use global syntax table

etags.el tags-search use global syntax table

[isaac.to]

Hi,

When using AucTeX to edit some LaTeX files, I want to search for some
word globally.  My version is not new enough to support inter-file
searching directly, so I've used tags-search.  I'm searching for
something like "\bx\b".  It turned out that some of the matches are
missed, because $ is considered as a word (as in the fundamental
mode).  This is out of my expectation, since it is a LaTeX file and in
the LaTeX mode of AucTeX, $ is considered not a word.  Perhaps when
Emacs performs a tags-search it do it in a temp buffer which does not
have the correct mode loaded?  I consider this a bug in etags.

Regards,
Isaac

Re: etags.el tags-search use global syntax table

[Francesco Potorti`]

isaac.to@gmail.com:
>When using AucTeX to edit some LaTeX files, I want to search for some
>word globally.  My version is not new enough to support inter-file
>searching directly, so I've used tags-search.  I'm searching for
>something like "\bx\b".  It turned out that some of the matches are
>missed, because $ is considered as a word (as in the fundamental
>mode).  This is out of my expectation, since it is a LaTeX file and in
>the LaTeX mode of AucTeX, $ is considered not a word.  Perhaps when
>Emacs performs a tags-search it do it in a temp buffer which does not
>have the correct mode loaded?  I consider this a bug in etags.

This is indeed a bug in etags.el that was present in Emacs 21 and is
still there in Emacs 22.  It has nothing to do with Auctex, but pops up
whenever:
- you start a tags-search
- the file where you should see a match is not currently visited
- the search in Fundamental mode gives different results from the search
  in the proper mode of the buffer

The reason is that the next-file function in etags.el loads non-visited
files in a temporary buffer with insert-file-contents, rather than using
find-file, so the mode remains Fundamental.  I am not sure why it is so.
These are the relevant lines at the end of next-file:

      ;; Like find-file, but avoids random warning messages.
      (set-buffer (get-buffer-create " *next-file*"))
      (kill-all-local-variables)
      (erase-buffer)
      (setq new next)
      (insert-file-contents new nil))

Generally speaking, tags-search looks for all files in the TAGS file: if
they are visited, they are searched and when no match is found point is
restored.  If they are not visited, they are loaded into a temporary
buffer which is overwritten when no matches are found.  However, search
in this buffer is performed in Fundamental mode.

The cure is to use the syntax table relative to the natural mode of the
file when doing the search.  Is it really necessary to avoid using
find-file?  If yes, how can one guess the right syntax table and apply
it to the temporary buffer?

Re: etags.el tags-search use global syntax table

[Richard Stallman]

    The reason is that the next-file function in etags.el loads non-visited
    files in a temporary buffer with insert-file-contents, rather than using
    find-file, so the mode remains Fundamental.  I am not sure why it is so.

There are two reasons for this.  One is that some major modes can be
rather intrusive in what they do.  The other is that visiting a file
is much slower.

Of course, it was all based on the idea that it wouldn't really matter
for searching, and if it actually finds a match, then it visits the
file properly.  This bug shows that the major mode does matter in some
cases for searching.

I am not sure whether it is better to fix this bug or not.
If it is a big slowdown, we are better off not fixing it.

Re: etags.el tags-search use global syntax table

[Francesco Potorti`]

>    The reason is that the next-file function in etags.el loads non-visited
>    files in a temporary buffer with insert-file-contents, rather than using
>    find-file, so the mode remains Fundamental.  I am not sure why it is so.
>
>There are two reasons for this.  One is that some major modes can be
>rather intrusive in what they do.  The other is that visiting a file
>is much slower.
>
>Of course, it was all based on the idea that it wouldn't really matter
>for searching, and if it actually finds a match, then it visits the
>file properly.  This bug shows that the major mode does matter in some
>cases for searching.
>
>I am not sure whether it is better to fix this bug or not.
>If it is a big slowdown, we are better off not fixing it.

A reasonable compromise would be using for searching the same syntax
table of the buffer that is current when the tags-search command is
issued.  This would not work in various situations, but would work in
most cases, and probably would work in many more cases than the current
situation.

If nothing smarter comes up and no one does it first, I'll try to
implement that in the next days.

Re: etags.el tags-search use global syntax table

[Andreas Schwab]

Francesco Potorti` <pot@gnu.org> writes:

> A reasonable compromise would be using for searching the same syntax
> table of the buffer that is current when the tags-search command is
> issued.  This would not work in various situations, but would work in
> most cases, and probably would work in many more cases than the current
> situation.

You can initiate a tags search from any buffer, independent of the list
of files indexed by the tags file, thus you may get different results
depending on where you started it.

Andreas.

-- 
Andreas Schwab, SuSE Labs, schwab@suse.de
SuSE Linux Products GmbH, Maxfeldstraße 5, 90409 Nürnberg, Germany
PGP key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: etags.el tags-search use global syntax table

[Francesco Potorti`]

>> A reasonable compromise would be using for searching the same syntax
>> table of the buffer that is current when the tags-search command is
>> issued.  This would not work in various situations, but would work in
>> most cases, and probably would work in many more cases than the current
>> situation.
>
>You can initiate a tags search from any buffer, independent of the list
>of files indexed by the tags file, thus you may get different results
>depending on where you started it.

Yes.  Until now we have two different ways to go:

1. Leave everything as it is.  This means that invoking tags-search on
   Latex files will not recover the "\bx\b" pattern even if some of the
   files contain the "$x = y*z" string, no matter form where you start
   the search

2. Apply the change I propose.  This means that the search will recover
   the desiderd pattern in the usual case of starting the search from a
   Latex buffer, but will not recover it if you start the search from a
   text buffer

Does anyone have any opinion on the above alternatives, or any
additional alternative to suggest?

Re: etags.el tags-search use global syntax table

[Richard Stallman]

    A reasonable compromise would be using for searching the same syntax
    table of the buffer that is current when the tags-search command is
    issued.

That is not always right.  If you want to fix this, I'm sure you could
make etags first look up auto-mode-alist manually, without actually setting
the major mode, then have its own alist mapping mode functions into
syntax tables.

Or it could actually apply the major mode.  That might still be a lot
faster than visiting the file.  If (get MODE 'mode-class) is `special'
then it would not apply the major mode.

You could try that, and see if the slowdown is enough to be a real
reason not to do it.

Re: etags.el tags-search use global syntax table

[Francesco Potorti`]

This is a first try at applying the major mode to files that are loaded
by etags.el's next-file.  It works in the single test case I tried.  If
I am following the correct road, I will check that nothing is broken by
these changes and I will make speed tests, so please comment.

================================================================
--- etags.el	25 Feb 2007 15:54:43 +0100	1.194
+++ etags.el	19 Jul 2007 18:40:46 +0200	
@@ -1701,10 +1701,11 @@ if the file was newly read in, the value
 	(set-buffer (find-file-noselect next novisit))
       ;; Like find-file, but avoids random warning messages.
       (set-buffer (get-buffer-create " *next-file*"))
-      (kill-all-local-variables)
       (erase-buffer)
       (setq new next)
-      (insert-file-contents new nil))
+      (insert-file-contents new nil)
+      (let ((buffer-file-name new))
+	(set-auto-mode t (list 'special))))
     new))
 
 (defvar tags-loop-operate nil
================================================================

The above works because I modified set-auto-mode to accept one more
optional argument, as detailed in the following addition to the help
string:

+The optional argument KEEP-IF-CLASS is a list of symbols. If the
+major mode has any of these symbols in its mode-class property, or if
+one the symbols in the list is t, we do not set the major mode."

================================================================
--- files.el	19 Jul 2007 18:00:42 +0200	1.913
+++ files.el	19 Jul 2007 18:46:14 +0200	
@@ -2218,8 +2218,8 @@ If FUNCTION is nil, then it is not calle
   "Upper limit on `magic-mode-alist' regexp matches.
 Also applies to `magic-fallback-mode-alist'.")
 
-(defun set-auto-mode (&optional keep-mode-if-same)
-  "Select major mode appropriate for current buffer.
+(defun set-auto-mode (&optional keep-if-same keep-if-class)
+  "Select major mode appropriate for current buffer and return it.
 
 To find the right major mode, this function checks for a -*- mode tag,
 checks if it uses an interpreter listed in `interpreter-mode-alist',
@@ -2233,9 +2233,13 @@ Local Variables section of the file; for
 If `enable-local-variables' is nil, this function does not check for a
 -*- mode tag.
 
-If the optional argument KEEP-MODE-IF-SAME is non-nil, then we
+If the optional argument KEEP-IF-SAME is non-nil, then we
 set the major mode only if that would change it.  In other words
-we don't actually set it to the same mode the buffer already has."
+we don't actually set it to the same mode the buffer already has.
+
+The optional argument KEEP-IF-CLASS is a list of symbols. If the
+major mode has any of these symbols in its mode-class property, or if
+one the symbols in the list is t, we do not set the major mode."
   ;; Look for -*-MODENAME-*- or -*- ... mode: MODENAME; ... -*-
   (let (end done mode modes)
     ;; Find a -*- mode tag
@@ -2270,7 +2274,7 @@ we don't actually set it to the same mod
 	    (if (not (functionp mode))
 		(message "Ignoring unknown mode `%s'" mode)
 	      (setq done t)
-	      (or (set-auto-mode-0 mode keep-mode-if-same)
+	      (or (setq mode (set-auto-mode-0 mode keep-if-same keep-if-class))
 		  ;; continuing would call minor modes again, toggling them off
 		  (throw 'nop nil))))))
     ;; If we didn't, look for an interpreter specified in the first line.
@@ -2287,11 +2291,11 @@ we don't actually set it to the same mod
 	    done (assoc (file-name-nondirectory mode)
 			interpreter-mode-alist))
       ;; If we found an interpreter mode to use, invoke it now.
-      (if done
-	  (set-auto-mode-0 (cdr done) keep-mode-if-same)))
+      (when done
+	(setq mode (set-auto-mode-0 (cdr done) keep-if-same keep-if-class))))
     ;; Next try matching the buffer beginning against magic-mode-alist.
     (unless done
-      (if (setq done (save-excursion
+      (when (setq done (save-excursion
 		       (goto-char (point-min))
 		       (save-restriction
 			 (narrow-to-region (point-min)
@@ -2302,7 +2306,7 @@ we don't actually set it to the same mod
 					  (if (functionp re)
 					      (funcall re)
 					    (looking-at re)))))))
-	  (set-auto-mode-0 done keep-mode-if-same)))
+	(setq mode (set-auto-mode-0 done keep-if-same keep-if-class))))
     ;; Next compare the filename against the entries in auto-mode-alist.
     (unless done
       (if buffer-file-name
@@ -2335,11 +2339,11 @@ we don't actually set it to the same mod
 			name (substring name 0 (match-beginning 0)))
 		(setq name))
 	      (when mode
-		(set-auto-mode-0 mode keep-mode-if-same)
+		(setq mode (set-auto-mode-0 mode keep-if-same keep-if-class))
 		(setq done t))))))
     ;; Next try matching the buffer beginning against magic-fallback-mode-alist.
     (unless done
-      (if (setq done (save-excursion
+      (when (setq done (save-excursion
 		       (goto-char (point-min))
 		       (save-restriction
 			 (narrow-to-region (point-min)
@@ -2350,20 +2354,28 @@ we don't actually set it to the same mod
 					  (if (functionp re)
 					      (funcall re)
 					    (looking-at re)))))))
-	  (set-auto-mode-0 done keep-mode-if-same)))))
+	(setq mode (set-auto-mode-0 done keep-if-same keep-if-class))))
+    mode))
 
-;; When `keep-mode-if-same' is set, we are working on behalf of
+;; When `keep-if-same' is set, we are working on behalf of
 ;; set-visited-file-name.  In that case, if the major mode specified is the
 ;; same one we already have, don't actually reset it.  We don't want to lose
 ;; minor modes such as Font Lock.
-(defun set-auto-mode-0 (mode &optional keep-mode-if-same)
+(defun set-auto-mode-0 (mode &optional keep-if-same keep-if-class)
   "Apply MODE and return it.
-If optional arg KEEP-MODE-IF-SAME is non-nil, MODE is chased of
+If optional arg KEEP-IF-SAME is non-nil, MODE is chased of
 any aliases and compared to current major mode.  If they are the
-same, do nothing and return nil."
-  (unless (and keep-mode-if-same
+same, do nothing and return nil.
+
+The optional argument KEEP-IF-CLASS is a list of symbols. If the
+major mode has any of these symbols in its mode-class property, or if
+one the symbols in the list is t, we do not set the major mode."
+  (unless (or
+	   (and keep-if-same
 	       (eq (indirect-function mode)
 		   (indirect-function major-mode)))
+	   (memq t keep-if-class)
+	   (memq (get mode 'mode-class) keep-if-class))
     (when mode
       (funcall mode)
       mode)))
================================================================

Re: etags.el tags-search use global syntax table

[Richard Stallman]

    +The optional argument KEEP-IF-CLASS is a list of symbols. If the
    +major mode has any of these symbols in its mode-class property, or if
    +one the symbols in the list is t, we do not set the major mode."

I am not sure what that means.  Does it mean that if I pass (t foo)
for KEEP-IF-CLASS, then set-auto-mode will certainly not set the major
mode?

That is what the words seem to say -- but it doesn't seem useful.
What's the point of calling set-auto-mode and never setting the major mode?

Also, if you want a way to do that, why make the signal be a list
containing t?  Why not just pass t as the argument?  Why make it a
list if the rest of the list is irrelevant?

One other question.  Since set-auto-mode doesn't look at mode: local
bindings, is it sufficient?


Aside from that, it looks good.  Next question is how fast it runs.

Re: etags.el tags-search use global syntax table

[Francesco Potorti`]

>    +The optional argument KEEP-IF-CLASS is a list of symbols. If the
>    +major mode has any of these symbols in its mode-class property, or if
>    +one the symbols in the list is t, we do not set the major mode."
>
>I am not sure what that means.  Does it mean that if I pass (t foo)
>for KEEP-IF-CLASS, then set-auto-mode will certainly not set the major
>mode?

Yes.

>That is what the words seem to say -- but it doesn't seem useful.
>What's the point of calling set-auto-mode and never setting the major mode?

Those three lines were meant to present the gist of the change, that is,
an abstract of the following patch.  But I made another change, that is,
the patched set-auto-mode returns the chosen mode.  I think this may be
useful for programs that want to know which mode a buffer should be in,
but do not want to set it.

>Also, if you want a way to do that, why make the signal be a list
>containing t?  Why not just pass t as the argument?  Why make it a
>list if the rest of the list is irrelevant?

Yes, passing just t would certainly be clearer.  I thought that having
always a list would be simpler to use and possibly more flexible, but I
would certainly make that change if you think it is best.

>One other question.  Since set-auto-mode doesn't look at mode: local
>bindings, is it sufficient?

Well, no, it is not.  I had overlooked that problem, thanks.

>Aside from that, it looks good.  Next question is how fast it runs.

When I manage to find some more time, I'll look into the mode: local
bindings and do some speed tests.  

Re: etags.el tags-search use global syntax table

[Richard Stallman]

      But I made another change, that is,
    the patched set-auto-mode returns the chosen mode.  I think this may be
    useful for programs that want to know which mode a buffer should be in,
    but do not want to set it.

    >Also, if you want a way to do that, why make the signal be a list
    >containing t?  Why not just pass t as the argument?  Why make it a
    >list if the rest of the list is irrelevant?

    Yes, passing just t would certainly be clearer.  I thought that having
    always a list would be simpler to use and possibly more flexible, but I
    would certainly make that change if you think it is best.

Please use just t for that meaning, and document why it's useful.

Re: etags.el tags-search use global syntax table

[Francesco Potorti`]

isaac.to@gmail.com:
>When using AucTeX to edit some LaTeX files, I want to search for some
>word globally.  My version is not new enough to support inter-file
>searching directly, so I've used tags-search.  I'm searching for
>something like "\bx\b".  It turned out that some of the matches are
>missed, because $ is considered as a word (as in the fundamental
>mode).  This is out of my expectation, since it is a LaTeX file and in
>the LaTeX mode of AucTeX, $ is considered not a word.  Perhaps when
>Emacs performs a tags-search it do it in a temp buffer which does not
>have the correct mode loaded?  I consider this a bug in etags.

This is indeed a bug in etags.el that was present in Emacs 21 and is
still there in Emacs 22.  It has nothing to do with Auctex, but pops up
whenever:
- you start a tags-search
- the file where you should see a match is not currently visited
- the search in Fundamental mode gives different results from the search
  in the proper mode of the buffer

The reason is that the next-file function in etags.el loads non-visited
files in a temporary buffer with insert-file-contents, rather than using
find-file, so the mode remains Fundamental.  I am not sure why it is so.
These are the relevant lines at the end of next-file:

      ;; Like find-file, but avoids random warning messages.
      (set-buffer (get-buffer-create " *next-file*"))
      (kill-all-local-variables)
      (erase-buffer)
      (setq new next)
      (insert-file-contents new nil))

Generally speaking, tags-search looks for all files in the TAGS file: if
they are visited, they are searched and when no match is found point is
restored.  If they are not visited, they are loaded into a temporary
buffer which is overwritten when no matches are found.  However, search
in this buffer is performed in Fundamental mode.

The cure is to use the syntax table relative to the natural mode of the
file when doing the search.  Is it really necessary to avoid using
find-file?  If yes, how can one guess the right syntax table and apply
it to the temporary buffer?

Re: etags.el tags-search use global syntax table

[John Dennis]

[-- Attachment #1: Type: text/plain, Size: 2645 bytes --]

>> The reason is that the next-file function in etags.el loads
>> non-visited files in a temporary buffer with insert-file-contents,
>> rather than using find-file, so the mode remains Fundamental.  I am
>> not sure why it is so.

> There are two reasons for this.  One is that some major modes can be 
> rather intrusive in what they do.  The other is that visiting a file 
> is much slower.

> Of course, it was all based on the idea that it wouldn't really
> matter for searching, and if it actually finds a match, then it
> visits the file properly.  This bug shows that the major mode does
> matter in some cases for searching.

The the major mode does matter for searching, in fact it's often 
critical for correct searching. There are two key things a major mode 
does which affects searching, it establishes a syntax and often defines 
the value of case-fold-search. If either of these is not done before the 
search proceeds you'll often get incorrect search results. This is 
especially critical with program source code, which almost by definition 
is the context when a tags search is performed.

I consider this to be a serious flaw in etags and have been fixing it in 
my local version of emacs for quite a while now. The novisit behavior 
has another down side to my thinking as well, the buffer is not 
available in the buffer list unless a match was found. This makes for an 
odd and arbitrary set of buffers. My preference is to have all buffers 
which participated in the search to be available. Yes, this does mean 
more memory is used and loading is slower. But virtual memory is a 
wonderful thing, universally supported, and I can't see on any modern 
system the performance difference. Plus, if one does repeated searches 
isn't it more efficient to have the buffer already loaded rather than 
repeatedly loading it and then throwing it away?

I've also been bothered by the interface to tags-search, unlike the 
other tags functions it does not default to the word surrounded by 
point. It's really nice to be able to put your point on what you want to 
search for and just hit a function key.

Attached is a patch that I believe fixes these issues and allows for 
customizing the behavior. I don't see an easy fix for establishing the 
syntax table and value of case-fold-search for the novisit case. I guess 
if a user want to continue to utilize novisit that's a liability you'll 
have to live with in order to retain that feature.

Also, as an aside, should we still be supporting the slow baud rate 
stuff? It adds complexity to the code which I suspect is no longer relevant.

-- 
John Dennis <jdennis@redhat.com>

[-- Attachment #2: etags.patch --]
[-- Type: text/x-patch, Size: 1526 bytes --]

--- etags.el.orig	2007-11-30 13:46:35.000000000 -0500
+++ etags.el	2007-11-30 15:12:28.000000000 -0500
@@ -102,6 +102,11 @@
   :group 'etags
   :type 'boolean)
 
+(defcustom tags-novisit t
+  "*Non-nil means use a temporary buffer to save time and avoid uninteresting warnings."
+  :type 'boolean
+  :group 'etags)
+
 (defvar tags-table-computed-list nil
   "List of tags tables to search, computed from `tags-table-list'.
 This includes tables implicitly included by other tables.  The list is not
@@ -1698,7 +1703,8 @@
 	 (with-current-buffer buffer
 	   (revert-buffer t)))
     (if (not (and new novisit))
-	(set-buffer (find-file-noselect next novisit))
+	(progn (set-buffer (find-file-noselect next t))
+	       (setq new nil))
       ;; Like find-file, but avoids random warning messages.
       (set-buffer (get-buffer-create " *next-file*"))
       (kill-all-local-variables)
@@ -1761,7 +1767,7 @@
 	      (goto-char original-point))
 
 	    (setq file-finished nil)
-	    (setq new (next-file first-time t))
+	    (setq new (next-file first-time tags-novisit))
 
 	    ;; If NEW is non-nil, we got a temp buffer,
 	    ;; and NEW is the file name.
@@ -1803,7 +1809,7 @@
 To continue searching for next match, use command \\[tags-loop-continue].
 
 See documentation of variable `tags-file-name'."
-  (interactive "sTags search (regexp): ")
+  (interactive (find-tag-interactive "Tags search (regexp): "))
   (if (and (equal regexp "")
 	   (eq (car tags-loop-scan) 're-search-forward)
 	   (null tags-loop-operate))

Re: etags.el tags-search use global syntax table

[Richard Stallman]

    The the major mode does matter for searching, in fact it's often 
    critical for correct searching. There are two key things a major mode 
    does which affects searching, it establishes a syntax and often defines 
    the value of case-fold-search.

I am concerned that making etags invoke all major modes
might lead to a substantial slowdown (as it did in the past).
We could try it and see.  If it does, the best thing to do
would be to indicate somehow which modes really need to be selected
during tags search.

    The novisit behavior 
    has another down side to my thinking as well, the buffer is not 
    available in the buffer list unless a match was found. This makes for an 
    odd and arbitrary set of buffers. My preference is to have all buffers 
    which participated in the search to be available.

I strongly disagree.