bug#44494: etags.el xref-location-marker does not handle TAGS references to .el.gz files

bug#44494: etags.el xref-location-marker does not handle TAGS references to .el.gz files

[Pierre Rouleau]

[-- Attachment #1: Type: text/plain, Size: 5588 bytes --]

This problem was detected in emacs 26.3, but is also present in emacs 27.1,
according to the code posted inside
https://github.com/emacs-mirror/emacs/blob/master/test/manual/etags/el-src/emacs/lisp/progmodes/etags.el#L2139

Problem description follows:



----------------------------BUG DESCRIPTION [ ------------------------------

The current version of etags distributed with emacs 27.0 is capable of
parsing compressed files like something.el.gz but unfortunately the
reference inside the generated TAGS file uses the name "something.el"
instead of the complete file name "something.el.gz".  The same is true
for the other compressed files (.bz2, .xz and .lzma).

When trying to use the etags xref back-end to move point to a definition
inside such a file, the xref-find-definitions command will fail because the
TAGS file identifies the .el file instead of the .el.gz file.

xref-find-definitions fails because it only checks if the uncompressed
file exists, it does not try to see if the compressed file exists.

The issue is inside etags.el xref-location-marker method.

The current code is:

(cl-defmethod xref-location-marker ((l xref-etags-location))
  (with-slots (tag-info file) l
    (let ((buffer (find-file-noselect file)))
      (with-current-buffer buffer
        (save-excursion
          (etags-goto-tag-location tag-info)
          (point-marker))))))

One could consider that the issue is inside the etags utility.

An alternative would be to provide Emacs with the ability to check
for the presence of the .el file and if it is not present look for the
equivalent compressed files.

Here's a proposal for a solution:


(defun etags-file-or-compressed-file-for (fname)
  "Return the valid file name for FNAME.
Check if FNAME is an existing file name, if not
try FNAME appended with the following compression extensions:
- \".gz\", the extension of compressed files created by gzip
- \".bz2\", the extension for compressed files created by bzip2
- \".xz\", the extension for compressed files created by xz
- \".lzma\", the extension for compressed files created by xz.

Return the file that exists or nil if nothing found."
  (let ((fpath nil))
    (cl-dolist (ext '(""
                      ".gz"
                      ".bz2"
                      ".xz"
                      ".lzma"))
      (setq fpath (concat fname ext))
      (when (file-exists-p fpath)
        (cl-return fpath)))))

(cl-defmethod xref-location-marker ((l xref-etags-location))
  (with-slots (tag-info file) l
    (let (buffer
          (fname (pel-file-or-compressed-file-for file)))
      (if fname
          (setq buffer (find-file-noselect fname))
        (user-error "file %s (or .gz, .bz2, .xz, .lzma) does not exist"
file))
      (with-current-buffer buffer
        (save-excursion
          (etags-goto-tag-location tag-info)
          (point-marker))))))

------------------------------ END OF DESCRIPTION ]
--------------------------------


In GNU Emacs 26.3 (build 1, x86_64-apple-darwin18.6.0)
 of 2019-08-30 built on Mojave.local
Recent messages:
For information about GNU Emacs and the GNU system, type C-h C-a.

Configured using:
 'configure --disable-dependency-tracking --disable-silent-rules
 --enable-locallisppath=/usr/local/share/emacs/site-lisp
 --infodir=/usr/local/Cellar/emacs/26.3/share/info/emacs
 --prefix=/usr/local/Cellar/emacs/26.3 --with-gnutls --without-x
 --with-xml2 --without-dbus --with-modules --without-ns
 --without-imagemagick'

Configured features:
NOTIFY ACL GNUTLS LIBXML2 ZLIB MODULES THREADS

Important settings:
  value of $LANG: en_CA.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Lisp Interaction

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  electric-indent-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message rmc puny seq byte-opt gv
bytecomp byte-compile cconv cl-loaddefs cl-lib dired dired-loaddefs
format-spec rfc822 mml easymenu mml-sec password-cache epa derived epg
epg-config gnus-util rmail tool-bar rmail-loaddefs mm-decode mm-bodies
mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader sendmail
regexp-opt rfc2047 rfc2045 ietf-drums mm-util mail-prsvr mail-utils
term/xterm xterm time-date elec-pair mule-util tooltip eldoc electric
uniquify ediff-hook vc-hooks lisp-float-type tabulated-list replace
newcomment text-mode elisp-mode lisp-mode prog-mode register page
menu-bar rfn-eshadow isearch timer select mouse jit-lock font-lock
syntax facemenu font-core term/tty-colors frame cl-generic cham georgian
utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean
japanese eucjp-ms cp51932 hebrew greek romanian slovak czech european
ethiopic indian cyrillic chinese composite charscript charprop
case-table epa-hook jka-cmpr-hook help simple abbrev obarray minibuffer
cl-preloaded nadvice loaddefs button faces cus-face macroexp files
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote threads kqueue
multi-tty make-network-process emacs)

Memory information:
((conses 16 96177 5831)
 (symbols 48 19813 1)
 (miscs 40 33 96)
 (strings 32 28253 1011)
 (string-bytes 1 748198)
 (vectors 16 11977)
 (vector-slots 8 455846 6474)
 (floats 8 48 566)
 (intervals 56 197 0)
 (buffers 992 11))

;; ---------------------------

Thank you,

/Pierre Rouleau

[-- Attachment #2: Type: text/html, Size: 6816 bytes --]

bug#44494: etags.el xref-location-marker does not handle TAGS references to .el.gz files

[Dmitry Gutov]

On 07.11.2020 01:22, Pierre Rouleau wrote:
> This problem was detected in emacs 26.3, but is also present in emacs 
> 27.1, according to the code posted inside 
> https://github.com/emacs-mirror/emacs/blob/master/test/manual/etags/el-src/emacs/lisp/progmodes/etags.el#L2139

Hi!

Do you see the same problem with 'M-x find-tag'?

bug#44494: etags.el xref-location-marker does not handle TAGS references to .el.gz files

[Pierre Rouleau]

[-- Attachment #1: Type: text/plain, Size: 2966 bytes --]

On Fri, Nov 6, 2020 at 9:22 PM Dmitry Gutov <dgutov@yandex.ru> wrote:

> On 07.11.2020 01:22, Pierre Rouleau wrote:
> > This problem was detected in emacs 26.3, but is also present in emacs
> > 27.1, according to the code posted inside
> >
> https://github.com/emacs-mirror/emacs/blob/master/test/manual/etags/el-src/emacs/lisp/progmodes/etags.el#L2139
>
> Hi!
>
> Do you see the same problem with 'M-x find-tag'?
>

Short answer:  yes

Longer answer:  you can try it on Emacs lib files.

For example, I created a TAGS file that contains the following:

define-globalized-minor-mode global-prettify-symbols-mode^?247,10355
(define-derived-mode prog-mode ^?251,10485
^L
/usr/local/Cellar/emacs/26.3/share/emacs/26.3/lisp/progmodes/cc-bytecomp.el,1014
(defvar cc-bytecomp-unbound-variables ^?76,2968
(defvar cc-bytecomp-original-functions ^?77,3011
(defvar cc-bytecomp-original-properties ^?78,3055
(defvar cc-bytecomp-loaded-files ^?79,3100
(defvar cc-bytecomp-environment-set ^?86,3302
(defmacro cc-bytecomp-debug-msg ^?88,3344
(defun cc-bytecomp-compiling-or-loading ^?93,3432
(defsubst cc-bytecomp-is-compiling ^?134,4714
(defsubst cc-bytecomp-is-loading ^?138,4857
(defun cc-bytecomp-setup-environment ^?143,5065
(defun cc-bytecomp-restore-environment ^?191,6703
(defun cc-bytecomp-load ^?256,8749
(defmacro cc-require ^?293,10222
(defmacro cc-conditional-require ^?305,10617
(defmacro cc-conditional-require-after-load ^?318,11068
(defmacro cc-provide ^?333,11627
(defmacro cc-load ^?340,11887
(defmacro cc-require-when-compile ^?351,12266
(defmacro cc-external-require ^?362,12703
(defmacro cc-bytecomp-defvar ^?371,13055
(defmacro cc-bytecomp-defun ^?392,13857
(defmacro cc-bytecomp-put ^?419,14990
(defmacro cc-bytecomp-boundp ^?437,15739
(defmacro cc-bytecomp-fboundp ^?447,16140
^L
/usr/local/Cellar/emacs/26.3/share/emacs/26.3/lisp/progmodes/make-mode.el,6494
(defgroup makefile ^?95,3661
(defface makefile-space^?101,3839
(defface makefile-targets^?107,4026
(defface makefile-shell^?114,4302


Then with the file
/usr/local/Cellar/emacs/26.3/share/emacs/26.3/lisp/progmodes/cc-cmds.el.gz
in a buffer and cc-bytecomp not loaded trying both

M-x xref-find-definitions cc-require

and

M-x find-tag cc-require

I get:

Rerun etags: ‘^(defmacro cc-require ’ not found in
/usr/local/Cellar/emacs/26.3/share/emacs/26.3/lisp/progmodes/cc-bytecomp.el

If I update the etags.el with my suggested code, M-x xref-find-definition
works but find-tag still does not.
Since find-tag is obsolete since emacs 25.1 I did not look into it.


I did not look into the etags utility itself yet.  This looks like a
consistency problem.
I would think that the etags utility should generate a reference for el.gz
files listing
the real file name.  On the other hand having the fix in both places both
in etags utility
and in emacs etags.el would reduce probability of errors.


-- 
/Pierre

[-- Attachment #2: Type: text/html, Size: 3989 bytes --]

bug#44494: etags.el xref-location-marker does not handle TAGS references to .el.gz files

[Eli Zaretskii]

> From: Pierre Rouleau <prouleau001@gmail.com>
> Date: Fri, 6 Nov 2020 18:22:46 -0500
> 
> One could consider that the issue is inside the etags utility.

I don't think this is the best alternative.  TAGS tables are supposed
to provide information in a way that doesn't require re-running etags
each time some change in the sources is made.  The way things are now,
compressing or decompressing a file doesn't require re-running etags,
which is a Good Thing.

Handling this in the code which accesses and interprets TAGS sounds
like a better alternative.

> Here's a proposal for a solution:
> 
> (defun etags-file-or-compressed-file-for (fname)
>   "Return the valid file name for FNAME.
> Check if FNAME is an existing file name, if not
> try FNAME appended with the following compression extensions:
> - \".gz\", the extension of compressed files created by gzip
> - \".bz2\", the extension for compressed files created by bzip2
> - \".xz\", the extension for compressed files created by xz
> - \".lzma\", the extension for compressed files created by xz.
> 
> Return the file that exists or nil if nothing found."
>   (let ((fpath nil))
>     (cl-dolist (ext '(""
>                       ".gz"
>                       ".bz2"
>                       ".xz"
>                       ".lzma"))
>       (setq fpath (concat fname ext))
>       (when (file-exists-p fpath)
>         (cl-return fpath)))))

We already have in etags.el support for finding compressed files, see
tag-find-file-of-tag-noselect.  Can't that be reused?  If not, let's
at least reuse tags-compression-info-list, okay?

bug#44494: etags.el xref-location-marker does not handle TAGS references to .el.gz files

[Eli Zaretskii]

> From: Pierre Rouleau <prouleau001@gmail.com>
> Date: Fri, 6 Nov 2020 22:31:02 -0500
> Cc: 44494@debbugs.gnu.org
> 
>  Do you see the same problem with 'M-x find-tag'?
> 
> Short answer:  yes
> 
> Longer answer:  you can try it on Emacs lib files.  
> 
> For example, I created a TAGS file that contains the following:
> 
> define-globalized-minor-mode global-prettify-symbols-mode^?247,10355
> (define-derived-mode prog-mode ^?251,10485
> ^L
> /usr/local/Cellar/emacs/26.3/share/emacs/26.3/lisp/progmodes/cc-bytecomp.el,1014
> (defvar cc-bytecomp-unbound-variables ^?76,2968
> (defvar cc-bytecomp-original-functions ^?77,3011
> (defvar cc-bytecomp-original-properties ^?78,3055
> (defvar cc-bytecomp-loaded-files ^?79,3100
> (defvar cc-bytecomp-environment-set ^?86,3302
> (defmacro cc-bytecomp-debug-msg ^?88,3344
> (defun cc-bytecomp-compiling-or-loading ^?93,3432
> (defsubst cc-bytecomp-is-compiling ^?134,4714
> (defsubst cc-bytecomp-is-loading ^?138,4857
> (defun cc-bytecomp-setup-environment ^?143,5065
> (defun cc-bytecomp-restore-environment ^?191,6703
> (defun cc-bytecomp-load ^?256,8749
> (defmacro cc-require ^?293,10222
> (defmacro cc-conditional-require ^?305,10617
> (defmacro cc-conditional-require-after-load ^?318,11068
> (defmacro cc-provide ^?333,11627
> (defmacro cc-load ^?340,11887
> (defmacro cc-require-when-compile ^?351,12266
> (defmacro cc-external-require ^?362,12703
> (defmacro cc-bytecomp-defvar ^?371,13055
> (defmacro cc-bytecomp-defun ^?392,13857
> (defmacro cc-bytecomp-put ^?419,14990
> (defmacro cc-bytecomp-boundp ^?437,15739
> (defmacro cc-bytecomp-fboundp ^?447,16140
> ^L
> /usr/local/Cellar/emacs/26.3/share/emacs/26.3/lisp/progmodes/make-mode.el,6494
> (defgroup makefile ^?95,3661
> (defface makefile-space^?101,3839
> (defface makefile-targets^?107,4026
> (defface makefile-shell^?114,4302
> 
> Then with the file /usr/local/Cellar/emacs/26.3/share/emacs/26.3/lisp/progmodes/cc-cmds.el.gz in a buffer
> and cc-bytecomp not loaded trying both
> 
> M-x xref-find-definitions cc-require
> 
> and
> 
> M-x find-tag cc-require
> 
> I get: 
> 
> Rerun etags: ‘^(defmacro cc-require ’ not found in
> /usr/local/Cellar/emacs/26.3/share/emacs/26.3/lisp/progmodes/cc-bytecomp.el

I cannot reproduce this: find-tag works in this situation, at least in
the current emacs-27 branch and in stock Emacs 27.1.  Which doesn't
surprise me, since etags.el already has code that handles compressed
files.

Moreover, M-. (which uses xref) works as well.  So I'm no longer sure
I understand what is the problem you are seeing.  If you see this in
Emacs 26, please retry in Emacs 27, and let's take this from there.

FTR, the steps I used for reproducing were slightly different:

  . "make TAGS" in the top-level directory of the Emacs source tree
  . gzip lisp/abbrevs.el
  . emacs -Q
  . C-x C-f lisp/simple.el
  . C-u M-. kill-all-abbrevs RET

And for find-tag, replace the last 2 commands with

  . M-x visit-tags-table RET RET
  . M-x find-tag RET kill-all-abbrevs RET

Both of these work and show abbrevs.el.gz at the correct line.

bug#44494: etags.el xref-location-marker does not handle TAGS references to .el.gz files

[Pierre Rouleau]

[-- Attachment #1: Type: text/plain, Size: 5743 bytes --]

On Sat, Nov 7, 2020 at 3:00 AM Eli Zaretskii <eliz@gnu.org> wrote:

> > From: Pierre Rouleau <prouleau001@gmail.com>
> > Date: Fri, 6 Nov 2020 22:31:02 -0500
> > Cc: 44494@debbugs.gnu.org
> >
> >  Do you see the same problem with 'M-x find-tag'?
> >
> > Short answer:  yes
> >
> > Longer answer:  you can try it on Emacs lib files.
> >
> > For example, I created a TAGS file that contains the following:
> >
> > define-globalized-minor-mode global-prettify-symbols-mode^?247,10355
> > (define-derived-mode prog-mode ^?251,10485
> > ^L
> >
> /usr/local/Cellar/emacs/26.3/share/emacs/26.3/lisp/progmodes/cc-bytecomp.el,1014
> > (defvar cc-bytecomp-unbound-variables ^?76,2968
> > (defvar cc-bytecomp-original-functions ^?77,3011
> > (defvar cc-bytecomp-original-properties ^?78,3055
> > (defvar cc-bytecomp-loaded-files ^?79,3100
> > (defvar cc-bytecomp-environment-set ^?86,3302
> > (defmacro cc-bytecomp-debug-msg ^?88,3344
> > (defun cc-bytecomp-compiling-or-loading ^?93,3432
> > (defsubst cc-bytecomp-is-compiling ^?134,4714
> > (defsubst cc-bytecomp-is-loading ^?138,4857
> > (defun cc-bytecomp-setup-environment ^?143,5065
> > (defun cc-bytecomp-restore-environment ^?191,6703
> > (defun cc-bytecomp-load ^?256,8749
> > (defmacro cc-require ^?293,10222
> > (defmacro cc-conditional-require ^?305,10617
> > (defmacro cc-conditional-require-after-load ^?318,11068
> > (defmacro cc-provide ^?333,11627
> > (defmacro cc-load ^?340,11887
> > (defmacro cc-require-when-compile ^?351,12266
> > (defmacro cc-external-require ^?362,12703
> > (defmacro cc-bytecomp-defvar ^?371,13055
> > (defmacro cc-bytecomp-defun ^?392,13857
> > (defmacro cc-bytecomp-put ^?419,14990
> > (defmacro cc-bytecomp-boundp ^?437,15739
> > (defmacro cc-bytecomp-fboundp ^?447,16140
> > ^L
> >
> /usr/local/Cellar/emacs/26.3/share/emacs/26.3/lisp/progmodes/make-mode.el,6494
> > (defgroup makefile ^?95,3661
> > (defface makefile-space^?101,3839
> > (defface makefile-targets^?107,4026
> > (defface makefile-shell^?114,4302
> >
> > Then with the file
> /usr/local/Cellar/emacs/26.3/share/emacs/26.3/lisp/progmodes/cc-cmds.el.gz
> in a buffer
> > and cc-bytecomp not loaded trying both
> >
> > M-x xref-find-definitions cc-require
> >
> > and
> >
> > M-x find-tag cc-require
> >
> > I get:
> >
> > Rerun etags: ‘^(defmacro cc-require ’ not found in
> >
> /usr/local/Cellar/emacs/26.3/share/emacs/26.3/lisp/progmodes/cc-bytecomp.el
>
> I cannot reproduce this: find-tag works in this situation, at least in
> the current emacs-27 branch and in stock Emacs 27.1.  Which doesn't
> surprise me, since etags.el already has code that handles compressed
> files.
>
> Moreover, M-. (which uses xref) works as well.  So I'm no longer sure
> I understand what is the problem you are seeing.  If you see this in
> Emacs 26, please retry in Emacs 27, and let's take this from there.
>
> FTR, the steps I used for reproducing were slightly different:
>
>   . "make TAGS" in the top-level directory of the Emacs source tree
>   . gzip lisp/abbrevs.el
>   . emacs -Q
>   . C-x C-f lisp/simple.el
>   . C-u M-. kill-all-abbrevs RET
>
> And for find-tag, replace the last 2 commands with
>
>   . M-x visit-tags-table RET RET
>   . M-x find-tag RET kill-all-abbrevs RET
>
> Both of these work and show abbrevs.el.gz at the correct line.
>

Hi,  I did try your test, both on emacs 26.3 and 27.1,
and for the exact scenario you described, it works if you search for
kill-all-abbrevs.

I should have better describe the failure scenario.
- Problem  occurs in emacs 26.3 and 27.1.
- I search for cc-require ,  a defmacro, not a defun, and not something
  that is already loaded.
- on my system all files emacs/lisp directory are .el.gz file, however,
  I do not think that the fact they are all compressed is significant.

With both emacs 26.3 and 27.1, iI do:
. emacs -Q
. C-x C-f lisp/simple.el
. C-u M-x cc-require
==> message: No definitions found for cc-require

At this point:
- the xref backend is the default for .el files; etags.
- xref-backend-functions is set to (elisp--xref-backend t), a local setting.
- the abbrev feature is loaded, as M-: (featurep 'abbrev) returns t.
- So looking for kill-all-abbrevs using the elisp--xref-backend works, my
  understanding is that it works because the file that holds the definition
  is loaded.

I wanted to be able to access definitions from code that might not have
already been loaded so I thought I would use the etags--xref-backend
instead.

With emacs 26.3 and 27.1 with all lisp/simple files compressed,
 I do (and get approximately same results, shown below):
. emacs -Q
. C-x C-f lisp/simple.el.gz
. M-x xref-etags-mode
. C-u M-x cc-require
emacs==> prompts Visit tags table (default TAGS): ....
me ====> I select the TAGS file where all definitions are stored and hit RET
- emacs 26.3 ==> Rerun etags: ‘^(defmacro cc-require ’ not found in
/usr/local/Cellar/emacs/26.3/share/emacs/26.3/lisp/progmod\
es/cc-bytecomp.el
- emacs 27.1 ==> Rerun etags: ‘^(defmacro cc-require ’ not found in
/usr/local/Cellar/emacs/27.1/share/emacs/27.1/lisp/progmodes/cc-bytecomp.el

If I look inside the TAGS file and search for cc-require,
I see it listed under the cc-bytecomp.el file.

If at this point, I load the 2 functions I wrote and try again,
it succeeds in finding cc-require.

To recap you need to try searching for something that is not already
loaded and use the etags xref backend with a file that contains the
definition
of what one is searching and that is located inside a compressed file.

Let me know if you want me to try other scenarios.

Thanks!

/Pierre Rouleau

[-- Attachment #2: Type: text/html, Size: 7173 bytes --]

bug#44494: etags.el xref-location-marker does not handle TAGS references to .el.gz files

[Eli Zaretskii]

> From: Pierre Rouleau <prouleau001@gmail.com>
> Date: Sat, 7 Nov 2020 09:15:12 -0500
> Cc: dgutov@yandex.ru, 44494@debbugs.gnu.org
> 
> With both emacs 26.3 and 27.1, iI do:
> . emacs -Q
> . C-x C-f lisp/simple.el
> . C-u M-x cc-require
> ==> message: No definitions found for cc-require
> 
> At this point:
> - the xref backend is the default for .el files; etags.
> - xref-backend-functions is set to (elisp--xref-backend t), a local setting.
> - the abbrev feature is loaded, as M-: (featurep 'abbrev) returns t.
> - So looking for kill-all-abbrevs using the elisp--xref-backend works, my 
>   understanding is that it works because the file that holds the definition 
>   is loaded.

How does this explain the fact that find-tag also worked for me?  Does
it work for you?

bug#44494: etags.el xref-location-marker does not handle TAGS references to .el.gz files

[Eli Zaretskii]

> From: Pierre Rouleau <prouleau001@gmail.com>
> Date: Sat, 7 Nov 2020 09:15:12 -0500
> Cc: dgutov@yandex.ru, 44494@debbugs.gnu.org
> 
> To recap you need to try searching for something that is not already 
> loaded and use the etags xref backend with a file that contains the definition
> of what one is searching and that is located inside a compressed file.

OK, I've now tried this with paren.el, which is not loaded in "emacs
-Q".  I can confirm that M-. fails to find functions in paren.el (I
tried show-paren-function, FTR), even if I use xref-etags-mode, but
"M-x find-tag" succeeds.

So I think we should try to understand why find-tag does work in this
case, and see how to make xref-find-definitions do the same.  Could
you perhaps do that?

> . emacs -Q
> . C-x C-f lisp/simple.el.gz
> . M-x xref-etags-mode
> . C-u M-x cc-require
> emacs==> prompts Visit tags table (default TAGS): ....
> me ====> I select the TAGS file where all definitions are stored and hit RET
> - emacs 26.3 ==> Rerun etags: ‘^(defmacro cc-require ’ not found in
> /usr/local/Cellar/emacs/26.3/share/emacs/26.3/lisp/progmod\
> es/cc-bytecomp.el
> - emacs 27.1 ==> Rerun etags: ‘^(defmacro cc-require ’ not found in
> /usr/local/Cellar/emacs/27.1/share/emacs/27.1/lisp/progmodes/cc-bytecomp.el

Note that at this point, you have an empty cc-bytecomp.el buffer.
Which I think gives a clue as to where the problem lies.

bug#44494: etags.el xref-location-marker does not handle TAGS references to .el.gz files

[Pierre Rouleau]

[-- Attachment #1: Type: text/plain, Size: 1916 bytes --]

On Sat, Nov 7, 2020 at 9:37 AM Eli Zaretskii <eliz@gnu.org> wrote:

> > From: Pierre Rouleau <prouleau001@gmail.com>
> > Date: Sat, 7 Nov 2020 09:15:12 -0500
> > Cc: dgutov@yandex.ru, 44494@debbugs.gnu.org
> >
> > To recap you need to try searching for something that is not already
> > loaded and use the etags xref backend with a file that contains the
> definition
> > of what one is searching and that is located inside a compressed file.
>
> OK, I've now tried this with paren.el, which is not loaded in "emacs
> -Q".  I can confirm that M-. fails to find functions in paren.el (I
> tried show-paren-function, FTR), even if I use xref-etags-mode, but
> "M-x find-tag" succeeds.
>
> So I think we should try to understand why find-tag does work in this
> case, and see how to make xref-find-definitions do the same.  Could
> you perhaps do that?
>
> > . emacs -Q
> > . C-x C-f lisp/simple.el.gz
> > . M-x xref-etags-mode
> > . C-u M-x cc-require
> > emacs==> prompts Visit tags table (default TAGS): ....
> > me ====> I select the TAGS file where all definitions are stored and hit
> RET
> > - emacs 26.3 ==> Rerun etags: ‘^(defmacro cc-require ’ not found in
> > /usr/local/Cellar/emacs/26.3/share/emacs/26.3/lisp/progmod\
> > es/cc-bytecomp.el
> > - emacs 27.1 ==> Rerun etags: ‘^(defmacro cc-require ’ not found in
> >
> /usr/local/Cellar/emacs/27.1/share/emacs/27.1/lisp/progmodes/cc-bytecomp.el
>
> Note that at this point, you have an empty cc-bytecomp.el buffer.
> Which I think gives a clue as to where the problem lies.
>

You are correct, I tried it with find-tag in emacs 26.3 and 27.1 and
find-tag cc-require
does find it, even with the xref-backend-functions set to its default of
(elisp--xref-backend t).

It fails with xref-find-definitions but works with find-tag.

I agree there's a need to see what differs there.

Thanks

-- 
/Pierre

[-- Attachment #2: Type: text/html, Size: 2720 bytes --]

bug#44494: etags.el xref-location-marker does not handle TAGS references to .el.gz files

[Pierre Rouleau]

[-- Attachment #1: Type: text/plain, Size: 2804 bytes --]

On Sat, Nov 7, 2020 at 9:48 AM Pierre Rouleau <prouleau001@gmail.com> wrote:

> On Sat, Nov 7, 2020 at 9:37 AM Eli Zaretskii <eliz@gnu.org> wrote:
>
>> > From: Pierre Rouleau <prouleau001@gmail.com>
>> > Date: Sat, 7 Nov 2020 09:15:12 -0500
>> > Cc: dgutov@yandex.ru, 44494@debbugs.gnu.org
>> >
>> > To recap you need to try searching for something that is not already
>> > loaded and use the etags xref backend with a file that contains the
>> definition
>> > of what one is searching and that is located inside a compressed file.
>>
>> OK, I've now tried this with paren.el, which is not loaded in "emacs
>> -Q".  I can confirm that M-. fails to find functions in paren.el (I
>> tried show-paren-function, FTR), even if I use xref-etags-mode, but
>> "M-x find-tag" succeeds.
>>
>> So I think we should try to understand why find-tag does work in this
>> case, and see how to make xref-find-definitions do the same.  Could
>> you perhaps do that?
>>
>> > . emacs -Q
>> > . C-x C-f lisp/simple.el.gz
>> > . M-x xref-etags-mode
>> > . C-u M-x cc-require
>> > emacs==> prompts Visit tags table (default TAGS): ....
>> > me ====> I select the TAGS file where all definitions are stored and
>> hit RET
>> > - emacs 26.3 ==> Rerun etags: ‘^(defmacro cc-require ’ not found in
>> > /usr/local/Cellar/emacs/26.3/share/emacs/26.3/lisp/progmod\
>> > es/cc-bytecomp.el
>> > - emacs 27.1 ==> Rerun etags: ‘^(defmacro cc-require ’ not found in
>> >
>> /usr/local/Cellar/emacs/27.1/share/emacs/27.1/lisp/progmodes/cc-bytecomp.el
>>
>> Note that at this point, you have an empty cc-bytecomp.el buffer.
>> Which I think gives a clue as to where the problem lies.
>>
>
> You are correct, I tried it with find-tag in emacs 26.3 and 27.1 and
> find-tag cc-require
> does find it, even with the xref-backend-functions set to its default of
> (elisp--xref-backend t).
>
> It fails with xref-find-definitions but works with find-tag.
>
> I agree there's a need to see what differs there.
>
> Thanks
>
> --
> /Pierre
>

One difference is that when using find-tag is using code from etags.el
exclusively:
- find-tag-noselect
. - find-tag-in-order   , which tries different predicates and the one that
succeeds is tag-implicit-name-match-p
.  - it identifies the cc-bytecomp.el.gz
.- calls etags-goto-tag-location

When using xref-find-definition the xref backend is used.  It's not the
same code.
The xref backend code for elisp does not find it.  The backend code for
etags does not find it either.
It tries to open cc-bytecomp.el as its the file name it gets from the TAGS
file.
It detects the file not being present and reports it as missing, assuming
the file have been removed.

To me the 2 sets of code look very different.

-- 
/Pierre

[-- Attachment #2: Type: text/html, Size: 4019 bytes --]

bug#44494: etags.el xref-location-marker does not handle TAGS references to .el.gz files

[Eli Zaretskii]

> From: Pierre Rouleau <prouleau001@gmail.com>
> Date: Sat, 7 Nov 2020 10:39:41 -0500
> Cc: dgutov@yandex.ru, 44494@debbugs.gnu.org
> 
> One difference is that when using find-tag is using code from etags.el exclusively:
> - find-tag-noselect
> . - find-tag-in-order   , which tries different predicates and the one that succeeds is
> tag-implicit-name-match-p 
> .  - it identifies the cc-bytecomp.el.gz 
> .- calls etags-goto-tag-location

The Xref etags backend also uses tag-implicit-name-match-p:

  (defvar etags-xref-find-definitions-tag-order '(tag-exact-match-p
						  tag-implicit-name-match-p)
    "Tag order used in `xref-backend-definitions' to look for definitions.

  If you want `xref-find-definitions' to find the tagged files by their
  file name, add `tag-partial-file-name-match-p' to the list value.")

> When using xref-find-definition the xref backend is used.  It's not the same code.  
> The xref backend code for elisp does not find it.  The backend code for etags does not find it either.
> It tries to open cc-bytecomp.el as its the file name it gets from the TAGS file.
> It detects the file not being present and reports it as missing, assuming the file have been removed.
> 
> To me the 2 sets of code look very different.

They share some of the code, at least when xref-etags-mode is used.
So it sounds like some information found by tag-implicit-name-match-p
doesn't get handed back to Xref?

For the Xref's own ELisp backend, we will probably need to code
something in xref.el.

bug#44494: etags.el xref-location-marker does not handle TAGS references to .el.gz files

[Pierre Rouleau]

[-- Attachment #1: Type: text/plain, Size: 2032 bytes --]

On Sat, Nov 7, 2020 at 10:52 AM Eli Zaretskii <eliz@gnu.org> wrote:

> > From: Pierre Rouleau <prouleau001@gmail.com>
> > Date: Sat, 7 Nov 2020 10:39:41 -0500
> > Cc: dgutov@yandex.ru, 44494@debbugs.gnu.org
> >
> > One difference is that when using find-tag is using code from etags.el
> exclusively:
> > - find-tag-noselect
> > . - find-tag-in-order   , which tries different predicates and the one
> that succeeds is
> > tag-implicit-name-match-p
> > .  - it identifies the cc-bytecomp.el.gz
> > .- calls etags-goto-tag-location
>
> The Xref etags backend also uses tag-implicit-name-match-p:
>
>   (defvar etags-xref-find-definitions-tag-order '(tag-exact-match-p
>
> tag-implicit-name-match-p)
>     "Tag order used in `xref-backend-definitions' to look for definitions.
>
>   If you want `xref-find-definitions' to find the tagged files by their
>   file name, add `tag-partial-file-name-match-p' to the list value.")
>
> > When using xref-find-definition the xref backend is used.  It's not the
> same code.
> > The xref backend code for elisp does not find it.  The backend code for
> etags does not find it either.
> > It tries to open cc-bytecomp.el as its the file name it gets from the
> TAGS file.
> > It detects the file not being present and reports it as missing,
> assuming the file have been removed.
> >
> > To me the 2 sets of code look very different.
>
> They share some of the code, at least when xref-etags-mode is used.
> So it sounds like some information found by tag-implicit-name-match-p
> doesn't get handed back to Xref?
>
> For the Xref's own ELisp backend, we will probably need to code
> something in xref.el.
>

Sorry, I meant that find-tag-in-order is able to find the reference in the
TAGS file when
it tries the tag-implicit-name-match-p predicate.  find-tag-in-order
returns a marker that identifies the .el.gz file.

I agree that for the ELisp backend something probably needs to be done to
support it.
But I also think that something must also be done for the etags xref
backend.
-- 
/Pierre

[-- Attachment #2: Type: text/html, Size: 2927 bytes --]