bug#21934: 24.5; find-tag: reading TAGS file incorrectly

bug#21934: 24.5; find-tag: reading TAGS file incorrectly

[Andreas Matthias]

[-- Attachment #1: Type: text/plain, Size: 252 bytes --]

find-tag does not read the TAGS file correctly. This concerns
all tags in the TAGS file which contain a period, e.g.: Shape.getPos.

Attached is a lua-file with its corresponding TAGS file. Running
M-x find-tag <TAB> does not list the tags correctly.


[-- Attachment #2: test.lua --]
[-- Type: application/octet-stream, Size: 91 bytes --]

Rectangle = {}
function Rectangle.getPos ()
end

Circle = {}
function Circle.getPos ()
end

[-- Attachment #3: TAGS --]
[-- Type: application/octet-stream, Size: 75 bytes --]

[-- Attachment #4: Type: text/plain, Size: 98 bytes --]



I think the problem is in function etags-tags-completion-table() in etags.el.
Here is a patch:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #5: etags.patch --]
[-- Type: text/x-diff, Size: 600 bytes --]

--- etags.el.orig	2015-11-16 20:25:20.153470934 +0100
+++ etags.el	2015-11-16 20:25:27.101470859 +0100
@@ -1285,8 +1285,8 @@
       ;;   \6 is the line to start searching at;
       ;;   \7 is the char to start searching at.
       (while (re-search-forward
-	      "^\\(\\([^\177]+[^-a-zA-Z0-9_+*$:\177]+\\)?\
-\\([-a-zA-Z0-9_+*$?:]+\\)[^-a-zA-Z0-9_+*$?:\177]*\\)\177\
+	      "^\\(\\([^\177]+[^-.a-zA-Z0-9_+*$:\177]+\\)?\
+\\([-.a-zA-Z0-9_+*$?:]+\\)[^-.a-zA-Z0-9_+*$?:\177]*\\)\177\
 \\(\\([^\n\001]+\\)\001\\)?\\([0-9]+\\)?,\\([0-9]+\\)?\n"
 	      nil t)
 	(intern	(prog1 (if (match-beginning 5)

[-- Attachment #6: Type: text/plain, Size: 3649 bytes --]


Andreas





In GNU Emacs 24.5.2 (x86_64-unknown-linux-gnu, GTK+ Version 3.10.8)
 of 2015-08-25 on winky
Windowing system distributor `The X.Org Foundation', version 11.0.11501000
System Description:	Ubuntu 14.04.3 LTS

Configured using:
 `configure --prefix=/home/andreas/local/emacs-24.5'

Important settings:
  value of $LC_COLLATE: en_US.UTF-8
  value of $LC_CTYPE: en_US.UTF-8
  value of $LC_MESSAGES: en_US.UTF-8
  value of $LANG: en_US.UTF-8
  value of $XMODIFIERS: @im=ibus
  locale-coding-system: utf-8-unix

Major mode: Group

Minor modes in effect:
  gnus-topic-mode: t
  gnus-undo-mode: t
  show-paren-mode: t
  TeX-PDF-mode: t
  global-auto-complete-mode: t
  tooltip-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  buffer-read-only: t
  column-number-mode: t
  line-number-mode: t
  transient-mark-mode: t
  abbrev-mode: t


Load-path shadows:
None found.

Features:
(shadow flyspell ispell nnir emacsbug misearch multi-isearch derived
apropos help-mode thingatpt ac-etags etags lua-mode rx flow-fill
mule-util w3m-form w3m browse-url doc-view jka-compr dired image-mode
w3m-hist w3m-fb bookmark-w3m w3m-ems w3m-ccl ccl w3m-favicon w3m-image
w3m-proc w3m-util mm-archive qp vc-dispatcher vc-svn sort gnus-cite
mail-extr gnus-async gnus-bcklg gnus-ml disp-table gnus-topic nndraft
nnmh nnfolder parse-time bbdb-gnus bbdb-mua bbdb-com netrc
network-stream auth-source eieio byte-opt bytecomp byte-compile cl-extra
cconv eieio-core starttls tls gnus-agent gnus-srvr gnus-score score-mode
nnvirtual gnus-msg gnus-art mm-uu mml2015 mm-view mml-smime smime
password-cache dig mailcap nntp gnus-cache gnus-sum nnoo gnus-group
gnus-undo nnmail mail-source gnus-start gnus-spec gnus-int gnus-range
message sendmail format-spec rfc822 mml mml-sec mm-decode mm-bodies
mm-encode mail-parse rfc2231 rfc2047 rfc2045 ietf-drums mailabbrev
gmm-utils mailheader gnus-win gnus gnus-ems nnheader gnus-util
mail-utils mm-util mail-prsvr wid-edit flymake compile comint ansi-color
ring paren cus-start cus-load bbdb bbdb-site timezone git-messenger
auto-complete-auctex latex easy-mmode tex-style tex dbus xml crm advice
help-fns mmm-noweb mmm-mode mmm-univ mmm-class mmm-region mmm-auto
mmm-vars mmm-utils mmm-compat cl auto-complete-config auto-complete
edmacro kmacro cl-macs gv popup cl-loaddefs cl-lib tex-site info
easymenu package epg-config time-date tooltip electric uniquify
ediff-hook vc-hooks lisp-float-type mwheel x-win x-dnd tool-bar dnd
fontset image regexp-opt fringe tabulated-list newcomment lisp-mode
prog-mode register page menu-bar rfn-eshadow timer select scroll-bar
mouse jit-lock font-lock syntax facemenu font-core frame cham georgian
utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean
japanese hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese case-table epa-hook jka-cmpr-hook help simple abbrev
minibuffer nadvice loaddefs button faces cus-face macroexp files
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote make-network-process
dbusbind gfilenotify dynamic-setting system-font-setting
font-render-setting move-toolbar gtk x-toolkit x multi-tty emacs)

Memory information:
((conses 16 463756 69676)
 (symbols 48 162620 0)
 (miscs 40 205 925)
 (strings 32 181716 11817)
 (string-bytes 1 4773158)
 (vectors 16 34424)
 (vector-slots 8 759595 16405)
 (floats 8 357 427)
 (intervals 56 17755 62)
 (buffers 960 35)
 (heap 1024 70832 3857))

bug#21934: 24.5; find-tag: reading TAGS file incorrectly

[Dmitry Gutov]

Eli, please take a look at TAGS attached to this bug report. Do the 
entries there satisfy the "implicit name" conditions?

etags that comes with Emacs outputs a TAGS file with explicit tags for 
these functions, and that naturally works with find-tag.

But I wonder if the example TAGS (produced by a different program, 
apparently) is also valid. And if so, why Emacs's etags doesn't use the 
implicit tags.

bug#21934: 24.5; find-tag: reading TAGS file incorrectly

[Stephen Leake]

Andreas Matthias <andreas.matthias@gmail.com> writes:

> Attached is a lua-file with its corresponding TAGS file. 

What program produced the TAGS file?


-- 
-- Stephe

bug#21934: 24.5; find-tag: reading TAGS file incorrectly

[Eli Zaretskii]

> Cc: Andreas Matthias <andreas.matthias@gmail.com>
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Tue, 17 Nov 2015 06:01:27 +0200
> 
> Eli, please take a look at TAGS attached to this bug report.

Will do.  It could take a couple of days, though, before I have enough
time for that.

bug#21934: 24.5; find-tag: reading TAGS file incorrectly

[Andreas Matthias]

Stephen Leake wrote:

> Andreas Matthias <andreas.matthias@gmail.com> writes:
>
>> Attached is a lua-file with its corresponding TAGS file. 
>
> What program produced the TAGS file?

It's etags from the emacs distribution.

Andreas

bug#21934: 24.5; find-tag: reading TAGS file incorrectly

[Andreas Matthias]

Dmitry Gutov wrote:

> Eli, please take a look at TAGS attached to this bug report. Do the entries
> there satisfy the "implicit name" conditions?
>
> etags that comes with Emacs outputs a TAGS file with explicit tags for these
> functions, and that naturally works with find-tag.
>
> But I wonder if the example TAGS (produced by a different program, apparently)
> is also valid. And if so, why Emacs's etags doesn't use the implicit tags.

The TAGS file was created with etags from the emacs distribution.

But I do not know the difference between implicit and explicit tags, sorry.


Andreas

bug#21934: 24.5; find-tag: reading TAGS file incorrectly

[Dmitry Gutov]

On 11/17/2015 07:21 PM, Andreas Matthias wrote:

> The TAGS file was created with etags from the emacs distribution.

Not the version from master, though?

bug#21934: 24.5; find-tag: reading TAGS file incorrectly

[Andreas Matthias]

Dmitry Gutov wrote:

> On 11/17/2015 07:21 PM, Andreas Matthias wrote:
>
>> The TAGS file was created with etags from the emacs distribution.
>
> Not the version from master, though?

Tested with:

  etags (GNU Emacs 24.3)

  etags (GNU Emacs 24.5)

bug#21934: 24.5; find-tag: reading TAGS file incorrectly

[Dmitry Gutov]

On 11/17/2015 07:40 PM, Andreas Matthias wrote:

> Tested with:
>
>    etags (GNU Emacs 24.3)
>
>    etags (GNU Emacs 24.5)

Could you test with emacs from the branch emacs-25? We've had some 
major-ish changes in etags recently.

bug#21934: 24.5; find-tag: reading TAGS file incorrectly

[Andreas Matthias]

Dmitry Gutov wrote:

> On 11/17/2015 07:40 PM, Andreas Matthias wrote:
>
>> Tested with:
>>
>>    etags (GNU Emacs 24.3)
>>
>>    etags (GNU Emacs 24.5)
>
> Could you test with emacs from the branch emacs-25? We've had some major-ish
> changes in etags recently.

commit 58e6235007e6761fb9734b942ecff94bf4e9ba68

etags (GNU Emacs 25.1.50)

This one creates the same TAGS file... Hmm... Am I doing something wrong...?


Andreas

bug#21934: 24.5; find-tag: reading TAGS file incorrectly

[Dmitry Gutov]

On 11/17/2015 09:38 PM, Andreas Matthias wrote:

> This one creates the same TAGS file... Hmm... Am I doing something wrong...?

I'm terribly sorry, I had it backwards: Emacs's etags generates TAGS 
file just like you submitted.

I also have a different etags in my system, that comes from Exuberant 
Ctags, and it generates a different TAGS, looking like this:

test.lua,98
function Rectangle.getPos ()^?Rectangle.getPos ^A2,15
function Circle.getPos ()^?Circle.getPos ^A6,61

And that one works with `find-tag' just fine. So it would be nice if Eli 
could comment on the difference, and whether etags should be patched 
instead.

bug#21934: 24.5; find-tag: reading TAGS file incorrectly

[Eli Zaretskii]

> Cc: Andreas Matthias <andreas.matthias@gmail.com>
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Tue, 17 Nov 2015 06:01:27 +0200

Sorry for the delay; Life™ intervened.

> Eli, please take a look at TAGS attached to this bug report. Do the 
> entries there satisfy the "implicit name" conditions?

Yes, they do.  Etags doesn't treat a period '.' as ending an
identifier, except in C-like languages.  And that is good, since Lua
evidently wants to support identifiers with embedded periods.

> I also have a different etags in my system, that comes from Exuberant 
> Ctags, and it generates a different TAGS, looking like this:
> 
> test.lua,98
> function Rectangle.getPos ()^?Rectangle.getPos ^A2,15
> function Circle.getPos ()^?Circle.getPos ^A6,61
> 
> And that one works with `find-tag' just fine. So it would be nice if Eli 
> could comment on the difference, and whether etags should be patched 
> instead.

I've looked at the related code, and my conclusion is that there's
more to this than meets the eye.

The OP complained about _completion_ on tag names, and suggested to
fix a regexp used by etags-tags-completion-table.  That regexp indeed
doesn't allow a period in an identifier name (probably because it's
disallowed in C-like languages).  However, find-tag itself doesn't use
that regexp, so typing "M-x find-tag RET Rectangle.getPos RET" finds
that identifier with no problems.

Now, find-tag is deprecated in Emacs 25, and M-. invokes
xref-find-definitions instead.  AFAIU, etags-tags-completion-table is
no longer relevant with xref.  xref-find-definitions, if it's invoked
with a prefix argument, and if you type "Rectangle.getPos RET" at its
prompt, not surprisingly also finds the identifier.  Trying to invoke
completion after "C-u M-.", with test.lua as the current buffer,
doesn't succeed to complete even on Rectangle, so the situation here
is somewhat worse, but I'm not sure why.

If we want "M-." without prefix argument to be able to find these
identifiers, we need first to take care of how it determines the
symbol at point.  Currently, it calls (thing-at-point 'symbol), which
predictably guesses wrong.

IOW, we could fix the regexp as suggested by the OP (AFAICS, that
should not cause any regressions for etags.el), but that won't solve
the problems "M-." in Emacs 25 will have with such identifiers in Lua
sources.

Suggestions and comments are welcome.

bug#21934: 24.5; find-tag: reading TAGS file incorrectly

[Dmitry Gutov]

On 11/21/2015 03:07 PM, Eli Zaretskii wrote:

> Yes, they do.  Etags doesn't treat a period '.' as ending an
> identifier, except in C-like languages.  And that is good, since Lua
> evidently wants to support identifiers with embedded periods.

In that case, the submitted patch would indeed make things better. 
However, I wonder if we should rewrite the whole regexp in 
etags-tags-completion-table, to be more faithful to the "implicit tag" 
spec, and instead of whitelisting characters that can be used in tags, 
allow any character in NONAM.

(Or rather, I wonder why it hasn't been written that way to begin with, 
and whether there are some performance pitfalls in doing so).

> I've looked at the related code, and my conclusion is that there's
> more to this than meets the eye.

Indeed.

> The OP complained about _completion_ on tag names, and suggested to
> fix a regexp used by etags-tags-completion-table.  That regexp indeed
> doesn't allow a period in an identifier name (probably because it's
> disallowed in C-like languages).  However, find-tag itself doesn't use
> that regexp,

If does, for completion. Type M-x find-tag RET TAB, and you'll see the 
result of calling etags-tags-completion-table.

> so typing "M-x find-tag RET Rectangle.getPos RET" finds
> that identifier with no problems.

That's right: the logic to "list all tags" and to "find tag named xyz" 
is implemented in different places. And the latter is more faithful to 
the "implicit tag name" definition; it's the completion table logic that 
needs to be fixed.

> Now, find-tag is deprecated in Emacs 25, and M-. invokes
> xref-find-definitions instead.  AFAIU, etags-tags-completion-table is
> no longer relevant with xref.

It's (almost) just as relevant: type C-u M-. TAB.

> xref-find-definitions, if it's invoked
> with a prefix argument, and if you type "Rectangle.getPos RET" at its
> prompt, not surprisingly also finds the identifier.  Trying to invoke
> completion after "C-u M-.", with test.lua as the current buffer,
> doesn't succeed to complete even on Rectangle, so the situation here
> is somewhat worse, but I'm not sure why.

Is it really any different? M-x find-tag RET TAB doesn't show anything 
beginning with "Rectangle" either. I only get "getPos" as a completion 
either way.

> If we want "M-." without prefix argument to be able to find these
> identifiers, we need first to take care of how it determines the
> symbol at point.  Currently, it calls (thing-at-point 'symbol), which
> predictably guesses wrong.

Right, I haven't thought about it.

But what else, really, could (xref-backend-identifier-at-point 'etags) 
return here? It only knows the current buffer's syntax table, and that 
its data source is etags.

Either etags will have to consider that in both example declarations the 
tag name must be "getPos", not "xxx.getPos", and put this tag name 
explicitly into the entries, or lua-mode will have to change the syntax 
class of "." to "symbol", so that (thing-at-point 'symbol) returns 
"Circle.getPos" as the tag name.

That is, of course, if "." always goes after the name of the 
class/module/etc that "getPos" is defined in, and it can't follow a 
variable name. I'm not familiar with Lua's syntax.

bug#21934: 24.5; find-tag: reading TAGS file incorrectly

[Dmitry Gutov]

On 11/22/2015 03:17 AM, Dmitry Gutov wrote:

> (Or rather, I wonder why it hasn't been written that way to begin with,
> and whether there are some performance pitfalls in doing so).

Pushed to emacs-25. So far, the result is uniformly positive, with up to 
25% speedup (in the Linux Kernel case).

Unless it causes a problem somewhere else, I believe this bug can be 
closed (but the accompanying discussion may well be continued).

bug#21934: 24.5; find-tag: reading TAGS file incorrectly

[Andreas Matthias]

Dmitry Gutov wrote:

> That is, of course, if "." always goes after the name of the class/module/etc
> that "getPos" is defined in, and it can't follow a variable name. I'm not
> familiar with Lua's syntax.

In the example given Rectangle is a data structure called table and a
table is an associative array. In Lua you can put variables and
functions into a table. So Rectangle.getPos() is the function getPos()
of table Rectangle. Though Lua does not know the concept of classes, it
is easy to model object-oriented behavior by means of tables.

In addition to the dot-operator there's also a colon-operator in Lua which
acts like the dot-operator but hides the self/this parameter of OOP.

bug#21934: 24.5; find-tag: reading TAGS file incorrectly

[Dmitry Gutov]

On 11/22/2015 04:08 PM, Andreas Matthias wrote:

> In the example given Rectangle is a data structure called table and a
> table is an associative array. In Lua you can put variables and
> functions into a table. So Rectangle.getPos() is the function getPos()
> of table Rectangle.

So I think what you're saying is lua-mode should add "." to the 
syntax-class "symbol". However:

> Though Lua does not know the concept of classes, it
> is easy to model object-oriented behavior by means of tables.
>
> In addition to the dot-operator there's also a colon-operator in Lua which
> acts like the dot-operator but hides the self/this parameter of OOP.

Is it usual that, when defining classes that way, you *will* define 
methods using the dot notation, and then later use them with an 
"instance variable", using the colon notation? Like in this article:

http://www.lua.org/pil/16.html

You define the method with "function Account.withdraw (self, v)",
and then use it in "a1.withdraw(a1, 100.00)".

It seems that etags should at least output two tag names for this 
declaration: both "Account.withdraw" and just "withdraw".

bug#21934: 24.5; find-tag: reading TAGS file incorrectly

[Dmitry Gutov]

On 11/22/2015 04:33 PM, Dmitry Gutov wrote:

> You define the method with "function Account.withdraw (self, v)",
> and then use it in "a1.withdraw(a1, 100.00)".

Sorry, I meant "a:withdraw(100.00)".

"a1.withdraw" doesn't look like it's used often, from the description. 
But feel free to correct me there, because if it is used, then "." 
mustn't be a symbol constituent.

bug#21934: 24.5; find-tag: reading TAGS file incorrectly

[Andreas Matthias]

Dmitry Gutov wrote:

> On 11/22/2015 04:08 PM, Andreas Matthias wrote:
>
>> In the example given Rectangle is a data structure called table and a
>> table is an associative array. In Lua you can put variables and
>> functions into a table. So Rectangle.getPos() is the function getPos()
>> of table Rectangle.
>
> So I think what you're saying is lua-mode should add "." to the syntax-class
> "symbol". However:

I'm sorry, I'm not familiar with emacs internals like the syntax table.

>> Though Lua does not know the concept of classes, it
>> is easy to model object-oriented behavior by means of tables.
>>
>> In addition to the dot-operator there's also a colon-operator in Lua which
>> acts like the dot-operator but hides the self/this parameter of OOP.
>
> Is it usual that, when defining classes that way, you *will* define methods
> using the dot notation, and then later use them with an "instance variable",
> using the colon notation? Like in this article:
>
> http://www.lua.org/pil/16.html
>
> You define the method with "function Account.withdraw (self, v)",
> and then use it in "a1.withdraw(a1, 100.00)".

Tables are the main data structure of Lua. Although the dot operator
can be used in the sense of OOP, more often than not the dot operator
is just used to access elements of a table.

> It seems that etags should at least output two tag names for this declaration:
> both "Account.withdraw" and just "withdraw".

Maybe. But how do you handle getPos() from the example which exists
twice, once in table Rectangle and once in table Circle?

bug#21934: 24.5; find-tag: reading TAGS file incorrectly

[Dmitry Gutov]

On 11/22/2015 05:06 PM, Andreas Matthias wrote:

>> So I think what you're saying is lua-mode should add "." to the syntax-class
>> "symbol". However:
>
> I'm sorry, I'm not familiar with emacs internals like the syntax table.

The above would mean that (thing-at-point 'symbol) will return 
`Rectangle.getPos', and not just `getPos'.

So when you press M-., that's what xref-find-definitions (or find-tag) 
will be searching for.

> Tables are the main data structure of Lua. Although the dot operator
> can be used in the sense of OOP, more often than not the dot operator
> is just used to access elements of a table.

If you use anonymous tables as well, and copy/inherit/modify them, like

   a = {
     withdraw = function(self, v)
       self.balance = self.balance - v
     end
   }

   b = a.copy()

   b:withdraw(10)

then having "widthdraw" as the tag name seems useful.

> Maybe. But how do you handle getPos() from the example which exists
> twice, once in table Rectangle and once in table Circle?

You add both entries to TAGS, one after another. Yes, it will double the 
size of the TAGS file, more or less.

I think we already do that by default for C++.

bug#21934: 24.5; find-tag: reading TAGS file incorrectly

[Eli Zaretskii]

> Cc: 21934@debbugs.gnu.org, andreas.matthias@gmail.com
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sun, 22 Nov 2015 03:17:09 +0200
> 
> > The OP complained about _completion_ on tag names, and suggested to
> > fix a regexp used by etags-tags-completion-table.  That regexp indeed
> > doesn't allow a period in an identifier name (probably because it's
> > disallowed in C-like languages).  However, find-tag itself doesn't use
> > that regexp,
> 
> If does, for completion. Type M-x find-tag RET TAB, and you'll see the 
> result of calling etags-tags-completion-table.

Completion is not an integral part of find-tag.

> > Now, find-tag is deprecated in Emacs 25, and M-. invokes
> > xref-find-definitions instead.  AFAIU, etags-tags-completion-table is
> > no longer relevant with xref.
> 
> It's (almost) just as relevant: type C-u M-. TAB.

As I said, completion in M-. in Emacs 25 works worse than in find-tag
in this case: it doesn't even succeed to complete "Rec TAB" into
"Rectangle".  I don't know why.

> > xref-find-definitions, if it's invoked
> > with a prefix argument, and if you type "Rectangle.getPos RET" at its
> > prompt, not surprisingly also finds the identifier.  Trying to invoke
> > completion after "C-u M-.", with test.lua as the current buffer,
> > doesn't succeed to complete even on Rectangle, so the situation here
> > is somewhat worse, but I'm not sure why.
> 
> Is it really any different? M-x find-tag RET TAB doesn't show anything 
> beginning with "Rectangle" either. I only get "getPos" as a completion 
> either way.

It's different if you type "Rec TAB".

> Either etags will have to consider that in both example declarations the 
> tag name must be "getPos", not "xxx.getPos", and put this tag name 
> explicitly into the entries, or lua-mode will have to change the syntax 
> class of "." to "symbol", so that (thing-at-point 'symbol) returns 
> "Circle.getPos" as the tag name.

Something like that, yes.

My main point is that it's easy to solve the completion case, but that
hardly help in using TAGS with Lua and the new xref commands.
Something else is missing.

bug#21934: 24.5; find-tag: reading TAGS file incorrectly

[Andreas Matthias]

Dmitry Gutov wrote:

> On 11/22/2015 05:06 PM, Andreas Matthias wrote:
>
>>> So I think what you're saying is lua-mode should add "." to the syntax-class
>>> "symbol". However:
>>
>> I'm sorry, I'm not familiar with emacs internals like the syntax table.
>
> The above would mean that (thing-at-point 'symbol) will return
> `Rectangle.getPos', and not just `getPos'.
>
> So when you press M-., that's what xref-find-definitions (or find-tag) will be
> searching for.

If you use the table as a "normal" table, you access it like:

   Retangle.getPos()

If you use it for OOP then you would access the function through an
object like:

   myRec.getPos()

In the first case returning `Rectangle.getPos' would be usefull, in the
second case just `getPos'.


I hope I don't get lost in this discussion. It's hard to distinguish
between "complete the tag" and "find the tag"...

bug#21934: 24.5; find-tag: reading TAGS file incorrectly

[Dmitry Gutov]

On 11/22/2015 06:41 PM, Andreas Matthias wrote:

> I hope I don't get lost in this discussion. It's hard to distinguish
> between "complete the tag" and "find the tag"...

Hopefully, complet-able tags and find-able tags are the same set now, 
after the latest change.

bug#21934: 24.5; find-tag: reading TAGS file incorrectly

[Dmitry Gutov]

On 11/22/2015 06:12 PM, Eli Zaretskii wrote:
> Completion is not an integral part of find-tag.

It's not an integral part of xref-find-definitions either.

> As I said, completion in M-. in Emacs 25 works worse than in find-tag
> in this case: it doesn't even succeed to complete "Rec TAB" into
> "Rectangle".  I don't know why.

I didn't see any difference between them. Anyway, it should be fixed now.

>> Is it really any different? M-x find-tag RET TAB doesn't show anything
>> beginning with "Rectangle" either. I only get "getPos" as a completion
>> either way.
>
> It's different if you type "Rec TAB".

If I revert 51fd4a01395885077909c60b17ae3d7d42b8bb0a and reopen the TAGS 
file, the only completion I get is "getPos", either way.

It *is* different if you type "Rec RET", however, because in the end 
"Rec" gets a match via `tag-any-match-p', which 
etags--xref-find-definitions doesn't use.

But the user can add it to etags-xref-find-definitions-tag-order, if 
they want.

> My main point is that it's easy to solve the completion case, but that
> hardly help in using TAGS with Lua and the new xref commands.
> Something else is missing.

I believe I've addressed that.

And if I understand Andreas right, etags should output two tags for each 
such definition. Just like we do for C++.

bug#21934: 24.5; find-tag: reading TAGS file incorrectly

[Eli Zaretskii]

> Cc: andreas.matthias@gmail.com, 21934@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sun, 22 Nov 2015 18:54:40 +0200
> 
> if I understand Andreas right, etags should output two tags for each
> such definition. Just like we do for C++.

If that's the conclusion, I think I can make etags do that for Lua.

Should that be done for tokens that include '.' and ':'?  Can there be
more than one of these in a token, and if so, what should etags do?
IOW, if we have foo.bar.baz or foo:bar.baz etc., what should be the
result?

bug#21934: 24.5; find-tag: reading TAGS file incorrectly

[Dmitry Gutov]

On 11/22/2015 07:38 PM, Eli Zaretskii wrote:

> If that's the conclusion, I think I can make etags do that for Lua.
>
> Should that be done for tokens that include '.' and ':'?

I think so.

> Can there be
> more than one of these in a token, and if so, what should etags do?
> IOW, if we have foo.bar.baz or foo:bar.baz etc., what should be the
> result?

Just two, I think. foo.bar is not a function, and bar.baz probably won't 
be a "symbol at point".

So just the fully qualified tag name, and the local tag name ("baz").

bug#21934: 24.5; find-tag: reading TAGS file incorrectly

[Andreas Matthias]

Dmitry Gutov wrote:

> On 11/22/2015 07:38 PM, Eli Zaretskii wrote:
>
>> If that's the conclusion, I think I can make etags do that for Lua.
>>
>> Should that be done for tokens that include '.' and ':'?
>
> I think so.

Yes, both.

>> Can there be
>> more than one of these in a token, and if so, what should etags do?
>> IOW, if we have foo.bar.baz or foo:bar.baz etc., what should be the
>> result?
>
> Just two, I think. foo.bar is not a function, and bar.baz probably won't be a
> "symbol at point".
>
> So just the fully qualified tag name, and the local tag name ("baz").

Yes, that's reasonable.

bug#21934: 24.5; find-tag: reading TAGS file incorrectly

[Eli Zaretskii]

> Cc: andreas.matthias@gmail.com, 21934@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sun, 22 Nov 2015 19:43:48 +0200
> 
> On 11/22/2015 07:38 PM, Eli Zaretskii wrote:
> 
> > If that's the conclusion, I think I can make etags do that for Lua.
> >
> > Should that be done for tokens that include '.' and ':'?
> 
> I think so.
> 
> > Can there be
> > more than one of these in a token, and if so, what should etags do?
> > IOW, if we have foo.bar.baz or foo:bar.baz etc., what should be the
> > result?
> 
> Just two, I think. foo.bar is not a function, and bar.baz probably won't 
> be a "symbol at point".

Sorry, I'm not sure I understand: does Lua allow syntax such as above,
or doesn't it?  If it allows that, then what is the meaning of those?
Can a table has a function that is also a table?  (Apologies if I'm
talking nonsense.)

bug#21934: 24.5; find-tag: reading TAGS file incorrectly

[Eli Zaretskii]

> From: Andreas Matthias <andreas.matthias@gmail.com>
> Cc: Eli Zaretskii <eliz@gnu.org>,  21934@debbugs.gnu.org
> Date: Sun, 22 Nov 2015 18:49:32 +0100
> 
> >> Can there be
> >> more than one of these in a token, and if so, what should etags do?
> >> IOW, if we have foo.bar.baz or foo:bar.baz etc., what should be the
> >> result?
> >
> > Just two, I think. foo.bar is not a function, and bar.baz probably won't be a
> > "symbol at point".
> >
> > So just the fully qualified tag name, and the local tag name ("baz").
> 
> Yes, that's reasonable.

I'm sorry: can I have a complete specification, please?  Given a
token that includes one or more of '.' and ':', how to determine the 2
tags that etags should produce?

TIA

bug#21934: 24.5; find-tag: reading TAGS file incorrectly

[Andreas Matthias]

Eli Zaretskii wrote:

>> Cc: andreas.matthias@gmail.com, 21934@debbugs.gnu.org
>> From: Dmitry Gutov <dgutov@yandex.ru>
>> Date: Sun, 22 Nov 2015 19:43:48 +0200
>> 
>> On 11/22/2015 07:38 PM, Eli Zaretskii wrote:
>> 
>> > If that's the conclusion, I think I can make etags do that for Lua.
>> >
>> > Should that be done for tokens that include '.' and ':'?
>> 
>> I think so.
>> 
>> > Can there be
>> > more than one of these in a token, and if so, what should etags do?
>> > IOW, if we have foo.bar.baz or foo:bar.baz etc., what should be the
>> > result?
>> 
>> Just two, I think. foo.bar is not a function, and bar.baz probably won't 
>> be a "symbol at point".
>
> Sorry, I'm not sure I understand: does Lua allow syntax such as above,
> or doesn't it?  If it allows that, then what is the meaning of those?
> Can a table has a function that is also a table?  (Apologies if I'm
> talking nonsense.)

Yes, it's valid Lua. You can stick one table into an element of another table.

E.g.: foo.bar.baz

This is table `foo' which has got an element called `bar'; and `bar'
happens to be a table, which has got an element `baz'.


E.g.: foo:bar.baz

A class `foo' with a table `bar' containing an element called `baz'

bug#21934: 24.5; find-tag: reading TAGS file incorrectly

[Dmitry Gutov]

On 11/22/2015 08:27 PM, Andreas Matthias wrote:

> E.g.: foo:bar.baz
>
> A class `foo' with a table `bar' containing an element called `baz'

So, "foo:bar" is a valid notation, even when it's not a function call, 
and even when 'bar' is not a function? Is it any different from 
"foo.bar" then?

bug#21934: 24.5; find-tag: reading TAGS file incorrectly

[Andreas Matthias]

Dmitry Gutov wrote:

> On 11/22/2015 08:27 PM, Andreas Matthias wrote:
>
>> E.g.: foo:bar.baz
>>
>> A class `foo' with a table `bar' containing an element called `baz'
>
> So, "foo:bar" is a valid notation, even when it's not a function call, and even
> when 'bar' is not a function? Is it any different from "foo.bar" then?

You are right, that was nonsense. The : must be the last operator
because it inserts the `self' or `this' element into the following
function call.

bug#21934: 24.5; find-tag: reading TAGS file incorrectly

[Andreas Matthias]

Eli Zaretskii wrote:

>> From: Andreas Matthias <andreas.matthias@gmail.com>
>> Cc: Eli Zaretskii <eliz@gnu.org>,  21934@debbugs.gnu.org
>> Date: Sun, 22 Nov 2015 18:49:32 +0100
>> 
>> >> Can there be
>> >> more than one of these in a token, and if so, what should etags do?
>> >> IOW, if we have foo.bar.baz or foo:bar.baz etc., what should be the
>> >> result?
>> >
>> > Just two, I think. foo.bar is not a function, and bar.baz probably won't be a
>> > "symbol at point".
>> >
>> > So just the fully qualified tag name, and the local tag name ("baz").
>> 
>> Yes, that's reasonable.
>
> I'm sorry: can I have a complete specification, please?  Given a
> token that includes one or more of '.' and ':', how to determine the 2
> tags that etags should produce?

See `funcname' in:

http://www.lua.org/manual/5.3/manual.html#9

bug#21934: 24.5; find-tag: reading TAGS file incorrectly

[Eli Zaretskii]

> From: Andreas Matthias <andreas.matthias@gmail.com>
> Cc: dgutov@yandex.ru,  21934@debbugs.gnu.org
> Date: Mon, 23 Nov 2015 17:50:40 +0100
> 
> Eli Zaretskii wrote:
> 
> >> From: Andreas Matthias <andreas.matthias@gmail.com>
> >> Cc: Eli Zaretskii <eliz@gnu.org>,  21934@debbugs.gnu.org
> >> Date: Sun, 22 Nov 2015 18:49:32 +0100
> >> 
> >> >> Can there be
> >> >> more than one of these in a token, and if so, what should etags do?
> >> >> IOW, if we have foo.bar.baz or foo:bar.baz etc., what should be the
> >> >> result?
> >> >
> >> > Just two, I think. foo.bar is not a function, and bar.baz probably won't be a
> >> > "symbol at point".
> >> >
> >> > So just the fully qualified tag name, and the local tag name ("baz").
> >> 
> >> Yes, that's reasonable.
> >
> > I'm sorry: can I have a complete specification, please?  Given a
> > token that includes one or more of '.' and ':', how to determine the 2
> > tags that etags should produce?
> 
> See `funcname' in:
> 
> http://www.lua.org/manual/5.3/manual.html#9

Thanks, but that's not all the information I need.  What's missing is
which part(s) of a funcname would you like etags to record.  Do you
want each "Name" component to be recorded, or just some of them?

IOW, if we have foo.bar.baz.more, what should be recorded in addition
to the whole foo.bar.baz.more token?  Just bar.baz.more, or something
else?  What about foo.bar.baz:more?

bug#21934: 24.5; find-tag: reading TAGS file incorrectly

[Andreas Matthias]

Eli Zaretskii wrote:

>> From: Andreas Matthias <andreas.matthias@gmail.com>
>> Cc: dgutov@yandex.ru,  21934@debbugs.gnu.org
>> Date: Mon, 23 Nov 2015 17:50:40 +0100
>> 
>> Eli Zaretskii wrote:
>> 
>> >> From: Andreas Matthias <andreas.matthias@gmail.com>
>> >> Cc: Eli Zaretskii <eliz@gnu.org>,  21934@debbugs.gnu.org
>> >> Date: Sun, 22 Nov 2015 18:49:32 +0100
>> >> 
>> >> >> Can there be
>> >> >> more than one of these in a token, and if so, what should etags do?
>> >> >> IOW, if we have foo.bar.baz or foo:bar.baz etc., what should be the
>> >> >> result?
>> >> >
>> >> > Just two, I think. foo.bar is not a function, and bar.baz probably won't be a
>> >> > "symbol at point".
>> >> >
>> >> > So just the fully qualified tag name, and the local tag name ("baz").
>> >> 
>> >> Yes, that's reasonable.
>> >
>> > I'm sorry: can I have a complete specification, please?  Given a
>> > token that includes one or more of '.' and ':', how to determine the 2
>> > tags that etags should produce?
>> 
>> See `funcname' in:
>> 
>> http://www.lua.org/manual/5.3/manual.html#9
>
> Thanks, but that's not all the information I need.  What's missing is
> which part(s) of a funcname would you like etags to record.  Do you
> want each "Name" component to be recorded, or just some of them?
>
> IOW, if we have foo.bar.baz.more, what should be recorded in addition
> to the whole foo.bar.baz.more token?  Just bar.baz.more, or something
> else?  What about foo.bar.baz:more?

As Dmitry said. Both the fully qualified tag name and the local tag name.


foo.bar.baz.more should be tagged as:

   foo.bar.baz.more
   more


foo.bar.baz:more should be tagged as:

   foo.bar.baz:more
   more

bug#21934: 24.5; find-tag: reading TAGS file incorrectly

[Eli Zaretskii]

> From: Andreas Matthias <andreas.matthias@gmail.com>
> Cc: dgutov@yandex.ru,  21934@debbugs.gnu.org
> Date: Mon, 23 Nov 2015 18:28:14 +0100
> 
> foo.bar.baz.more should be tagged as:
> 
>    foo.bar.baz.more
>    more
> 
> 
> foo.bar.baz:more should be tagged as:
> 
>    foo.bar.baz:more
>    more

Thanks, will do.

bug#21934: 24.5; find-tag: reading TAGS file incorrectly

[Dmitry Gutov]

On 11/22/2015 03:17 AM, Dmitry Gutov wrote:

> But what else, really, could (xref-backend-identifier-at-point 'etags)
> return here? It only knows the current buffer's syntax table, and that
> its data source is etags.

I take that back: even thought it's not widely used, etags has the 
find-tag-default-function variable/major-mode-property, and starting 
with 5ffa4e3e3c853ed75e4f4b441e813252112f2d24 xref delegates to it.

It's a bit weird (looks at buffer contents before point if point is not 
at any symbol), but hopefully that won't be much of a problem.

bug#21934: 24.5; find-tag: reading TAGS file incorrectly

[Eli Zaretskii]

> Date: Mon, 23 Nov 2015 20:17:38 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: 21934@debbugs.gnu.org, dgutov@yandex.ru
> 
> > From: Andreas Matthias <andreas.matthias@gmail.com>
> > Cc: dgutov@yandex.ru,  21934@debbugs.gnu.org
> > Date: Mon, 23 Nov 2015 18:28:14 +0100
> > 
> > foo.bar.baz.more should be tagged as:
> > 
> >    foo.bar.baz.more
> >    more
> > 
> > 
> > foo.bar.baz:more should be tagged as:
> > 
> >    foo.bar.baz:more
> >    more
> 
> Thanks, will do.

Done in commit 690ccf0 on the emacs-25 branch.  Please see if the
results are satisfactory, and close the bug if so.

Thanks.

bug#21934: 24.5; find-tag: reading TAGS file incorrectly

[Andreas Matthias]

Eli Zaretskii wrote:

>> Date: Mon, 23 Nov 2015 20:17:38 +0200
>> From: Eli Zaretskii <eliz@gnu.org>
>> Cc: 21934@debbugs.gnu.org, dgutov@yandex.ru
>> 
>> > From: Andreas Matthias <andreas.matthias@gmail.com>
>> > Cc: dgutov@yandex.ru,  21934@debbugs.gnu.org
>> > Date: Mon, 23 Nov 2015 18:28:14 +0100
>> > 
>> > foo.bar.baz.more should be tagged as:
>> > 
>> >    foo.bar.baz.more
>> >    more
>> > 
>> > 
>> > foo.bar.baz:more should be tagged as:
>> > 
>> >    foo.bar.baz:more
>> >    more
>> 
>> Thanks, will do.
>
> Done in commit 690ccf0 on the emacs-25 branch.  Please see if the
> results are satisfactory, and close the bug if so.
>
> Thanks.

Looks very good to me. Thank you Eli.

Andreas

bug#21934: 24.5; find-tag: reading TAGS file incorrectly

[Andreas Matthias]

bug#21934: 24.5; find-tag: reading TAGS file incorrectly

[Eli Zaretskii]

> From: Andreas Matthias <andreas.matthias@gmail.com>
> Cc: 21934@debbugs.gnu.org,  dgutov@yandex.ru
> Date: Sat, 28 Nov 2015 18:25:25 +0100
> 
> Eli Zaretskii wrote:
> 
> >> Date: Mon, 23 Nov 2015 20:17:38 +0200
> >> From: Eli Zaretskii <eliz@gnu.org>
> >> Cc: 21934@debbugs.gnu.org, dgutov@yandex.ru
> >> 
> >> > From: Andreas Matthias <andreas.matthias@gmail.com>
> >> > Cc: dgutov@yandex.ru,  21934@debbugs.gnu.org
> >> > Date: Mon, 23 Nov 2015 18:28:14 +0100
> >> > 
> >> > foo.bar.baz.more should be tagged as:
> >> > 
> >> >    foo.bar.baz.more
> >> >    more
> >> > 
> >> > 
> >> > foo.bar.baz:more should be tagged as:
> >> > 
> >> >    foo.bar.baz:more
> >> >    more
> >> 
> >> Thanks, will do.
> >
> > Done in commit 690ccf0 on the emacs-25 branch.  Please see if the
> > results are satisfactory, and close the bug if so.
> >
> > Thanks.
> 
> Looks very good to me. Thank you Eli.

Thanks for testing.  I also added these cases to the etags test suite.

No further comments, so I'm closing the bug.