bug#28403: 25.2; find-tag works, but xref-find-definitions doesn't; bug?

bug#28403: 25.2; find-tag works, but xref-find-definitions doesn't; bug?

[Winston]

[Emacs 25.2; it looks (to me) like a regexp match pattern issue in
 xref.el, not an O/S issue.]

   Today I noticed the message about find-tag being (supposedly)
obsoleted by xref-find-definitions as of Emacs version 25.1, so I tried
the new command.  Unfortunately, the new one failed completely on an
entire class of function definitions.

   After some experimenting, it appears that the problem may be that
xref-find-definitions only works when the function definition is of the
form: name spacing* '('.

   The C code in question uses macros around function arguments in its
definitions.  E.g.,

        name _ARGS1(type,variable)

   find-tag (and etags) work just fine with that, and such function
definition lines appear in the TAGS file (as they should), but
xref-find-definitions fails to find such function tags, saying instead
"No definitions found for: name".

   Changing the function definition line to

        name (type variable)

as a test, re-running etags, and reloading TAGS, xref-find-definitions
found the tag and went to it.

   So, xref-find-definitions is not yet a complete replacement for
find-tag.  Since etags puts such lines in TAGS and xref-find-definitions
is unable to match up the name with the tag, it looks like a bug /
deficiency in xref-find-definitions.
 -WBE

bug#28403: 25.2; find-tag works, but xref-find-definitions doesn't; bug?

[Drew Adams]

> Today I noticed the message about find-tag being (supposedly)
> obsoleted by xref-find-definitions as of Emacs version 25.1,

Does it really say that `find-tag' is "obsolete"?  Ah, yes
it does.  I'm surprised.  That's too bad.

We couldn't just have *added* the `xref' stuff, as an
alternative?  Bad enough that keys bound by default to other
commands were immediately sacrificed to xref, but deprecating
those other commands?

> So, xref-find-definitions is not yet a complete replacement
> for find-tag.  Since etags puts such lines in TAGS and
> xref-find-definitions is unable to match up the name with
> the tag, it looks like a bug / deficiency in
> xref-find-definitions.

Dommage.  On n'arrete pas le progres...

bug#28403: 25.2; find-tag works, but xref-find-definitions doesn't; bug?

[Dmitry Gutov]

On 9/10/17 1:39 AM, Winston wrote:

>     The C code in question uses macros around function arguments in its
> definitions.  E.g.,
> 
>          name _ARGS1(type,variable)
> 
>     find-tag (and etags) work just fine with that, and such function
> definition lines appear in the TAGS file (as they should), but
> xref-find-definitions fails to find such function tags, saying instead
> "No definitions found for: name".

Which program are you generating TAGS with? Is it etags that comes with 
Emacs? xref-find-definitions is somewhat stricter about its input than 
find-tag.

What does the entry for this function inside TAGS look like? You can 
paste it into the reply. The TAGS format is close to plain text.

I'm guessing it looks like:

name _ARGS1(

which is an "implicit tag name" entry for "_ARGS1", but not for "name". 
IOW, etags doesn't understand macros.

>     So, xref-find-definitions is not yet a complete replacement for
> find-tag.  Since etags puts such lines in TAGS and xref-find-definitions
> is unable to match up the name with the tag, it looks like a bug /
> deficiency in xref-find-definitions.

Try adding `tag-symbol-match-p' to 
etags-xref-find-definitions-tag-order. This example should work then, 
but you'll get more false positives (like treating return types as 
function names).

bug#28403: 25.2; find-tag works, but xref-find-definitions

[Winston]

I previously wrote:
>>     The C code in question uses macros around function arguments in its
>> definitions.  E.g.,
>> 
>>          name _ARGS1(type,variable)
>> 
>>     find-tag (and etags) work just fine with that, and such function
>> definition lines appear in the TAGS file (as they should), but
>> xref-find-definitions fails to find such function tags, saying instead
>> "No definitions found for: name".

Dmitry kindly replied:
> Which program are you generating TAGS with? Is it etags that comes with 
> Emacs?

Yes, and "etags --version" prints: etags (GNU Emacs 25.2)

> xref-find-definitions is somewhat stricter about its input than 
> find-tag.

Yes, that's what's causing the difference.  ;-)

> What does the entry for this function inside TAGS look like? [...]
> I'm guessing it looks like:
> 
> name _ARGS1(

Exactly.  E.g.,
name _ARGS1(^?188,5710

> which is an "implicit tag name" entry for "_ARGS1", but not for "name". 
> IOW, etags doesn't understand macros.

   Whether etags understands macros or not, it is correctly identifying
the lines containing function names, so I see no problem there.

Addressing the difference between find-tag and xref-find-definitions:

   find-tag acts as if it uses "^\([^(]+\) *(" [or similar] and
considers the function name to be &1.  I.e., it treats the entire string
"name _ARGS1" as the function name.  When I change the number of
arguments, e.g., "name _ARGS2(...)" and haven't yet updated TAGS,
find-tag reports that it searched for "name _ARGS1" and didn't find it.

   Sure, it would be great if find-tag knew that only the part before
the space is the function name, but *FOR THE PURPOSE OF MAKING
xref-find-definitions WORK AS WELL AS find-tag*, treating [^(]+ (the
entire string up to the '(', or the entire string between the return
value type and the '(' if find-tag is that smart) as the function name
looks like it would do well enough to allow xref-find-definitions to
obsolete find-tag.

>>     So, xref-find-definitions is not yet a complete replacement for
>> find-tag.  Since etags puts such lines in TAGS and xref-find-definitions
>> is unable to match up the name with the tag, it looks like a bug /
>> deficiency in xref-find-definitions.

> Try adding `tag-symbol-match-p' to 
> etags-xref-find-definitions-tag-order. This example should work then, 
> but you'll get more false positives (like treating return types as 
> function names).

   Noted for future reference...

   Since doing that doesn't change what etags writes to TAGS, I'm not
sure how that elisp change would result in function return types being
matched as function names, but no matter.  For the moment I think I'll
just continue to use find-tag and hope that xref-find-definitions will
eventually work as well as find-tag before find-tag disappears.  :)

   Thanks,
 -WBE

bug#28403: 25.2; find-tag works, but xref-find-definitions

[Dmitry Gutov]

On 9/10/17 5:49 AM, Winston wrote:

> Yes, and "etags --version" prints: etags (GNU Emacs 25.2)

Thanks.

> Exactly.  E.g.,
> name _ARGS1(^?188,5710
> 
>> which is an "implicit tag name" entry for "_ARGS1", but not for "name".
>> IOW, etags doesn't understand macros.
> 
>     Whether etags understands macros or not, it is correctly identifying
> the lines containing function names, so I see no problem there.

It has a line, but it's a line for tag name "_ARGS1", not "name", by 
etags rules. find-tag just falls back to full text search, which 
xref-find-definitions doesn't, by default.

Because false positives will be more noticeable and annoying in its UI, 
compared to find-tag's.

>> Try adding `tag-symbol-match-p' to
>> etags-xref-find-definitions-tag-order. This example should work then,
>> but you'll get more false positives (like treating return types as
>> function names).
> 
>     Noted for future reference...
> 
>     Since doing that doesn't change what etags writes to TAGS, I'm not
> sure how that elisp change would result in function return types being
> matched as function names, but no matter.

Have you even tried this?

> For the moment I think I'll
> just continue to use find-tag and hope that xref-find-definitions will
> eventually work as well as find-tag before find-tag disappears.  :)

It works better already (more precise results, for inputs that etags 
understands well).

bug#28403: 25.2; find-tag works, but xref-find-definitions

[Eli Zaretskii]

> Date: Sat,  9 Sep 2017 22:50 EDT
> From: Winston <wbe@psr.com>
> 
> Dmitry kindly replied:
> > Which program are you generating TAGS with? Is it etags that comes with 
> > Emacs?
> 
> Yes, and "etags --version" prints: etags (GNU Emacs 25.2)
> 
> > xref-find-definitions is somewhat stricter about its input than 
> > find-tag.
> 
> Yes, that's what's causing the difference.  ;-)
> 
> > What does the entry for this function inside TAGS look like? [...]
> > I'm guessing it looks like:
> > 
> > name _ARGS1(
> 
> Exactly.  E.g.,
> name _ARGS1(^?188,5710

Thanks.  Could you please post a complete example of the code in
question, including the definition of the _ARGS1 macro, and any other
macros and typedefs that would make the example stand-alone?  I think
I understand what has happened, but I'd like to be sure before we
decide what to do about it.

> > Try adding `tag-symbol-match-p' to 
> > etags-xref-find-definitions-tag-order. This example should work then, 
> > but you'll get more false positives (like treating return types as 
> > function names).

Dmitry, how about providing a more user-friendly customization to that
effect?  As a "fire escape"?

> For the moment I think I'll just continue to use find-tag and hope
> that xref-find-definitions will eventually work as well as find-tag
> before find-tag disappears.  :)

I'd rather like to encourage you to continue using xref and report any
issue you find.  We want to make xref as good as the features it
replaces and better.  It is already better in several areas: it is
much more accurate (so many times lands you right on the spot, whereas
find-tag might require to go sequentially through several
alternatives), and in the rare cases where there are more than a
single candidate, it allows you to select the right one much faster.
We would like to solve any remaining problems, and that will be much
harder if people don't report those problems to us.  It is even
possible that, given the details I requested above, I will be able to
help you get your use case working with xref, so please don't give up
on xref, not just yet.

Thanks.

bug#28403: 25.2; find-tag works, but xref-find-definitions

[Dmitry Gutov]

On 9/10/17 5:29 PM, Eli Zaretskii wrote:

>>> Try adding `tag-symbol-match-p' to
>>> etags-xref-find-definitions-tag-order. This example should work then,
>>> but you'll get more false positives (like treating return types as
>>> function names).
> 
> Dmitry, how about providing a more user-friendly customization to that
> effect?  As a "fire escape"?

We can turn etags-xref-find-definitions-tag-order into a defcustom, with 
descriptions of what every possible element means. Problems:

1) I have hard time imagining how we're going to have descriptions for 
both tag-exact-match-p and tag-implicit-name-match-p that are different 
and make sense to the user.

2) The user will have to find out about 
etags-xref-find-definitions-tag-order first anyway.

Or we could add a custom variable to xref with a higher-level meaning... 
that would require support from backends, then.

bug#28403: 25.2; find-tag works, but xref-find-definitions

[Eli Zaretskii]

> Cc: 28403@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Mon, 11 Sep 2017 00:43:51 +0300
> 
> > Dmitry, how about providing a more user-friendly customization to that
> > effect?  As a "fire escape"?
> 
> We can turn etags-xref-find-definitions-tag-order into a defcustom, with 
> descriptions of what every possible element means.

I rather had in mind a variable with a few simple values, and a :set
function which would put what's needed into
etags-xref-find-definitions-tag-order.  IOW, hide the complexity from
the UI.

> 1) I have hard time imagining how we're going to have descriptions for 
> both tag-exact-match-p and tag-implicit-name-match-p that are different 
> and make sense to the user.

Simple values with solve that. For starters, we could use just 2:
'exact' and 'fuzzy'.

> 2) The user will have to find out about 
> etags-xref-find-definitions-tag-order first anyway.

Will be solved by having the defcustom under a more descriptive name.

And since this is a "fire escape", it will be needed relatively
rarely.  The point is to have it, so we could point users to it.

> Or we could add a custom variable to xref with a higher-level meaning... 

Yes.

> that would require support from backends, then.

Why?

bug#28403: 25.2; find-tag works, but xref-find-definitions

[Dmitry Gutov]

On 9/11/17 5:38 AM, Eli Zaretskii wrote:

>> that would require support from backends, then.

To make "fuzzy" work.

bug#28403: 25.2; find-tag works, but xref-find-definitions

[Eli Zaretskii]

> Cc: wbe@psr.com, 28403@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Mon, 11 Sep 2017 11:58:34 +0300
> 
> On 9/11/17 5:38 AM, Eli Zaretskii wrote:
> 
> >> that would require support from backends, then.
> 
> To make "fuzzy" work.

It would be good to have other back-ends support that, but I don't
think it's a must.  First, no one said all the back-ends must
distinguish between the two modes; it could even be that only one
makes sense for some language.  And second, I'm guess that this new
method will only be useful to those who are accustomed to etags.

So I think we could add this with only the etags back-end support for
now, and extend it later to other back-ends if needed.

Thanks.

bug#28403: 25.2; find-tag works, but xref-find-definitions

[Dmitry Gutov]

On 9/11/17 5:43 PM, Eli Zaretskii wrote:

>>>> that would require support from backends, then.
>>
>> To make "fuzzy" work.
> 
> It would be good to have other back-ends support that, but I don't
> think it's a must.  First, no one said all the back-ends must
> distinguish between the two modes; it could even be that only one
> makes sense for some language.

How will we define that it "makes sense"?

There are two ways it could work:

- Like etags, try to match all words on the same line as the legitimate 
definitions. For most backends, this will simply be impossible (for 
those that use a concise index instead of a flat list of strings). E.g., 
the elisp backend can't implement that.

- Allow substring matching for definition names? But xref-find-apropos 
already lets you do that.

> And second, I'm guess that this new
> method will only be useful to those who are accustomed to etags.
> 
> So I think we could add this with only the etags back-end support for
> now, and extend it later to other back-ends if needed.

Should it be a defcustom in etags.el, then? That's not far from what I 
suggested in the first place.

I hesitate to add a defcustom to xref.el that will only affect etags.

bug#28403: 25.2; find-tag works, but xref-find-definitions

[Eli Zaretskii]

> Cc: wbe@psr.com, 28403@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Wed, 13 Sep 2017 02:40:21 +0300
> 
> > It would be good to have other back-ends support that, but I don't
> > think it's a must.  First, no one said all the back-ends must
> > distinguish between the two modes; it could even be that only one
> > makes sense for some language.
> 
> How will we define that it "makes sense"?

I thought it would be self-evident, but maybe I was wrong.

> > So I think we could add this with only the etags back-end support for
> > now, and extend it later to other back-ends if needed.
> 
> Should it be a defcustom in etags.el, then? That's not far from what I 
> suggested in the first place.

That'd be okay, if the UI will be more friendly than letting the user
concoct a list of function symbols.

Thanks.

bug#28403: 25.2; find-tag works, but xref-find-definitions

[Dmitry Gutov]

On 9/13/17 6:32 PM, Eli Zaretskii wrote:

>> Should it be a defcustom in etags.el, then? That's not far from what I
>> suggested in the first place.
> 
> That'd be okay, if the UI will be more friendly than letting the user
> concoct a list of function symbols.

I'd approve probably any patch that makes 
etags-xref-find-definitions-tag-order into a defcustom, or even creates 
a new option governing it.

But maybe it's not necessary after all? I think we've addressed 
Winston's needs in different ways by now.

Maybe the manual should advertise the necessity to call etags with 
--regexp in certain cases more prominently instead.

bug#28403: 25.2; find-tag works, but xref-find-definitions

[Eli Zaretskii]

> Cc: wbe@psr.com, 28403@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Thu, 14 Sep 2017 15:13:33 +0300
> 
> On 9/13/17 6:32 PM, Eli Zaretskii wrote:
> 
> >> Should it be a defcustom in etags.el, then? That's not far from what I
> >> suggested in the first place.
> > 
> > That'd be okay, if the UI will be more friendly than letting the user
> > concoct a list of function symbols.
> 
> I'd approve probably any patch that makes 
> etags-xref-find-definitions-tag-order into a defcustom, or even creates 
> a new option governing it.
> 
> But maybe it's not necessary after all? I think we've addressed 
> Winston's needs in different ways by now.

I just feel that having a user-friendly defcustom would be more
future-proof.  Like I said: it's a fire escape.  Every complex feature
needs one.

> Maybe the manual should advertise the necessity to call etags with 
> --regexp in certain cases more prominently instead.

Until very recently, --regex didn't even document the
back-substitution feature it provides.  So we still have a way to go
in that direction.

bug#28403: 25.2; find-tag works, but xref-find-definitions

[Dmitry Gutov]

On 9/14/17 8:13 PM, Eli Zaretskii wrote:

> I just feel that having a user-friendly defcustom would be more
> future-proof.  Like I said: it's a fire escape.  Every complex feature
> needs one.

My point is if we document it enough for the majority of the users to 
find it, it would also make it more likely that they go the fire escape 
route instead of using etags in the right way. And then get some other 
wrong impression about xref-find-definitions.

>> Maybe the manual should advertise the necessity to call etags with
>> --regexp in certain cases more prominently instead.
> 
> Until very recently, --regex didn't even document the
> back-substitution feature it provides.  So we still have a way to go
> in that direction.

No argument from me.

bug#28403: 25.2; find-tag works, but xref-find-definitions

[Eli Zaretskii]

> Cc: wbe@psr.com, 28403@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Tue, 19 Sep 2017 03:44:28 +0300
> 
> On 9/14/17 8:13 PM, Eli Zaretskii wrote:
> 
> > I just feel that having a user-friendly defcustom would be more
> > future-proof.  Like I said: it's a fire escape.  Every complex feature
> > needs one.
> 
> My point is if we document it enough for the majority of the users to 
> find it, it would also make it more likely that they go the fire escape 
> route instead of using etags in the right way. And then get some other 
> wrong impression about xref-find-definitions.

Fire escapes don't have to be documented, just pointed out when and if
someone really needs them.

bug#28403: 25.2; find-tag works, but xref-find-definitions

[Winston]

Dmitry kindly replied:
> find-tag just falls back to full text search, which 
> xref-find-definitions doesn't, by default.
> 
> Because false positives will be more noticeable and annoying in its UI, 
> compared to find-tag's.

and previously replied:
>>> Try adding `tag-symbol-match-p' to
>>> etags-xref-find-definitions-tag-order. This example should work
>>> then, but you'll get more false positives (like treating return
>>> types as function names).

to which I'd replied:
>>     Noted for future reference...
>> 
>>     Since doing that doesn't change what etags writes to TAGS, I'm not
>> sure how that elisp change would result in function return types being
>> matched as function names, but no matter.

> Have you even tried this?

As my reply indicated, I had not, mainly because of your warning that
that solution would cause false positives.

However, I just tried it now, and, at least initially, it seems to do
fine, so I'm now willing to switch over.

If you haven't already done so, it's probably worth documenting this
solution somewhere so that others converting from find-tag can find this
fix via search (Google or otherwise).

Thanks!  I'll now consider this bug solved.
 -WBE

bug#28403: 25.2; find-tag works, but xref-find-definitions

[Winston]

Eli kindly replied:
> Could you please post a complete example of the code in question,
> including the definition of the _ARGS1 macro, and any other macros and
> typedefs that would make the example stand-alone?  I think I
> understand what has happened, but I'd like to be sure before we decide
> what to do about it.

   Sure (well, it's a semi-complete example).  The code, in general,
seeks to be as widely compatible with any compiler, ancient or modern,
as possible.  That's in part because the earliest versions originated
back in the 90's when, for example, Sun's bundled cc was a K&R C
compiler.  The _ARGS* macros provide compatibility between K&R C's

	name (arg1, arg2)
	type arg1; type arg2;
	{ ... }

and ANSI C's

	name (type arg1, type arg2)
	{ ... }.

_ARGS1 through _ARGS# (enough for whatever function takes the most
arguments (or more)) are #define'd in a header file.

   The other coding style that may matter here is the practice of
putting the return value type on a separate line at the function
definition (as you'll see below).  That makes the function name easier
to see (always at the left edge) and allows more arguments to fit on the
function name line (nice when grep'ing).

   The following edited extracts are the basic pieces:

----------

   In a common header file:

#ifdef __STDC__

# define _ARGS(args) args

# define _ARGS0(void) \
	(void)
# define _ARGS1(A,a) \
	(A a)
# define _ARGS2(A,a,B,b) \
	(A a, B b)
# define _ARGS3(A,a,B,b,C,c) \
	(A a, B b, C c)
 [...]

#else /* !__STDC__ */

# define _ARGS(args) ()

# define _ARGS0(args)       ()
# define _ARGS1(A,a) \
	(a) \
	A a;
# define _ARGS2(A,a,B,b) \
	(a, b) \
	A a; B b;
# define _ARGS3(A,a,B,b,C,c) \
	(a, b, c) \
	A a; B b; C c;
 [...]

#endif /* __STDC__ */


   Function declarations use _ARGS():

extern void baz _ARGS((int x,  int y));


   Function definitions in the *.c files use _ARGS#():

static struct mumblefrotzage *
foo _ARGS2(const char *,s, int,n)
{
 ...
}

int
main _ARGS3(int,argc, char**,argv, char**,env)
{
  struct mumblefrotzage *bar = foo ("example", 3);
  ...
}

   etags would extract "foo _ARGS2(" and "main _ARGS3(" as tag lines.
(find-tag) would find foo and main.

----------

Earlier in this thread, Dmitry suggested:
>>> Try adding `tag-symbol-match-p' to
>>> etags-xref-find-definitions-tag-order. This example should work
>>> then, but you'll get more false positives (like treating return
>>> types as function names).

to which Eli replied:
> Dmitry, how about providing a more user-friendly customization to that
> effect?  As a "fire escape"?

I initially replied to Dmitry's suggestion:
>> For the moment I think I'll just continue to use find-tag and hope
>> that xref-find-definitions will eventually work as well as find-tag
>> before find-tag disappears.

> I'd rather like to encourage you to continue using xref and report any
> issue you find.

   As you may have seen in my reply to Dmitry, I tried his suggestion,
and it seemed to fix the problem in the quick testing I tried, so I'm
willing to switch.

   The main issue I haven't gotten to yet is how best to change the
value of etags-xref-find-definitions-tag-order (exfdto) in a way that
will maximize long-term compatibility with future Emacs versions.  Doing
just (setq exfdto '(..)) in ~/.emacs I would expect to be the worst,
because either the default value or the symbol names might change in
future releases; (setq exfdto (append exfdto '(tag-symbol-match-p)))
might be better, but requires preloading etags.el to get the variable's
default value.  There are also onload hooks, site local lisp, etc.  It's
not yet clear to me what the best approach is, so I like your suggestion
to Dmitry above.  :)

> [The new xref function] is already better in several areas:

   I noticed that, and I liked seeing the list of matches when there's >1,
rather than having to do {arg} find-tag, possibly repeatedly, when the
one find-tag finds first isn't the one I wanted.  However, if the basic
find wasn't going to work, or was going to have a lot of false positives,
then I'd stick with find-tag, which has generally worked well (almost no
wrong matches, but it helps that my code tries to have all function
names in all files be unique, even when they're static, so there's
rarely >1 match).

> It is even possible that, given the details I requested above, I will
> be able to help you get your use case working with xref, so please
> don't give up on xref, not just yet.

   I haven't.  :)

   Thanks!  HTH,
 -WBE

bug#28403: 25.2; find-tag works, but xref-find-definitions

[Eli Zaretskii]

> Date: Sun, 10 Sep 2017 14:27 EDT
> From: Winston <wbe@psr.com>
> Cc: dgutov@yandex.ru, 28403@debbugs.gnu.org
> 
> Eli kindly replied:
> > Could you please post a complete example of the code in question,
> > including the definition of the _ARGS1 macro, and any other macros and
> > typedefs that would make the example stand-alone?  I think I
> > understand what has happened, but I'd like to be sure before we decide
> > what to do about it.
> 
>    Sure (well, it's a semi-complete example). [...]

Thanks, this confirms my suspicion: as Dmitry says, etags produces
TAGS for the _ARG# macros, and doesn't see the function names.  So the
fact that find-tag finds the functions is just sheer luck: it falls
back to more or less simple text search, so it can find anything you
have in the TAGS tables.

To have xref-find-definitions work in this case, you need to help
etags a bit, see below.

> > [The new xref function] is already better in several areas:
> 
>    I noticed that, and I liked seeing the list of matches when there's >1,
> rather than having to do {arg} find-tag, possibly repeatedly, when the
> one find-tag finds first isn't the one I wanted.  However, if the basic
> find wasn't going to work, or was going to have a lot of false positives,
> then I'd stick with find-tag, which has generally worked well (almost no
> wrong matches, but it helps that my code tries to have all function
> names in all files be unique, even when they're static, so there's
> rarely >1 match).

Again, it seems to work well in your case because you take special
precautions to avoid producing symbols that would generate false
positives.  But that is a fragile solution, IMO.

> > It is even possible that, given the details I requested above, I will
> > be able to help you get your use case working with xref, so please
> > don't give up on xref, not just yet.
> 
>    I haven't.  :)

Good, because here's how I suggest you invoke etags to solve the
problems with the _ARGS# macros:

  etags --regex="/[ \t]*\([^ \t]+\)[ \t]+_ARGS/\1/" ...

(replace "..." with all the other arguments you normally give when you
invoke etags).  This will tell etags to tag the symbols immediately
preceding the _ARGS# macro invocations _in_addition_ to what it
already does.  Then you can use xref-find-definitions in its default
configuration, and it will find your functions.

(The --regexp switch to etags is described in the Emacs manual.)

bug#28403: 25.2; find-tag works, but xref-find-definitions

[Winston]

Eli kindly replied:
> It is even possible that, given the details I requested above, I will
> be able to help you get your use case working with xref,

Just thought...  (setq etags-leading-function-names true) ?
to indicate that function names are always the first thing on their
definition lines?
 -WBE

bug#28403: 25.2; find-tag works, but xref-find-definitions

[Winston]

Eli kindly replied:
> It is even possible that, given the details I requested above, I will
> be able to help you get your use case working with xref,

Just thought...  (setq etags-leading-function-names true) ?
to indicate that function names are always the first thing on their
definition lines?

The idea would be to have something one puts in a C file's
Local Variables section, when such style is used.
 -WBE

bug#28403: 25.2; find-tag works, but xref-find-definitions

[Winston]

Eli suggested:
> ... here's how I suggest you invoke etags to solve the
> problems with the _ARGS# macros:
> 
>   etags --regex="/[ \t]*\([^ \t]+\)[ \t]+_ARGS/\1/" ...

   I don't think that quite works because it doesn't flush the number.
Probably more like:

   etags --regex="/[ \t]*\([^ \t]+\)[ \t]+_ARGS[0-9]+/\1/" ...

but since, in this particular case, there's never any leading space on
the line before a function name and the code doesn't use tabs on those
lines, maybe

   etags --regex="/\([^ ]+\) +_ARGS[0-9]+/\1/" ...

would work as well?  (Does regex have an implicit '^'?  If not, I'll use
an explicit one.)

> This will tell etags to tag the symbols immediately preceding the
> _ARGS# macro invocations _in_addition_ to what it already does.

   [Too bad etags doesn't have a way of doing "s/[ \t]+_ARGS[0-9]*//" on
the lines it normally finds...]

> Then you can use xref-find-definitions in its default configuration,
> and it will find your functions.

   OK.  I like that approach better than having to do the setq,
especially since putting a suitable etags command in a makefile is easy.
That also has the benefit of tying the fix to the code written in that
style, rather than making a global change that would affect other code I
work on.

   Thanks!
 -WBE

bug#28403: 25.2; find-tag works, but xref-find-definitions

[Eli Zaretskii]

> Date: Sun, 10 Sep 2017 16:12 EDT
> From: Winston <wbe@psr.com>
> Cc: dgutov@yandex.ru, 28403@debbugs.gnu.org
> 
> Eli suggested:
> > ... here's how I suggest you invoke etags to solve the
> > problems with the _ARGS# macros:
> > 
> >   etags --regex="/[ \t]*\([^ \t]+\)[ \t]+_ARGS/\1/" ...
> 
>    I don't think that quite works

It worked with your example.

>    etags --regex="/\([^ ]+\) +_ARGS[0-9]+/\1/" ...
> 
> would work as well?

As long as you understand the principles, it's in your hands.

> Does regex have an implicit '^'?

Yes (as explained in the manual).

>    [Too bad etags doesn't have a way of doing "s/[ \t]+_ARGS[0-9]*//" on
> the lines it normally finds...]

Why do you need that?

>    OK.  I like that approach better than having to do the setq,
> especially since putting a suitable etags command in a makefile is easy.
> That also has the benefit of tying the fix to the code written in that
> style, rather than making a global change that would affect other code I
> work on.

Right.

I guess we can now close the bug report?

bug#28403: 25.2; find-tag works, but xref-find-definitions

[Winston]

   Progress, but it's not working as expected.

   I ran   etags --regex="/\([^ ]+\) +_ARGS[0-9]+/\1/" ...
It added lines of the form:

foo1 _ARGS2^?foo1^A341,12188
foo2 _ARGS1^?foo2^A368,12664

to TAGS, which looked reasonable.

   I then started up a fresh Emacs that uses xref-find-definitions
instead of find-tag, and that did not have the setq change.  The results
surprised me:

* Find "foo1" or "foo2" (complete names) worked, both when typed in and
   when extracting a name near (point).

* Find "foo" (partial name) didn't: rather than creating a window with
   the alternatives, it failed with "No definitions found for: foo".
   find-tag given the same string "foo" went to the first one.
   xref-find-definitions with Dmitry's suggested change also worked
   (yesterday, without the special etags --regex TAGS, popping up a
   window with the alternatives).

Any suggestions?
 -WBE

P.S. Also, I'm finding it mildly annoying that xref-find-definitions,
     when next to just about any word, including text in comments, tries
     immediately to go to that word as a tag rather than prompting me
     for a name with the word as the default, as find-tag did.  That
     means I'll have to remember to use my {arg} key fairly often to
     tell it "don't do that, prompt me."  Is there an "always prompt"
     option, or do I need to write a trivial wrapper function?

bug#28403: 25.2; find-tag works, but xref-find-definitions

[Dmitry Gutov]

On 9/11/17 12:18 AM, Winston wrote:

> * Find "foo" (partial name) didn't: rather than creating a window with
>     the alternatives, it failed with "No definitions found for: foo".

That is because 'foo' is not defined in that file. And showing either of 
the two lines as a match for it would be a false positive.

You should probably use project-find-regexp. Or use C-u M-x 
xref-find-definitions with completion: you type 'foo', TAB, then see the 
available tags that start with that string, pick one of them, and see 
its definitions.

>     find-tag given the same string "foo" went to the first one.
>     xref-find-definitions with Dmitry's suggested change also worked
>     (yesterday, without the special etags --regex TAGS, popping up a
>     window with the alternatives).
> 
> Any suggestions?

Just what I suggested before. But this problem is not related to your 
use of macros anymore.

>   -WBE
> 
> P.S. Also, I'm finding it mildly annoying that xref-find-definitions,
>       when next to just about any word, including text in comments, tries
>       immediately to go to that word as a tag rather than prompting me
>       for a name with the word as the default, as find-tag did.  That
>       means I'll have to remember to use my {arg} key fairly often to
>       tell it "don't do that, prompt me."  Is there an "always prompt"
>       option, or do I need to write a trivial wrapper function?

You can customize xref-prompt-for-identifier to t.

bug#28403: 25.2; find-tag works, but xref-find-definitions

[Eli Zaretskii]

> Date: Sun, 10 Sep 2017 17:19 EDT
> From: Winston <wbe@psr.com>
> Cc: dgutov@yandex.ru, 28403@debbugs.gnu.org
> 
> * Find "foo1" or "foo2" (complete names) worked, both when typed in and
>    when extracting a name near (point).
> 
> * Find "foo" (partial name) didn't: rather than creating a window with
>    the alternatives, it failed with "No definitions found for: foo".
>    find-tag given the same string "foo" went to the first one.

As expected.  You have bad habits from using find-tag.  With xref, you
need to do "M-. foo TAB".

> P.S. Also, I'm finding it mildly annoying that xref-find-definitions,
>      when next to just about any word, including text in comments, tries
>      immediately to go to that word as a tag rather than prompting me
>      for a name with the word as the default, as find-tag did.  That
>      means I'll have to remember to use my {arg} key fairly often to
>      tell it "don't do that, prompt me."  Is there an "always prompt"
>      option, or do I need to write a trivial wrapper function?

I got used to this very quickly, but if you don't want to try, write
your wrapper.

bug#28403: 25.2; find-tag works, but xref-find-definitions

[Winston]

I wrote:
>> * Find "foo" (partial name) didn't: ...

Eli replied:
> With xref, you need to do "M-. foo TAB".

   Ah!  OK.  Yes, that worked.

[My key bindings are entirely different, but I got the idea. :) ]
 -WBE

bug#28403: 25.2; find-tag works, but xref-find-definitions

[Winston]

Eli suggested:
>>>  etags --regex="/[ \t]*\([^ \t]+\)[ \t]+_ARGS/\1/" ...

and in reply to my changes said:
> It worked with your example.

   Yes.  The difference is a small one: "_ARGS[0-9]" only matches function
definition lines, while using only "_ARGS" will also match declarations.


>>    [Too bad etags doesn't have a way of doing "s/[ \t]+_ARGS[0-9]*//" on
>> the lines it normally finds...]

> Why do you need that?

   The idea was that instead of cluttering up the TAGS file with two
lines for every tag, such as

	foo _ARGS2^?foo^A50,100
	foo _ARGS2(^?50,100

as a result of using --regex, a post edit that allowed one to do
"s/[ \t]+_ARGS[0-9]*//" could simply remove the _ARGS2 part and reduce
the default etags line to:

	foo(^?50,100

thus (I'm guessing) clarifying things a different way.
 -WBE

bug#28403: 25.2; find-tag works, but xref-find-definitions

[Eli Zaretskii]

> Date: Mon, 11 Sep 2017 00:05 EDT
> From: Winston <wbe@psr.com>
> Cc: dgutov@yandex.ru, 28403@debbugs.gnu.org
> 
> Eli suggested:
> >>>  etags --regex="/[ \t]*\([^ \t]+\)[ \t]+_ARGS/\1/" ...
> 
> and in reply to my changes said:
> > It worked with your example.
> 
>    Yes.  The difference is a small one: "_ARGS[0-9]" only matches function
> definition lines, while using only "_ARGS" will also match declarations.

That's not a problem in practice, because your prototypes are laid out
differently: they have more than one symbol between the line beginning
and _ARGS.  But never mind.

> >>    [Too bad etags doesn't have a way of doing "s/[ \t]+_ARGS[0-9]*//" on
> >> the lines it normally finds...]
> 
> > Why do you need that?
> 
>    The idea was that instead of cluttering up the TAGS file with two
> lines for every tag, such as
> 
> 	foo _ARGS2^?foo^A50,100
> 	foo _ARGS2(^?50,100
> 
> as a result of using --regex, a post edit that allowed one to do
> "s/[ \t]+_ARGS[0-9]*//" could simply remove the _ARGS2 part and reduce
> the default etags line to:
> 
> 	foo(^?50,100
> 
> thus (I'm guessing) clarifying things a different way.

You shouldn't worry about that (and shouldn't look at TAGS anyway).
There will be extra entries there, but they should never get in your
way, as long as you use xref in its default "accurate" mode.

bug#28403: 25.2; find-tag works, but xref-find-definitions

[Winston]

[Eli also pointed out that TAB is the way to see possible completions
 for a tag string.  It worked.  Thanks to both of you.]

I asked:
>> Is there an "always prompt" option, or do I need to write a trivial
>> wrapper function?

Dmitry kindly replied:
> You can customize xref-prompt-for-identifier to t.

Hooray!  Thanks!
 -WBE

bug#28403: 25.2; find-tag works, but xref-find-definitions

[Winston]

Eli asked:
> I guess we can now close the bug report?

From my standpoint, yes.

However, since xref-find-definitions is not a drop-in replacement for
find-tag (differences incl. find partial name differs, goes to the
string at (point) without prompting, etc.), I probably won't be the last
person running into the differences.

Perhaps the simplest solution is to expand the documentation section of
xref-find-definitions with text such as the following (corrected for
anything I've got wrong below).  :-)

----------

xref-find-definitions differs from find-tag several ways:
* find-tag always prompted for a name, with the string at (point) as
  the default name.  With no argument, xref-find-definitions does not
  prompt, and tries immediately to go to a tag with that default name.
  Use (setq xref-prompt-for-identifier t) to force prompting. 
* find-tag allowed partial names.  xref-find-definitions does not.
  Use TAB after a partial name in a prompt for completion(s).
* If find-tag finds your tags OK but xref-find-definitions does not, you
  may need to use --regexp with etags to help it identify the tag name.
  (See etags man page.)  If even that doesn't help, try
							{whatever Dmitry
  and Eli decide is a good long-term easy way to add '(tag-symbol-match-p)
  to etags-xref-find-definitions-tag-order}.

----------

Thanks for your help!  HTH,
 -WBE

bug#28403: 25.2; find-tag works, but xref-find-definitions

[Eli Zaretskii]

> Date: Mon, 11 Sep 2017 01:02 EDT
> From: Winston <wbe@psr.com>
> Cc: dgutov@yandex.ru, 28403@debbugs.gnu.org
> 
> Eli asked:
> > I guess we can now close the bug report?
> 
> From my standpoint, yes.

Thanks, done.

> xref-find-definitions differs from find-tag several ways:
> * find-tag always prompted for a name, with the string at (point) as
>   the default name.  With no argument, xref-find-definitions does not
>   prompt, and tries immediately to go to a tag with that default name.
>   Use (setq xref-prompt-for-identifier t) to force prompting. 
> * find-tag allowed partial names.  xref-find-definitions does not.
>   Use TAB after a partial name in a prompt for completion(s).
> * If find-tag finds your tags OK but xref-find-definitions does not, you
>   may need to use --regexp with etags to help it identify the tag name.
>   (See etags man page.)  If even that doesn't help, try
> 							{whatever Dmitry
>   and Eli decide is a good long-term easy way to add '(tag-symbol-match-p)
>   to etags-xref-find-definitions-tag-order}.

Thanks.  Some of this is already in the documentation, other parts
will be if/when the code does what we are discussing now.

bug#28403: 25.2; find-tag works, but xref-find-definitions

[Winston]

Dmitry suggested:
> Maybe the manual should advertise the necessity to call etags with 
> --regexp in certain cases more prominently instead.

On the assumption that no one will read the manual until they have to,
I'd still suggest at least a 1-sentence mention of the issue in the
documentation for xref-find-definitions, even if it mostly says see the
manual if you're having problems finding tags.
 -WBE

bug#28403: 25.2; find-tag works, but xref-find-definitions

[Dmitry Gutov]

On 9/14/17 8:36 PM, Winston wrote:
> Dmitry suggested:
>> Maybe the manual should advertise the necessity to call etags with
>> --regexp in certain cases more prominently instead.
> 
> On the assumption that no one will read the manual until they have to,
> I'd still suggest at least a 1-sentence mention of the issue in the
> documentation for xref-find-definitions, even if it mostly says see the
> manual if you're having problems finding tags.

Yes, well, I probably agree from the standpoint of practicality. Even 
though xref-find-definitions itself knows zilch about etags.