xref-find-matches and stuff

xref-find-matches and stuff

[Dmitry Gutov]

Hi all,

I've been considering a set of minor changes:

- Use lisp/cedet/pulse.el to briefly highlight the current line after 
the jump.

- Add the xref-find-matches command (as well as a corresponding backend 
action), for project-wide string searches (hopefully with regexp 
support). That probably doesn't sound as exciting as the existing 
commands, but it will be "Grep in a new UI", and it will eliminate the 
"select the directory" step.

- Further, using the same data we could implement "search and replace", 
in the same way as xref-find-references can get us to "rename".

- With this, xref-find-function starts becoming pretty fat. The 
"references" and "matches" actions can be split into a new variable 
(xref-scan-function?).

The current find-references implementation, as well as the anticipated 
find-matches one, are language-agnostic, so we can expect this variable 
to often keep its default value, especially if the default 
implementations use some kind of "current project" facility. So far, 
both etags and elisp backend's references implementations only differ in 
the directories lists they use.

Thanks,
Dmitry.

Re: xref-find-matches and stuff

[Helmut Eller]

On Mon, May 04 2015, Dmitry Gutov wrote:

> - Add the xref-find-matches command (as well as a corresponding
> backend action), for project-wide string searches (hopefully with
> regexp support). That probably doesn't sound as exciting as the
> existing commands, but it will be "Grep in a new UI", and it will
> eliminate the "select the directory" step.

With a separate xref-find-matches command it would make sense for
xref-find-apropos to work more like the normal apropos, with doc-strings
and such.  Ideally use the same UI as Elisp's apropos.
>
> - Further, using the same data we could implement "search and
> replace", in the same way as xref-find-references can get us to
> "rename".

If we're going to modify something then location info better be super
accurate.

Helmut

Re: xref-find-matches and stuff

[Dmitry Gutov]

On 05/04/2015 12:17 PM, Helmut Eller wrote:

> With a separate xref-find-matches command it would make sense for
> xref-find-apropos to work more like the normal apropos, with doc-strings
> and such.

They still have different purposes, but sure, we can make an 
xref-with-doc element or something, which would render itself with a 
docstring.

> Ideally use the same UI as Elisp's apropos.

That might be more complicated: a) the element would need to be 
specialized for the elisp backend, with type and plist, b) pressing RET 
in M-x apropos output does not get you to the symbol's definition; 
instead you jump between various Help buffers. I don't know if we want 
that for xref-find-apropos, that's certainly hard to generalize across 
languages.

> If we're going to modify something then location info better be super
> accurate.

The "search and replace" interface would have to be something like M-x 
tags-query-replace, at least at first. It's using a regexp, so people 
are probably used to confirming each replacement.

Re: xref-find-matches and stuff

[Eli Zaretskii]

> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Mon, 4 May 2015 05:45:30 +0300
> Cc: Helmut Eller <eller.helmut@gmail.com>
> 
> - Use lisp/cedet/pulse.el to briefly highlight the current line after 
> the jump.

I see you've already done this; thanks.  FWIW, I like the highlighting
by compilation-goto-locus better: it only highlights the relevant part
of the line, not the whole line.

Re: xref-find-matches and stuff

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Dmitry Gutov <dgutov@yandex.ru>
>> Date: Mon, 4 May 2015 05:45:30 +0300
>> Cc: Helmut Eller <eller.helmut@gmail.com>
>> 
>> - Use lisp/cedet/pulse.el to briefly highlight the current line after 
>> the jump.
>
> I see you've already done this; thanks.  FWIW, I like the highlighting
> by compilation-goto-locus better: it only highlights the relevant part
> of the line, not the whole line.

What about the wished-by-some moving of CEDET et al. to GNU ELPA?
Creating dependencies on CEDET by core features makes that more
difficult.

Re: xref-find-matches and stuff

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Mon, 04 May 2015 18:37:12 +0200
> 
> What about the wished-by-some moving of CEDET et al. to GNU ELPA?

I didn't even know it was being considered.  IMO, it makes no sense to
do that, given that we decided to add it to Emacs not so long ago.
Not to mention the fact that IDE capabilities should be in the Emacs
core, IMO.

Re: xref-find-matches and stuff

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Óscar Fuentes <ofv@wanadoo.es>
>> Date: Mon, 04 May 2015 18:37:12 +0200
>> 
>> What about the wished-by-some moving of CEDET et al. to GNU ELPA?
>
> I didn't even know it was being considered.  IMO, it makes no sense to
> do that, given that we decided to add it to Emacs not so long ago.
> Not to mention the fact that IDE capabilities should be in the Emacs
> core, IMO.

IIRC it was suggested to segregate CEDET, Org and some other big
packages (Gnus?) that are developed outside of Emacs core to GNU ELPA,
but include them on the released source tarballs. The point was not to
remove them from Emacs, but to avoid the hassle associated with the
merges and other dual-maintenance chores.

Re: xref-find-matches and stuff

[Stefan Monnier]

> IIRC it was suggested to segregate CEDET, Org and some other big
> packages (Gnus?) that are developed outside of Emacs core to GNU ELPA,
> but include them on the released source tarballs. The point was not to
> remove them from Emacs, but to avoid the hassle associated with the
> merges and other dual-maintenance chores.

The inclusion of CEDET into Emacs had the intention to try and integrate
it better into Emacs.  Moving it to GNU ELPA would only be to ease
maintenance, if needed.

My current understanding is that moving Org to elpa.git would probably
be a good idea, but moving CEDET (as a whole) to elpa.git is probably
not such a good idea.  But in any case, if some part of CEDET is found
to be useful for the core, then it should be integrated into the core.
If CEDET were in elpa.git, that might simply mean duplicating/moving
the code.


        Stefan

Re: xref-find-matches and stuff

[Dmitry Gutov]

On 05/04/2015 07:37 PM, Óscar Fuentes wrote:

> What about the wished-by-some moving of CEDET et al. to GNU ELPA?
> Creating dependencies on CEDET by core features makes that more
> difficult.

I agree with that desire, but guess we'll have extract the common 
features before that (which may or may not happen before the 25.1 release).

pulse.el has no dependencies, so it's trivial to extract.

However, I've recently added dependency on semantic/symref, and through 
it, on cedet-idutils and its neighbors. They will be harder to extract 
(but worth it, I believe).

Re: xref-find-matches and stuff

[Dmitry Gutov]

On 05/04/2015 07:17 PM, Eli Zaretskii wrote:

> I see you've already done this; thanks.  FWIW, I like the highlighting
> by compilation-goto-locus better: it only highlights the relevant part
> of the line, not the whole line.

Thanks for the feedback, I've brought it closer to that behavior. By the 
way, "the relevant part" is not exactly right, it simply omits the 
indentation and the newline.

Maybe we should also adjust pulse-delay and pulse-iterations. I think 
it's trying too hard to be smooth, and increases the odds of text 
flickering doing that.

Re: xref-find-matches and stuff

[Helmut Eller]

On Mon, May 04 2015, Dmitry Gutov wrote:

> Maybe we should also adjust pulse-delay and pulse-iterations. I think
> it's trying too hard to be smooth, and increases the odds of text
> flickering doing that.

Definitely.  I find this "fading out" very annoying when stepping
through the list with . and ,.  It seems to slow down or block
something; it feels sluggish now.

Well, it turns out pulse calls sit-for.  Certainly not what I want.

Helmut

Re: xref-find-matches and stuff

[Dmitry Gutov]

On 05/05/2015 10:07 AM, Helmut Eller wrote:

> Definitely.  I find this "fading out" very annoying when stepping
> through the list with . and ,.  It seems to slow down or block
> something; it feels sluggish now.
>
> Well, it turns out pulse calls sit-for.  Certainly not what I want.

It seems you're commenting on something different, rather than the 
values of those variables. Specifically: that the cursor is displayed as 
being in the other window during the whole fadeout (or until you press a 
key), whereas it should be back in the xref buffer.

That's not impossible to implement using sit-for, but indeed, a timer 
will make it easier.

Re: xref-find-matches and stuff

[Dmitry Gutov]

On 05/05/2015 10:07 AM, Helmut Eller wrote:

> Well, it turns out pulse calls sit-for.  Certainly not what I want.

The latest changes should be to your liking.

Re: xref-find-matches and stuff

[Helmut Eller]

On Wed, May 06 2015, Dmitry Gutov wrote:

> On 05/05/2015 10:07 AM, Helmut Eller wrote:
>
>> Well, it turns out pulse calls sit-for.  Certainly not what I want.
>
> The latest changes should be to your liking.

It's less slow now but still noticeably slower as when
xref-pulse-on-jump is nil.  The flickering is also still there.  Don't
bother to fix this; I'll set xref-pulse-on-jump in my .emacs.

Helmut

Re: xref-find-matches and stuff

[Dmitry Gutov]

On 05/06/2015 08:45 PM, Helmut Eller wrote:

> It's less slow now but still noticeably slower as when
> xref-pulse-on-jump is nil.

You might want to look into that; I don't notice any particular 
slowness. Redrawing an overlay should be a fast enough operation, but 
you could tweak the related variables.

> The flickering is also still there.

It's http://debbugs.gnu.org/12363. The idea here is to annoy someone 
else into fixing it.

> Don't bother to fix this; I'll set xref-pulse-on-jump in my .emacs.

Okay. There's likely nothing left that I can improve there.

Re: xref-find-matches and stuff

[Helmut Eller]

On Wed, May 06 2015, Dmitry Gutov wrote:

> On 05/06/2015 08:45 PM, Helmut Eller wrote:
>
>> It's less slow now but still noticeably slower as when
>> xref-pulse-on-jump is nil.
>
> You might want to look into that; I don't notice any particular
> slowness. Redrawing an overlay should be a fast enough operation, but
> you could tweak the related variables.

My guess is that set-face-background which is called by pulse-reset-face
communicates with the X server and that it's causing the slow down and
the flickering.

BTW, I see things like

  Error in pre-command-hook (pulse-momentary-unhighlight):
   (wrong-type-argument timerp nil)  [6 times]

when trying it with emacs -Q -nw.

Helmut

Re: xref-find-matches and stuff

[Dmitry Gutov]

On 05/06/2015 09:34 PM, Helmut Eller wrote:

> My guess is that set-face-background which is called by pulse-reset-face
> communicates with the X server and that it's causing the slow down and
> the flickering.

It's causing redisplay, but that still should be fast enough, I think.

The cause of flickering has been discussed here: 
http://debbugs.gnu.org/cgi/bugreport.cgi?bug=16621#14

> BTW, I see things like
>
>    Error in pre-command-hook (pulse-momentary-unhighlight):
>     (wrong-type-argument timerp nil)  [6 times]
>
> when trying it with emacs -Q -nw.

Thanks, should be fixed now.

Re: xref-find-matches and stuff

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org, eller.helmut@gmail.com
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Mon, 4 May 2015 23:27:34 +0300
> 
> On 05/04/2015 07:17 PM, Eli Zaretskii wrote:
> 
> > I see you've already done this; thanks.  FWIW, I like the highlighting
> > by compilation-goto-locus better: it only highlights the relevant part
> > of the line, not the whole line.
> 
> Thanks for the feedback, I've brought it closer to that behavior.

Thanks, but I'd still prefer the Grep way, see below.

> By the way, "the relevant part" is not exactly right, it simply
> omits the indentation and the newline.

We are likely talking about different things.  I'm talking about the
highlighting done by Grep mode, when you go to the next/previous hit.
For me, it highlights only the match found by Grep the program.

I think xref-find-references is similar to Grep, in that it looks for
matches to a known string/regexp, so IWBNI it highlighted the match.

> Maybe we should also adjust pulse-delay and pulse-iterations. I think 
> it's trying too hard to be smooth, and increases the odds of text 
> flickering doing that.

I don't mind the result, FWIW, it doesn't seem to annoy me.

Re: xref-find-matches and stuff

[Dmitry Gutov]

On 05/05/2015 06:40 PM, Eli Zaretskii wrote:

> We are likely talking about different things.  I'm talking about the
> highlighting done by Grep mode, when you go to the next/previous hit.
> For me, it highlights only the match found by Grep the program.

Oh, right. I was looking at Compilation, not Grep.

> I think xref-find-references is similar to Grep, in that it looks for
> matches to a known string/regexp, so IWBNI it highlighted the match.

We should be able to do what Grep does when jumping to a match. However, 
when jumping back, the highlight can't do better that what it does now.

> I don't mind the result, FWIW, it doesn't seem to annoy me.

All right. Then I'm hoping this more prominent usage of pulse will 
eventually annoy someone into fixing bug#12363 (and 16621, which seems 
to be a duplicate).

Re: xref-find-matches and stuff

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org, eller.helmut@gmail.com
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Tue, 5 May 2015 19:10:44 +0300
> 
> > I think xref-find-references is similar to Grep, in that it looks for
> > matches to a known string/regexp, so IWBNI it highlighted the match.
> 
> We should be able to do what Grep does when jumping to a match. However, 
> when jumping back, the highlight can't do better that what it does now.

"Jumping back" where?  I'm not sure I understand what you mean.

Re: xref-find-matches and stuff

[Dmitry Gutov]

On 05/05/2015 07:33 PM, Eli Zaretskii wrote:

> "Jumping back" where?  I'm not sure I understand what you mean.

xref-pop-marker-stack, or M-,.

Re: xref-find-matches and stuff

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org, eller.helmut@gmail.com
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Tue, 5 May 2015 19:37:57 +0300
> 
> On 05/05/2015 07:33 PM, Eli Zaretskii wrote:
> 
> > "Jumping back" where?  I'm not sure I understand what you mean.
> 
> xref-pop-marker-stack, or M-,.

Ah, OK.  Well, highlighting on forward movement is half the job, so
it's better than nothing, I think.  As for going back, couldn't the
marker stack store more info?

Re: xref-find-matches and stuff

[Dmitry Gutov]

On 05/05/2015 07:45 PM, Eli Zaretskii wrote:

> As for going back, couldn't the marker stack store more info?

But what else would it store? A jump is performed from an arbitrary 
position. When going back to it, what extra data would the highlighter use?

Re: xref-find-matches and stuff

[Eli Zaretskii]

> X-Spam-Status: No, score=0.8 required=5.0 tests=BAYES_50,FREEMAIL_FROM,
> 	T_DKIM_INVALID autolearn=disabled version=3.3.2
> Cc: emacs-devel@gnu.org, eller.helmut@gmail.com
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Tue, 5 May 2015 19:48:17 +0300
> 
> When going back to it, what extra data would the highlighter use?

The pattern to highlight, I thought.

Re: xref-find-matches and stuff

[Dmitry Gutov]

On 05/05/2015 08:20 PM, Eli Zaretskii wrote:

> The pattern to highlight, I thought.

What would it be based on? Again, we're jumping back to a position where 
we were before the search was made.

Re: xref-find-matches and stuff

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org, eller.helmut@gmail.com
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Tue, 5 May 2015 20:29:11 +0300
> 
> On 05/05/2015 08:20 PM, Eli Zaretskii wrote:
> 
> > The pattern to highlight, I thought.
> 
> What would it be based on?

On what we found when we were there on the way forward.

Re: xref-find-matches and stuff

[Dmitry Gutov]

On 05/05/2015 09:33 PM, Eli Zaretskii wrote:

> On what we found when we were there on the way forward.

But in all likelihood, we didn't find anything there.

A forward jump goes to something specific. When you jump back, you 
simply return to the place you came from.

There's no guarantee that there were any forward jumps before that, or 
that one of them landed in that exact position.

Re: xref-find-matches and stuff

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org, eller.helmut@gmail.com
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Tue, 5 May 2015 21:38:31 +0300
> 
> On 05/05/2015 09:33 PM, Eli Zaretskii wrote:
> 
> > On what we found when we were there on the way forward.
> 
> But in all likelihood, we didn't find anything there.
> 
> A forward jump goes to something specific. When you jump back, you 
> simply return to the place you came from.
> 
> There's no guarantee that there were any forward jumps before that, or 
> that one of them landed in that exact position.

In that case, it's OK not to highlight anything.  But if that place
was another hit, then we will be able to highlight, right?

Re: xref-find-matches and stuff

[Dmitry Gutov]

On 05/05/2015 10:11 PM, Eli Zaretskii wrote:

 > But if that place
> was another hit, then we will be able to highlight, right?

In theory, yes, but we'd have to keep data from previous searches, which 
we currently don't. And either way, the probability of that position 
being the exact destination of a previous jump is not very high, IMO.

I like to see it highlighted anyway. If you prefer, `xref-pulse-on-jump' 
can grow another possible value, `forward', which will mean not to blink 
on a backward jump.

Re: xref-find-matches and stuff

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org, eller.helmut@gmail.com
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Wed, 6 May 2015 00:44:23 +0300
> 
> On 05/05/2015 10:11 PM, Eli Zaretskii wrote:
> 
>  > But if that place
> > was another hit, then we will be able to highlight, right?
> 
> In theory, yes, but we'd have to keep data from previous searches, which 
> we currently don't.

Right, and I was suggesting to keep that data.

> And either way, the probability of that position 
> being the exact destination of a previous jump is not very high, IMO.

Depends on the usage patterns, I guess.

> I like to see it highlighted anyway. If you prefer, `xref-pulse-on-jump' 
> can grow another possible value, `forward', which will mean not to blink 
> on a backward jump.

Fine with me, thanks.

Re: xref-find-matches and stuff

[Dmitry Gutov]

On 05/06/2015 05:40 AM, Eli Zaretskii wrote:

> Fine with me, thanks.

Here's why you might change your mind: even on forward jumps, we can 
only have a "match" data in certain cases, like using xref-find-references.

Using etags for xref-find-references, we don't have the column where the 
identifier name starts. So if we're jumping to last_height, we only have 
the information where "static int last_height;" starts.

I'm sure we could do a search-forward, but that's venturing into the 
hacks territory, and there's no guarantee, in general, that the 
identifier you've typed into the prompt occurs in the buffer as-is.

Re: xref-find-matches and stuff

[Dmitry Gutov]

On 05/06/2015 03:36 PM, Dmitry Gutov wrote:

> Using etags for xref-find-references, we don't have the column where the

Sorry,            ^ xref-find-definitions

> identifier name starts. So if we're jumping to last_height, we only have
> the information where "static int last_height;" starts.

Re: xref-find-matches and stuff

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org, eller.helmut@gmail.com
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Wed, 6 May 2015 15:36:22 +0300
> 
> Here's why you might change your mind: even on forward jumps, we can 
> only have a "match" data in certain cases, like using xref-find-references.

When we don't have the data, and no symbol in hand to search for, then
it's okay to highlight the entire line, I think.

> Using etags for xref-find-references, we don't have the column where the 
> identifier name starts. So if we're jumping to last_height, we only have 
> the information where "static int last_height;" starts.
> 
> I'm sure we could do a search-forward, but that's venturing into the 
> hacks territory, and there's no guarantee, in general, that the 
> identifier you've typed into the prompt occurs in the buffer as-is.

It's a nice added value to be able to highlight more specifically than
the entire line.  If it's hard, then the person who solves it will get
more bonus points from me.  But I see no reason to give up hope that
something like this will some day happen.

Re: xref-find-matches and stuff

[Dmitry Gutov]

On 05/06/2015 06:30 PM, Eli Zaretskii wrote:

> It's a nice added value to be able to highlight more specifically than
> the entire line.

Thinking about it more, we don't need to require it to be doable for any 
kind of xref. The default implementation could do the simplest thing 
(search for the exact input until the end of the line), and any backend 
could provide a smarter logic.

If all fails, then highlight the whole line.

Re: xref-find-matches and stuff

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org, eller.helmut@gmail.com
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Thu, 7 May 2015 01:37:26 +0300
> 
> Thinking about it more, we don't need to require it to be doable for any 
> kind of xref. The default implementation could do the simplest thing 
> (search for the exact input until the end of the line), and any backend 
> could provide a smarter logic.
> 
> If all fails, then highlight the whole line.

Sounds like a good plan to me.

Re: xref-find-matches and stuff

[Dmitry Gutov]

On 05/06/2015 06:30 PM, Eli Zaretskii wrote:

> It's a nice added value to be able to highlight more specifically than
> the entire line.  If it's hard, then the person who solves it will get
> more bonus points from me.  But I see no reason to give up hope that
> something like this will some day happen.

All right, it happens now for the xref-find-references (the built-in 
implementation, at least) and xref-find-regexp results.

Re: xref-find-matches and stuff

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org, eller.helmut@gmail.com
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Mon, 20 Jul 2015 04:55:07 +0300
> 
> On 05/06/2015 06:30 PM, Eli Zaretskii wrote:
> 
> > It's a nice added value to be able to highlight more specifically than
> > the entire line.  If it's hard, then the person who solves it will get
> > more bonus points from me.  But I see no reason to give up hope that
> > something like this will some day happen.
> 
> All right, it happens now for the xref-find-references (the built-in 
> implementation, at least) and xref-find-regexp results.

Looks good, thanks.

Re: xref-find-matches and stuff

[Andreas Politz]

Eli Zaretskii <eliz@gnu.org> writes:

>> Thanks for the feedback, I've brought it closer to that behavior.
>
> Thanks, but I'd still prefer the Grep way, see below.

Why aren't there any hooks running after pushing/popping the stack, like
next-error-hook, i.e. for recentering the window ?

-ap

Re: xref-find-matches and stuff

[Dmitry Gutov]

On 07/14/2015 11:30 AM, Andreas Politz wrote:

> Why aren't there any hooks running after pushing/popping the stack, like
> next-error-hook,

You mean a new, different hook? Feel free to file a feature request, 
with a usage example. But even now, you can just advise one of the two 
relevant functions (xref-push-marker-stack or xref-pop-marker-stack).

> i.e. for recentering the window ?

Hook for recentering the window? Anyway, we don't explicitly recenter 
the window after pushing or popping. Though that does happen when moving 
between xrefs in the list.

Re: xref-find-matches and stuff

[Andreas Politz]

Dmitry Gutov <dgutov@yandex.ru> writes:

> You mean a new, different hook? 

It seems to me that having a hook run after jumping somewhere is
pretty standard behaviour, e.g. occur-mode-find-occurrence-hook,
find-tag-hook, next-error-hook, imenu-after-jump-hook.

It also seems to me that not all people are going to like the pulse
highlighting and maybe want something different, e.g. recenter the
window. Thus turning this pulsing into a custom :options value for the
to be created hook would solve this problem as well.

> [...] But even now, you can just advise one of the two relevant
> functions (xref-push-marker-stack or xref-pop-marker-stack).

That's always true.

-ap

Re: xref-find-matches and stuff

[Vaidheeswaran C]

On Friday 17 July 2015 12:29 AM, Andreas Politz wrote:

> not all people are going to like the pulse highlighting

Count me in that set.

Re: xref-find-matches and stuff

[Dmitry Gutov]

On 07/16/2015 09:59 PM, Andreas Politz wrote:

> It seems to me that having a hook run after jumping somewhere is
> pretty standard behaviour, e.g. occur-mode-find-occurrence-hook,
> find-tag-hook, next-error-hook, imenu-after-jump-hook.

Added, two of them. Hopefully, we'll be able to consolidate all of those 
into just a couple of hooks, sometime.

> It also seems to me that not all people are going to like the pulse
> highlighting and maybe want something different, e.g. recenter the
> window. Thus turning this pulsing into a custom :options value for the
> to be created hook would solve this problem as well.

Feel free to add :options. I'm not sure it's particularly beneficial, 
considering the user can see the default value (which includes them) anyway.

>> [...] But even now, you can just advise one of the two relevant
>> functions (xref-push-marker-stack or xref-pop-marker-stack).
>
> That's always true.

Partially true. Before, there was no counterpart to 
xref-pop-marker-stack, just a variable holding the stack.

Re: xref-find-matches and stuff

[Andreas Politz]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 07/16/2015 09:59 PM, Andreas Politz wrote:
>
>> [...]  a hook [...]
>
> Added, two of them. [...]

Great, thanks.

-ap

Re: xref-find-matches and stuff

[Vitalie Spinu]

I got a bit of time to look through the abstractions. Here is my
concrete suggestion.

As I already said in other place, current old-school polymorphism with
`xref-find-function`, xref-identifier-at-point-function' and
`xref-identifier-completion-table-function` is rather clumsy if you want
to build an UI for backend manipulation. 

You are already using class machinery, so how about to shift the focus
on backends and use generics instead. More concretely:

    
   (cl-defmethod xref-definitions ((b (eql elisp)) id)
     (message "elisp definitions: %s" id))
   
   (cl-defmethod xref-references ((b (eql etags)) id)
     (message "etags references: %s" id))
   
   (cl-defmethod xref-apropos ((b (eql elisp)) pattern)
     (message "elisp apropos: %s" pattern))

   (cl-defmethod xref-matches ((b (eql elisp)) pattern)
     (message "elisp matches: %s" pattern))

Then you can simply.
   
   (xref-definitions 'etags "blabla")
   (xref-matches 'elisp "blabla")


Once that's done, you can introduce

  (setq xref-backends '(elisp etags ...))

for current backends. Then adding functionality to add/remove backends
from current list and rotate through different preset configurations of
backends is a piece of cake.

Alternatively you can still go with more monolithic

  (cl-defmethod xref-find ((b (eql elisp) action id)))

but this has an obvious drawback that whenever you add a new
`xref-find-SOMETHING` you need to go back and modify all `xref-find`
methods. Not quite an option for a good design IMO.

As to removing duplicates, I don't see it as a big deal. You can define
`xref-loc-name` methods for location objects to extract a
wanna-be-unique name to be specifically used in comparison between
backends. Even if you cannot get rid of all duplicates it's the task of
the *xref* UI to provide sorting and duplication removal on specific
attributes.

If you are willing to wait till 20th of May I can propose patches for
this abstraction change and other UI ideas that we have discussed
elsewhere.


  Vitalie

Re: xref-find-matches and stuff

[Dmitry Gutov]

On 05/05/2015 05:35 PM, Vitalie Spinu wrote:

> You are already using class machinery, so how about to shift the focus
> on backends and use generics instead. More concretely:

Look good to me. The initial implementation by Helmut 
(https://github.com/ellerh/xref) did almost exactly that, but it had to 
me more awkward because of preloading issues (and EIEIO not supporting 
(eql elisp) in the type qualifier.

> Once that's done, you can introduce
>
>    (setq xref-backends '(elisp etags ...))

Nope:

      (setq xref-backend 'elisp)

> for current backends. Then adding functionality to add/remove backends
> from current list and rotate through different preset configurations of
> backends is a piece of cake.

You would do (setq xref-backend 'xref-multi) in your minor mode, and 
write implementations delegating to the list that your mode stores 
internally.

> You can define
> `xref-loc-name` methods for location objects to extract a
> wanna-be-unique name to be specifically used in comparison between
> backends.

That's an interesting proposition. Will it help with duplication between

(defun system-name)

from elisp and

DEFUN ("system-name", Fsystem_name,

from etags?

> Even if you cannot get rid of all duplicates it's the task of
> the *xref* UI to provide sorting and duplication removal on specific
> attributes.

I don't really understand this, but I probably disagree. :)

> If you are willing to wait till 20th of May I can propose patches for
> this abstraction change and other UI ideas that we have discussed
> elsewhere.

Changing backends to use generics is fairly orthogonal to what I have 
planned next, so please do that at your convenience.

As for the UI, if you don't simply mean the said mode, but also the 
alternative xref output, I'd rather you wait with that for a while. The 
generics-based rendering and the grouping widgets will bring a lot of 
changes.

Re: xref-find-matches and stuff

[Vitalie Spinu]

 >>> Dmitry Gutov on Tue, 5 May 2015 18:23:54 +0300 wrote:

 > Nope:

 >      (setq xref-backend 'elisp)

 >> for current backends. Then adding functionality to add/remove backends
 >> from current list and rotate through different preset configurations of
 >> backends is a piece of cake.

 > You would do (setq xref-backend 'xref-multi) in your minor mode, and write
 > implementations delegating to the list that your mode stores
 > internally.

I don't quite understand your opposition. Multiple backends is such a
common requirement that you will need to set 'xref-multi by
default. Having a parallel xref-multi engine would necessary involve
replication of variable names and, hence, needless burden on the users
and developers. What's the point?

Anyways, I expect it to be no more than 100 lines of code. So I can
implement it both ways and we can see which one is better afterwards.

 >> You can define `xref-loc-name` methods for location objects to
 >> extract a wanna-be-unique name to be specifically used in comparison
 >> between backends.

 > That's an interesting proposition. Will it help with duplication between

 > (defun system-name)

 > from elisp and

 > DEFUN ("system-name", Fsystem_name,

 > from etags?

Of course. This one is a particularly interesting example. Both etags
and elisp return name and file, but elisp doesn't return a line. So in
order to make them compatible you must avoid using lines in the uniform
identifier. So it could be in file:symbol:type format, or
editfnc.c:system-name:defun in your example.

I think this is a general enough convention that will probably work
reasonably well in most of the cases. You can also have several levels
of "uniqueness" of the uniform identifier, xref-uname-lax, xref-uname
and xref-uname-strict. The lax one can be file:symbol; the strict one
can be file:symbol:type:line:column.

In *xref* buffer you might have a command that will rotate through
levels of "uniqueness".

 >> Even if you cannot get rid of all duplicates it's the task of
 >> the *xref* UI to provide sorting and duplication removal on specific
 >> attributes.

 > I don't really understand this, but I probably disagree. :)

I meant that if you already got into *xref* buffer with many
alternatives there should be commands to refine duplicates on different
criteria, just as I have described above.

The problem still remains when you have small number of matches that
give the same location. You don't want to popup *xref* in that case. But
I think the above uniform identifier idea will reduce this problem to
minimum.

 >> If you are willing to wait till 20th of May I can propose patches for
 >> this abstraction change and other UI ideas that we have discussed
 >> elsewhere.

 > Changing backends to use generics is fairly orthogonal to what I have
 > planned next, so please do that at your convenience.

Ok.

 > As for the UI, if you don't simply mean the said mode, but also the
 > alternative xref output, I'd rather you wait with that for a
 > while. The generics-based rendering and the grouping widgets will
 > bring a lot of changes.

This looks like an over-engineering to me. Why to allow locations to
display differently? To bother users with different colors and
arrangements? Do you have examples when different location classes need
to display differently and simple non-generic display would not work?

Anyways, I want my locations to display the same and have same UI for
all backends, so I am sticking to my tabular display. Would be nice if
your new generic system would allow me to easily overwrite the default
display methods for all xref-locations at once. I don't want to hunt
them one by one afterwards.

Thanks,

 Vitalie

Re: xref-find-matches and stuff

[Dmitry Gutov]

On 05/05/2015 08:17 PM, Vitalie Spinu wrote:

> I don't quite understand your opposition. Multiple backends is such a
> common requirement that you will need to set 'xref-multi by
> default.

I guess I don't see it that way. We currently don't have multiple values 
in any major mode. The third-party project I intend to use xref in, 
won't have them too. Feel free to make the case with specific examples; 
maybe implement a usable elisp+etags configuration.

Without multiple values in the core, I can avoid asking questions like:

- If I'm a major mode, what do I do with xref-backends? Do I set it to a 
single-value list? Do I prepend to it?

Out usual strategy is to use a hook (like a list of functions), but in 
those kind of setups, we usually run the hooks until one of the elements 
returns something interesting, rather than combining all results.

- If I'm a minor mode, what do I do with it?

- How are xref-identifier-completion-table-function values combined? 
completion-table-merge, or completion-table-in-turn? What are 
performance characteristics of calling completion-table-merge on five 
tables? Are you aware that a completion table produced by either of 
these functions can't handle certain advanced metadata that might be 
present in some of the original completion tables?

> Having a parallel xref-multi engine would necessary involve
> replication of variable names and, hence, needless burden on the users
> and developers. What's the point?

What variables? I can only see one: xref-multi-backends. Then you set 
xref-backend to `xref-multi' in the minor mode function and implement 
the four or so methods on it, delegating each to the current list in 
xref-multi-backends, and combining the results somehow (nconc, for most 
of them).

> Anyways, I expect it to be no more than 100 lines of code. So I can
> implement it both ways and we can see which one is better afterwards.

Since the simplest implementation can be very simple, it can also be an 
argument not to have it in the core. Backend author can make the choice, 
if it comes with this low a cost.

> Of course. This one is a particularly interesting example. Both etags
> and elisp return name and file, but elisp doesn't return a line. So in
> order to make them compatible you must avoid using lines in the uniform
> identifier. So it could be in file:symbol:type format, or
> editfnc.c:system-name:defun in your example.

Yup. Or rather /path/to/editfns.c:system-name:defun (etags has the full 
path, elisp doesn't; we should expand it).

> I think this is a general enough convention that will probably work
> reasonably well in most of the cases. You can also have several levels
> of "uniqueness" of the uniform identifier, xref-uname-lax, xref-uname
> and xref-uname-strict. The lax one can be file:symbol; the strict one
> can be file:symbol:type:line:column.

The last one is redundant: if we have file, line and column, we don't 
need the symbol and its type.

However, note that both representations you've chosen require a file 
name. Currently, an xref could only include some info that it needs to 
find out the file name (lazily), and it can return something else as its 
"group". We can only enable duplicate-remove-ability for certain types 
of xrefs, though.

> In *xref* buffer you might have a command that will rotate through
> levels of "uniqueness".

FWIW, that's too many user choices for my liking.

> I meant that if you already got into *xref* buffer with many
> alternatives there should be commands to refine duplicates on different
> criteria, just as I have described above.

See above. Don't ask the user to do what a machine could. If unwanted 
duplicates happen, we should write smarter backends.

> The problem still remains when you have small number of matches that
> give the same location. You don't want to popup *xref* in that case. But
> I think the above uniform identifier idea will reduce this problem to
> minimum.

I think duplicates removal is kind of important, especially for 
xref-find-definitions. It's the difference between having to choose and 
being taken to the right option immediately.

> This looks like an over-engineering to me. Why to allow locations to
> display differently? To bother users with different colors and
> arrangements? Do you have examples when different location classes need
> to display differently and simple non-generic display would not work?

It's an implementation technique. The fact that a backend implementor 
will be able to insert a nyancat animation after every line doesn't mean 
they will; everyone can mess will any Elisp function anyway, and don't 
get me started on custom faces that don't fit the default theme, or any 
light-background one. Anyway, the fact that some people could too easily 
change a piece of Emacs behavior to their liking shouldn't be an 
argument against this.

So far it looks right:

- You want shortened file paths, a "file path" group class would fit.

- Helmut suggested that xref-find-apropos output for Elisp looks closer 
to M-x apropos. We could do that with special elements. Likewise, 
apropos output for a different language would look different and use 
different properties, which we cannot anticipate. Flexibility here 
should be beneficial.

- Elements can have different roles; for instance, a group could also be 
an xref. Like, for instance, xref-find-references can match something in 
a function declaration (name or argument), and some references inside 
it. The output could have the method displayed as a match, as well as 
group the matches inside it.

- In certain cases, a file name might be a match for a search, so it 
could be fitted into the hierarchy. In an environment where classes and 
file names are usually tied (like Java or Ruby), a rename operation, 
ideally, should handle both.

If, on the other hand, a fixes set of structures is all we need for the 
data returned by the backends, the backend authors might limit 
themselves to the built-in ones, and avoid spending effort on extensions.

> Anyways, I want my locations to display the same and have same UI for
> all backends, so I am sticking to my tabular display.

If people add new structures, we should recommend to inherit from the 
built-in ones. So you'd still render them automatically, even if missing 
some extra fields.

 > Would be nice if
> your new generic system would allow me to easily overwrite the default
> display methods for all xref-locations at once. I don't want to hunt
> them one by one afterwards.

The location classes are unlikely to change much. We're displaying 
xref--xref instances, though, and that hierarchy will see some growth. 
Any suggestions for a better name, by the way? xref-item?

Re: xref-find-matches and stuff

[Vitalie Spinu]

 >>> Dmitry Gutov on Wed, 6 May 2015 03:48:14 +0300 wrote:

 > On 05/05/2015 08:17 PM, Vitalie Spinu wrote:
 >> I don't quite understand your opposition. Multiple backends is such
 >> a common requirement that you will need to set 'xref-multi by
 >> default.

 > Feel free to make the case with specific examples; maybe implement a
 > usable elisp+etags configuration.

I have the following patterns in mind:

  1) etags + mode-specific-backend 
  2) imenu-anywhere + mode-specific-backend
  3) etags + imenu-anywhere + mode-specific-backend

Imenu-anywhere [1] allows you to jump to imenu references in all files
with the same mode. It can get more refined as imenu-in-same-project
which could use some notion of a project.

I think everyone would like to have 2 or 3 by default because it makes
references available even if there is no repl running or tag tables
loaded. Imenu is also always uptodate, so should get higher priority.

Semantic references could also be used instead of imenu. I personally
don't use them.

 > - If I'm a major mode, what do I do with xref-backends? Do I set it
 > to a single-value list? Do I prepend to it?

That's major mode developers have to answer as they see fit. If you have
a clear semantics of how merged backends behave your job with xref is
done.

If you have a good UI that allows quick manipulation of backends, major
modes might want to define multiple backends to fit more user patterns.

 > - How are xref-identifier-completion-table-function values combined?
 > completion-table-merge, or completion-table-in-turn? What are performance
 > characteristics of calling completion-table-merge on five tables? Are you aware
 > that a completion table produced by either of these functions can't handle
 > certain advanced metadata that might be present in some of the original
 > completion tables?

That's the problem to be solved. And someone has to solve it anyways.

 >> Having a parallel xref-multi engine would necessary involve
 >> replication of variable names and, hence, needless burden on the
 >> users and developers. What's the point?

 > What variables? I can only see one: xref-multi-backends. 

 > Then you set xref-backend to `xref-multi' in the minor mode function
 > and implement the four or so methods on it, delegating each to the
 > current list in xref-multi-backends, and combining the results
 > somehow (nconc, for most of them).

You might be right. The only complication is that xref-backend is set by
major mode so I will have to overwrite it by xref-multi everywhere.

 > However, note that both representations you've chosen require a file
 > name. Currently, an xref could only include some info that it needs
 > to find out the file name (lazily), and it can return something else
 > as its "group". We can only enable duplicate-remove-ability for
 > certain types of xrefs, though.

I am not sure I understand. The only xref-loc that doesn't have explicit
:file slot is xref-buffer-location but you can easily extract the file
or simply use the buffer name for non-visiting files.

 > So far it looks right:

Ok. Let's see how it goes. My gut feeling is that it might get too
complex and very few backends will actually extend it.

 > The location classes are unlikely to change much. We're displaying
 > xref--xref instances, though, and that hierarchy will see some
 > growth. Any suggestions for a better name, by the way? xref-item?

Isn't this class a complication? Cannot you just have optional
:description slot in root xref-location class? Some backends will be
able to return directly the :description field.

By merging xref--xref and xref-location you will reduce the clutter and
simply call "xref" what you call "xref-location" right now. You get
"xref" instead of "xref-location" - "elisp-xref", "etag-xref"
etc. Cannot get simpler!

Displaying xref--xref instead of the xref-location is rather awkward
design from the backend developer prospective.  For each of the location
class you will need to define a pairing `xref--xref` class:

 (defclass my-xref-location ...)
 (defclass my-xref-xref ...)


BTW, :description is not very suggestive to me. How about :locstring to
emphasize that it's a string representation of the location?


  Vitalie



[1] https://github.com/vspinu/imenu-anywhere

Re: xref-find-matches and stuff

[Dmitry Gutov]

On 05/06/2015 04:54 PM, Vitalie Spinu wrote:

> I have the following patterns in mind:

Allow me to offer some sloppy critique.

>    1) etags + mode-specific-backend

Why not use etags, rather than rely on the major mode and it being able 
to only each the buffers currently open in Emacs?

If the problem is that etags is imperfect, let's refine its regexps. If 
its getting out of date is the problem, we could automate something in 
that area too, or switch to something more advanced. E.g. 
https://github.com/leoliu/ggtags boasts automatic updates as the number 
2 in its features list.

>    2) imenu-anywhere + mode-specific-backend

As long as tags are not available, why not simply leave it to the major 
mode? It's sure to be able to provide a more precise data than an 
adapter on top of Imenu.

There's not even a clear standard on the Imenu index structure: e.g. 
flat vs. nested.

>    3) etags + imenu-anywhere + mode-specific-backend

Use tags. :)

I think this list is a reasonable case for a hook of the usual kind 
though: whichever backend returns non-nil to 
`identifier-completion-table', gets used.

> Semantic references could also be used instead of imenu. I personally
> don't use them.

CEDET could be rough around the edges, but it'll beat Imenu and etags. 
Its parsing engine, the symbol database and integration with external 
tools map to all backend features we might have planned.

> That's the problem to be solved. And someone has to solve it anyways.

Like I said, I'd prefer to hold off on dealing with this yet. But don't 
let me stop you.

We could as well introduce merged backends as a core feature later, by 
setting xref-backend to a list, or use a combinator function. Although 
the latter might be tricky with a generics-based backends.

> You might be right. The only complication is that xref-backend is set by
> major mode so I will have to overwrite it by xref-multi everywhere.

That's what minor modes often do.

> I am not sure I understand. The only xref-loc that doesn't have explicit
> :file slot is xref-buffer-location but you can easily extract the file
> or simply use the buffer name for non-visiting files.

I mean that including a file name is not mandated by the xref interface. 
Backends are allowed to provide their own xref-location implementations, 
and some of those might be missing the file slot as well.

> Ok. Let's see how it goes. My gut feeling is that it might get too
> complex and very few backends will actually extend it.

We'll see. And few backends extending it sounds about ideal to me. We 
don't want the user experience to vastly differ between backends either.

> Isn't this class a complication? Cannot you just have optional
> :description slot in root xref-location class? Some backends will be
> able to return directly the :description field.

It might be. But so far, xref-location is effectively just an interface. 
Adding a slot will take us closer to inheritance.

Or a more practical side, maybe xref-description can become a generic 
method. If can certainly be deduced from the contents of 
xref-elisp-location, as well as xref-etags-location and 
xref-buffer-location. xref-file-location would need to store it in a slot.

Or, along the lines of the OO proposal, it could be entirely replaced 
with an xref-buffer-print generic method. You don't ask a location its 
summary, just ask the object to print itself.

That may be suboptimal from the standpoint of an alternative interface, 
though.

> By merging xref--xref and xref-location you will reduce the clutter and
> simply call "xref" what you call "xref-location" right now. You get
> "xref" instead of "xref-location" - "elisp-xref", "etag-xref"
> etc. Cannot get simpler!

That is indeed simpler, but the package is called xref already, maybe we 
should use some suffix in the name of the location base class.

Thanks for the merge suggestion either way.

> Displaying xref--xref instead of the xref-location is rather awkward
> design from the backend developer prospective.  For each of the location
> class you will need to define a pairing `xref--xref` class:

I suppose, if the "render me" method is defined on xref-xref, rather 
than on xref-location. But if not, xref-xref doesn't serve a lot of 
purpose indeed, right now it's simply a two-slot container without any 
associated generic methods.

Keeping it separate would be useful if we find two orthogonal 
hierarchies peeking though the planned functionality, but otherwise, 
merge it is.

> BTW, :description is not very suggestive to me. How about :locstring to
> emphasize that it's a string representation of the location?

Replace one common word with a portmanteau?

It's not really a representation, just something to show in the buffer. 
Call it summary maybe? Both descriptions and summaries are usually strings.

Re: xref-find-matches and stuff

[Vitalie Spinu]

 >>> Dmitry Gutov on Thu, 7 May 2015 03:09:37 +0300 wrote:

 >> 1) etags + mode-specific-backend

 > Why not use etags

 >> 2) imenu-anywhere + mode-specific-backend

 > As long as tags are not available, why not simply leave it to the
 > major mode?
 
 >> 3) etags + imenu-anywhere + mode-specific-backend

 > Use tags. :)

There are many situations when you cannot or might not want to use tags:

   - In many common situations tags don't apply. For example open a file
     with an example, or create a temporary scratch file to experiment
     with features.

   - There are plenty of languages that don't have tags support and
     nobody would care to add it.

   - A lot of less techy people would not use etags because it's a
     command line tool which is usually called with arguments. You said
     you are not using tags routinely. I do, and it's frustrating to run
     to terminal and visit tag table each time you need to jump to a
     reference in a new context.

   - Current emacs tags UI is clumsy and not project oriented. I am now
     always annoyed when elisp-xref takes me to emacs C source and from
     there etags brings me to my R projects.

   - When working with multiple projects, tags generation is
     frustrating. It assumes that you know all the projects and all
     non-project files you will be dealing with. That's commonly not
     true.

You would like to use imenu because:

   - Imenu is meaningful in non-programing modes and for non-visiting
     buffers.

   - Virtually all modes already define imenu, so it works out of the
     box everywhere, even if you read a language in which you will never
     write.

   - And finally, imenu operates on what you work on - open files - you
     don't get thousands of symbols from etags to deal with.

I have been using imenu-anywhere for a couple of years now and it has
been a much better, more uniform and more consistent experience than
anything else I have tried.

 > I think this list is a reasonable case for a hook of the usual kind
 > though: whichever backend returns non-nil to
 > `identifier-completion-table', gets used.

Yes. That can be. But I am setting xref-prompt-for-identifier /and
recommend it widely ;)/. That implies a merge.

 >> That's the problem to be solved. And someone has to solve it anyways.

 > Like I said, I'd prefer to hold off on dealing with this yet. But
 > don't let me stop you.

Sure, I will give it a try in a couple of weeks.

 >> I am not sure I understand. The only xref-loc that doesn't have explicit
 >> :file slot is xref-buffer-location but you can easily extract the file
 >> or simply use the buffer name for non-visiting files.

 > I mean that including a file name is not mandated by the xref
 > interface. 

Ok, I see. But uniform identifier (xref-uname) method should then be a
requirement.

 >> Isn't this class a complication? Cannot you just have optional
 >> :description slot in root xref-location class? Some backends will be
 >> able to return directly the :description field.

 > It might be. But so far, xref-location is effectively just an
 > interface. Adding a slot will take us closer to inheritance.

And that's not a bad thing. You would like to have as much uniformity as
you can.

I wonder if the whole interface could be made a bit stricter. It always
makes sense to have a string representation of the container (file-name,
buffer-name, document name), right? So maybe get the common root slot
:container or :context?

Also :line and :column slots always make sense even if they are not
populated from the start. For example xref-buffer-location can still
have :line and :column optional slots because some backends (like imenu)
could already have those precomputed.

 > Or a more practical side, maybe xref-description can become a generic
 > method. 

Could be. Then the backend can decide whether to store it as part of the
object or not.

On the other hand you would want to be able to regexp through the
descriptions without sniffing through all the files? So it might be
reasonable to "kindly suggest" pre-computing that information by adding
an optional :description slot.

 > If can certainly be deduced from the contents of xref-elisp-location,
 > as well as xref-etags-location and
 > xref-buffer-location. xref-file-location would need to store it in a
 > slot.

 > Or, along the lines of the OO proposal, it could be entirely replaced with an
 > xref-buffer-print generic method. You don't ask a location its summary, just ask
 > the object to print itself.

This doesn't fit nicely with my tabular-display intention, nor, I think,
with your custom displays idea. I would prefer a description method.

 > That may be suboptimal from the standpoint of an alternative
 > interface, though.

Exactly!

 >> By merging xref--xref and xref-location you will reduce the clutter
 >> and simply call "xref" what you call "xref-location" right now. You
 >> get "xref" instead of "xref-location" - "elisp-xref", "etag-xref"
 >> etc. Cannot get simpler!

 > That is indeed simpler, but the package is called xref already

What's the problem? I can still define a root class `xref` and use
`xref` to mean "an object that stores a location metadata".

Alternatively `xref-root` for the root class and `XXX-xref` for
everything else. 

 > but otherwise, merge it is.

Great!

 >> BTW, :description is not very suggestive to me. How about :locstring to
 >> emphasize that it's a string representation of the location?

 > It's not really a representation, just something to show in the
 > buffer. Call it summary maybe? Both descriptions and summaries are
 > usually strings.

My problem with "description" is that it's too general. Description can
mean anything - documentation of the symbol, description of the xref
object, description of the context at point, or, as it's currently used,
a declaration of the object from the original location.

Summary is better but it's also open to interpretation. How about saying
what it is - "declaration" ? Or even better "content" to emphasize that
it's a content at that physical location? Then print methods can
manipulate this string in whatever way they see fit.

  Vitalie

Re: xref-find-matches and stuff

[Dmitry Gutov]

On 05/07/2015 03:24 PM, Vitalie Spinu wrote:

> There are many situations when you cannot or might not want to use tags:

None of them, AFAICS, provide justification to use the "merge" strategy 
for backends, rather than "first one wins". And the latter is easier to 
implement.

>     - In many common situations tags don't apply. For example open a file
>       with an example, or create a temporary scratch file to experiment
>       with features.

If xref-backends is a list with a value like '(etags elisp imenu), first 
we'll try etags. If it's not applicable (no tags file), then the major 
mode, if there's an entry for it. Otherwise, imenu (that element has 
probably been added by a minor mode).

>     - There are plenty of languages that don't have tags support and
>       nobody would care to add it.

For people with aversion to contributing patches upstream, ctags also 
supports specifying necessary regexps on the command line: 
https://gist.github.com/wereHamster/1299204

That should work at least as well as most uses of Imenu.

>     - A lot of less techy people would not use etags because it's a
>       command line tool which is usually called with arguments. You said
>       you are not using tags routinely. I do, and it's frustrating to run
>       to terminal and visit tag table each time you need to jump to a
>       reference in a new context.

Then don't use ctags. Use GNU Global, or something else. I see you've 
skipped me mentioning ggtags in the previous message.

>     - Current emacs tags UI is clumsy and not project oriented. I am now
>       always annoyed when elisp-xref takes me to emacs C source and from
>       there etags brings me to my R projects.

Recalling previous discussions on the subject, that's pretty hilarious. 
Sorry. Like Stefan suggested before, maybe etags should look for the 
tags file automatically in the parent directories of the current file. 
Not everyone would like that, but it seems like it would be a better 
default.

>     - When working with multiple projects, tags generation is
>       frustrating. It assumes that you know all the projects and all
>       non-project files you will be dealing with. That's commonly not
>       true.

This is a big vague, so I'm not sure exactly what to propose. But it can 
probably be improved as well.

To sum up, my point is that if there are obvious deficiencies in the 
tools we have, it's better to improve or replace some of them, rather 
than try to simply cobble them together and let the user sort out the 
overlaps.

> You would like to use imenu because:

None of these examples seem to be calling for merging with results 
returned by etags either.

> I have been using imenu-anywhere for a couple of years now and it has
> been a much better, more uniform and more consistent experience than
> anything else I have tried.

That's good. But that may speak more about the availability and ease of 
use of other tools, rather than about imenu-anywhere itself.

> Yes. That can be. But I am setting xref-prompt-for-identifier /and
> recommend it widely ;)/. That implies a merge.

I don't understand. How does it imply that?

> Ok, I see. But uniform identifier (xref-uname) method should then be a
> requirement.

We can try to make it that, but it's not easy to come up with a good 
specification.

And what about etags? If we want to remove duplicates when merging etags 
with any third party language backend, it seems like etags should return 
close-to-ideal unames for tags in any language. Certainly without 
duplicates.

I can imagine us swinging it for Emacs Lisp (language with no classes in 
the modern-OO sense, no namespaces), but Ruby, Java, JavaScript?

>   > It might be. But so far, xref-location is effectively just an
>   > interface. Adding a slot will take us closer to inheritance.
>
> And that's not a bad thing. You would like to have as much uniformity as
> you can.

Actually, we'll probably remove xref-location altogether (it currently 
has no specialized method implementations now). Without this mandated 
paren class, locations can be structs as well as classes (cl-defmethod 
supports both), and we don't need to worry about classes being able to 
inherit from structs or vice versa.

> I wonder if the whole interface could be made a bit stricter. It always
> makes sense to have a string representation of the container (file-name,
> buffer-name, document name), right? So maybe get the common root slot
> :container or :context?

There might be no container. There'll likely be several containers 
nested inside one another, at least in certain outputs.

It seems to call for a separate types of values (different containers). 
You'll still be able to flatten that tree during output.

I'm not sure yet whether an element should have a reference to its 
parent container.

> Also :line and :column slots always make sense even if they are not
> populated from the start. For example xref-buffer-location can still
> have :line and :column optional slots because some backends (like imenu)
> could already have those precomputed.

But why? You can ask an xref-buffer-location for its line and column, 
and it'll tell you. No need to create slots for them.

> On the other hand you would want to be able to regexp through the
> descriptions without sniffing through all the files? So it might be
> reasonable to "kindly suggest" pre-computing that information by adding
> an optional :description slot.

If we find C-s insufficient, we could ask all of them for descriptions 
again. So far, the vast majority of time xref-find-references spends in 
is opening files. Everything else seems to be an order of magnitude faster.

If not, the interface could also cache descriptions separately.

> This doesn't fit nicely with my tabular-display intention, nor, I think,
> with your custom displays idea. I would prefer a description method.

Okay. Always having some sort of line identifying a reference makes 
sense anyway.

> What's the problem? I can still define a root class `xref` and use
> `xref` to mean "an object that stores a location metadata".

No problem, just fairly awkward, in my view. Also, M-. will always ask 
you which one you mean.

> My problem with "description" is that it's too general. Description can
> mean anything - documentation of the symbol, description of the xref
> object, description of the context at point,

True.

> Summary is better but it's also open to interpretation. How about saying
> what it is - "declaration" ? Or even better "content" to emphasize that
> it's a content at that physical location? Then print methods can
> manipulate this string in whatever way they see fit.

I agree summary is not ideal.

"Declaration" doesn't work in all situations we'll use it in: 
declaration of a grep match? Declaration of that place this function has 
been used in?

"Content" is better, but the elisp backend doesn't return buffer 
contents (for the obvious reason: it's not opening any files until 
asked). I can see other backends doing the same. Still, it's probably 
the best of the current options.

"Display" would be closer, I think, but it's Emacs slang, and not 
something any reader will understand.

Re: xref-find-matches and stuff

[Francesco Potortì]

Vitalie Spinu:
>>     - There are plenty of languages that don't have tags support and
>>       nobody would care to add it.
>
>For people with aversion to contributing patches upstream, ctags also 
>supports specifying necessary regexps on the command line: 
>https://gist.github.com/wereHamster/1299204

Etags provides this feature too.  From etags --help:

-r REGEXP, --regex=REGEXP or --regex=@regexfile
        Make a tag for each line matching a regular expression pattern
	in the following files.  {LANGUAGE}REGEXP uses REGEXP for LANGUAGE
	files only.  REGEXFILE is a file containing one REGEXP per line.
	REGEXP takes the form /TAGREGEXP/TAGNAME/MODS, where TAGNAME/ is
	optional.  The TAGREGEXP pattern is anchored (as if preceded by ^).
	If TAGNAME/ is present, the tags created are named.
	For example Tcl named tags can be created with:
	  --regex="/proc[ \t]+\([^ \t]+\)/\1/.".
	MODS are optional one-letter modifiers: `i' means to ignore case,
	`m' means to allow multi-line matches, `s' implies `m' and
	causes dot to match any character, including newline.

-R, --no-regex
        Don't create tags from regexps for the following files.

Note that new languages cannot be defined on the command line, even if
this is a problem only for tag files containing tags for mixed language
sources.  And anyway adding this capability to etags would be almost
trivial.

Re: xref-find-matches and stuff

[Dmitry Gutov]

On 05/08/2015 04:02 PM, Francesco Potortì wrote:

> Etags provides this feature too.

Yes, sorry. In my mind, etags and ctags are interchangeable, except the 
latter supports more languages out of the box.

Re: xref-find-matches and stuff

[martin rudalics]

 > And anyway adding this capability to etags would be almost
 > trivial.

Francesco,

while you're here: Could you pretty please make etags recognize
DEFVAR_BOOL, DEFVAR_LISP, DEFVAR_KBOARD and friends?

Many thanks in advance, martin

Re: xref-find-matches and stuff

[Eli Zaretskii]

> Date: Sun, 10 May 2015 14:31:13 +0200
> From: martin rudalics <rudalics@gmx.at>
> Cc: Vitalie Spinu <spinuvit@gmail.com>, Helmut Eller <eller.helmut@gmail.com>,
> 	emacs-devel <emacs-devel@gnu.org>
> 
>  > And anyway adding this capability to etags would be almost
>  > trivial.
> 
> Francesco,
> 
> while you're here: Could you pretty please make etags recognize
> DEFVAR_BOOL, DEFVAR_LISP, DEFVAR_KBOARD and friends?

Why is that important?  Doesn't --regex do a good enough job with
them?

Re: xref-find-matches and stuff

[martin rudalics]

 > Why is that important?  Doesn't --regex do a good enough job with
 > them?

With M-. it would have been nice if this worked out of the box.

martin

Re: xref-find-matches and stuff

[Eli Zaretskii]

> Date: Mon, 11 May 2015 12:08:45 +0200
> From: martin rudalics <rudalics@gmx.at>
> CC: pot@gnu.org, dgutov@yandex.ru, spinuvit@gmail.com, 
>  eller.helmut@gmail.com, emacs-devel@gnu.org
> 
>  > Why is that important?  Doesn't --regex do a good enough job with
>  > them?
> 
> With M-. it would have been nice if this worked out of the box.

Not sure I understand: it does already work, if you say "make TAGS".
What am I missing?

Re: xref-find-matches and stuff

[martin rudalics]

 > Not sure I understand: it does already work, if you say "make TAGS".

Alas not here (on Windows).

 > What am I missing?

When I am in frame.c with point on the first item of the line reading

   Vmake_pointer_invisible = Qt;

and I type M-. Emacs opens the file globals.h and moves there to the
line reading

#define Vmake_pointer_invisible globals.f_Vmake_pointer_invisible

which is the only entry in the TAGS file containing
Vmake_pointer_invisible.  So what am _I_ missing?

martin

Re: xref-find-matches and stuff

[Eli Zaretskii]

> Date: Tue, 12 May 2015 11:36:03 +0200
> From: martin rudalics <rudalics@gmx.at>
> CC: pot@gnu.org, emacs-devel@gnu.org
> 
> > Not sure I understand: it does already work, if you say "make TAGS".
>
> Alas not here (on Windows).

You mean, "make TAGS" doesn't run or doesn't create TAGS files in src/
and in lisp/ ?  It does for me.

> When I am in frame.c with point on the first item of the line reading
> 
>   Vmake_pointer_invisible = Qt;
> 
> and I type M-. Emacs opens the file globals.h and moves there to the
> line reading
> 
> #define Vmake_pointer_invisible globals.f_Vmake_pointer_invisible
> 
> which is the only entry in the TAGS file containing
> Vmake_pointer_invisible.  So what am _I_ missing?

I'm not sure.  Are we still talking about support for DEFVAR_LISP
etc.?  If so, perhaps you need to say

  C-u M-. make-pointer-invisible RET

?

IOW, when etags finds those DEF* declarations in C files, it tags the
Lisp symbol name, not the C symbol name.  See the regular expressions
we pass to etags: they instruct etags to take the first token that
matches "[^"]+" after an opening parenthesis.  If we want to tag the C
symbol, we need to add another regexp for that.

Other than that, I see no problems in what you get from M-. because
Vmake_pointer_invisible is indeed a C macro defined on that line of
globals.h.  What did you expect to get instead?

Re: xref-find-matches and stuff

[Dmitry Gutov]

On 05/12/2015 07:15 PM, Eli Zaretskii wrote:

> I'm not sure.  Are we still talking about support for DEFVAR_LISP
> etc.?  If so, perhaps you need to say
>
>    C-u M-. make-pointer-invisible RET

That doesn't work when tag-symbol-match-p is not in 
etags-xref-find-definitions-tag-order (and the etags backend is used).

Or to put it another way, 'make tags' doesn't create explicit tag names 
for those declarations.

Re: xref-find-matches and stuff

[Eli Zaretskii]

> Cc: pot@gnu.org, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Tue, 12 May 2015 19:46:35 +0300
> 
> On 05/12/2015 07:15 PM, Eli Zaretskii wrote:
> 
> > I'm not sure.  Are we still talking about support for DEFVAR_LISP
> > etc.?  If so, perhaps you need to say
> >
> >    C-u M-. make-pointer-invisible RET
> 
> That doesn't work when tag-symbol-match-p is not in 
> etags-xref-find-definitions-tag-order (and the etags backend is used).

Then it's a bug, because:

  fgrep make-pointer-invisible src/TAGS
  => src/TAGS:8019:  DEFVAR_LISP ("make-pointer-invisible"^?5068,158129

and

  emacs -Q
  M-x visit-tags-table RET src/ RET
  M-x find-tag RET make-pointer-invisible RET

shows the corresponding DEFVAR_LISP line from frame.c.

> Or to put it another way, 'make tags' doesn't create explicit tag names 
> for those declarations.

But tag-implicit-name-match-p should match it, no?

Re: xref-find-matches and stuff

[Eli Zaretskii]

> Date: Tue, 12 May 2015 20:37:20 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: rudalics@gmx.at, pot@gnu.org, emacs-devel@gnu.org
> 
> > Cc: pot@gnu.org, emacs-devel@gnu.org
> > From: Dmitry Gutov <dgutov@yandex.ru>
> > Date: Tue, 12 May 2015 19:46:35 +0300
> > 
> > On 05/12/2015 07:15 PM, Eli Zaretskii wrote:
> > 
> > > I'm not sure.  Are we still talking about support for DEFVAR_LISP
> > > etc.?  If so, perhaps you need to say
> > >
> > >    C-u M-. make-pointer-invisible RET
> > 
> > That doesn't work when tag-symbol-match-p is not in 
> > etags-xref-find-definitions-tag-order (and the etags backend is used).
> 
> Then it's a bug, because:
> 
>   fgrep make-pointer-invisible src/TAGS
>   => src/TAGS:8019:  DEFVAR_LISP ("make-pointer-invisible"^?5068,158129
> 
> and
> 
>   emacs -Q
>   M-x visit-tags-table RET src/ RET
>   M-x find-tag RET make-pointer-invisible RET
> 
> shows the corresponding DEFVAR_LISP line from frame.c.
> 
> > Or to put it another way, 'make tags' doesn't create explicit tag names 
> > for those declarations.
> 
> But tag-implicit-name-match-p should match it, no?

It didn't; I fixed that in 4f5382c.

Re: xref-find-matches and stuff

[Dmitry Gutov]

On 05/12/2015 10:57 PM, Eli Zaretskii wrote:

> It didn't; I fixed that in 4f5382c.

Thank you. But I was going to say that it doesn't match because NONAM 
(see the comment above make_tag) does not include the double-quote 
character (and tag-implicit-name-match-p obeys that).

Maybe it should be included?

Re: xref-find-matches and stuff

[Eli Zaretskii]

> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Tue, 12 May 2015 23:15:19 +0300
> Cc: emacs-devel@gnu.org
> 
> On 05/12/2015 10:57 PM, Eli Zaretskii wrote:
> 
> > It didn't; I fixed that in 4f5382c.
> 
> Thank you. But I was going to say that it doesn't match because NONAM 
> (see the comment above make_tag) does not include the double-quote 
> character (and tag-implicit-name-match-p obeys that).

Yes, that was the reason.  I just helped etags use the explicit-name
alternative in this case.

> Maybe it should be included?

That'd have required much more research, because its effect is on all
supported languages.  What I did fixed a problem whereby we weren't
telling etags where in the matched line to find the tag name, so it
couldn't make the decision whether an explicit tag name is necessary.
Using an implicit name is an optimization whose purpose is to keep the
TAGS file size as small as possible, so I see no reason to insist on
an implicit name.

Re: xref-find-matches and stuff

[Eli Zaretskii]

> Date: Wed, 13 May 2015 05:38:27 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: rudalics@gmx.at, emacs-devel@gnu.org
> 
> > Thank you. But I was going to say that it doesn't match because NONAM 
> > (see the comment above make_tag) does not include the double-quote 
> > character (and tag-implicit-name-match-p obeys that).
> 
> Yes, that was the reason.  I just helped etags use the explicit-name
> alternative in this case.

Btw, note the dramatic effect of this change on the results returned
by M-. for DEFVAR_LISP.

IOW, formerly src/TAGS would tell Emacs that all of this was the tag:

  DEFVAR_LISP ("pointer-make-invisible"

Which, of course, is incorrect, and caused a lot of false positives.

So the change to specify exactly where in the matched line is the tag
name was indeed in order, and the fact that \(..\) was used in the
regexp was a hint that it was intended.  If nothing else, the Emacs
manual explicitly advises to do so (which was how I became aware of
the feature in the first place).

Re: xref-find-matches and stuff

[Dmitry Gutov]

On 05/13/2015 07:23 PM, Eli Zaretskii wrote:

> Btw, note the dramatic effect of this change on the results returned
> by M-. for DEFVAR_LISP.
>
> IOW, formerly src/TAGS would tell Emacs that all of this was the tag:
>
>    DEFVAR_LISP ("pointer-make-invisible"
>
> Which, of course, is incorrect, and caused a lot of false positives.

Indeed, now it's much better. The only thing missing IMO is to default 
case-fold-search or tags-case-fold-search to nil. Or add a new 
etags-xref-find-definitions- variable for that.

As it is now, DEFVAR_LISP gives you two matches, and that not really 
correct.

Re: xref-find-matches and stuff

[Eli Zaretskii]

> Cc: rudalics@gmx.at, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Thu, 14 May 2015 03:46:04 +0300
> 
> Indeed, now it's much better. The only thing missing IMO is to default 
> case-fold-search or tags-case-fold-search to nil. Or add a new 
> etags-xref-find-definitions- variable for that.
> 
> As it is now, DEFVAR_LISP gives you two matches, and that not really 
> correct.

I'm not sure I agree with the not correct part.

I think the case sensitivity should be optional.

Re: xref-find-matches and stuff

[martin rudalics]

 > You mean, "make TAGS" doesn't run or doesn't create TAGS files in src/
 > and in lisp/ ?  It does for me.

It runs and creates the TAGS files.

 >> When I am in frame.c with point on the first item of the line reading
 >>
 >>    Vmake_pointer_invisible = Qt;
 >>
 >> and I type M-. Emacs opens the file globals.h and moves there to the
 >> line reading
 >>
 >> #define Vmake_pointer_invisible globals.f_Vmake_pointer_invisible
 >>
 >> which is the only entry in the TAGS file containing
 >> Vmake_pointer_invisible.  So what am _I_ missing?
 >
 > I'm not sure.  Are we still talking about support for DEFVAR_LISP
 > etc.?  If so, perhaps you need to say
 >
 >    C-u M-. make-pointer-invisible RET
 >
 > ?

I'll try to give a more meaningful example.  Suppose I'm on line 4818 of
w32term.c and want to know what focus_follows_mouse is and where it is
defined.  Typing M-. there prompts me with something like

.../globals.h
   #define focus_follows_mouse
     bool f_focus_follows_mouse;

How am I supposed to proceed from here?

 > IOW, when etags finds those DEF* declarations in C files, it tags the
 > Lisp symbol name, not the C symbol name.

But I'm in a C file and may not have any a priori knowledge that
focus_follows_mouse is a Lisp variable.

 > See the regular expressions
 > we pass to etags: they instruct etags to take the first token that
 > matches "[^"]+" after an opening parenthesis.  If we want to tag the C
 > symbol, we need to add another regexp for that.

That's precisely what I wanted.

 > Other than that, I see no problems in what you get from M-. because
 > Vmake_pointer_invisible is indeed a C macro defined on that line of
 > globals.h.  What did you expect to get instead?

That it goes to the definition of Vmake_pointer_invisible in frame.c
(and never to that in globals.h).

martin

Re: xref-find-matches and stuff

[Eli Zaretskii]

> Date: Wed, 13 May 2015 09:33:18 +0200
> From: martin rudalics <rudalics@gmx.at>
> CC: pot@gnu.org, emacs-devel@gnu.org
> 
> I'll try to give a more meaningful example.  Suppose I'm on line 4818 of
> w32term.c and want to know what focus_follows_mouse is and where it is
> defined.  Typing M-. there prompts me with something like
> 
> .../globals.h
>   #define focus_follows_mouse
>     bool f_focus_follows_mouse;
> 
> How am I supposed to proceed from here?

It depends on what you mean by "proceed".  I understand that you want
to see the corresponding DEFVAR_* line, is that right?

> > IOW, when etags finds those DEF* declarations in C files, it tags the
> > Lisp symbol name, not the C symbol name.
> 
> But I'm in a C file and may not have any a priori knowledge that
> focus_follows_mouse is a Lisp variable.

It's not a Lisp variable.  It's a C variable that has a Lisp binding.

> > See the regular expressions
> > we pass to etags: they instruct etags to take the first token that
> > matches "[^"]+" after an opening parenthesis.  If we want to tag the C
> > symbol, we need to add another regexp for that.
> 
> That's precisely what I wanted.

Your wish has been granted in commit 8d69f38.

> > Other than that, I see no problems in what you get from M-. because
> > Vmake_pointer_invisible is indeed a C macro defined on that line of
> > globals.h.  What did you expect to get instead?
> 
> That it goes to the definition of Vmake_pointer_invisible in frame.c
> (and never to that in globals.h).

It will now offer both.  (I don't think I agree with that "never"
part, because it is important, and exactly what the user wants, in
some use cases, although evidently not in yours.)

Re: xref-find-matches and stuff

[martin rudalics]

 >> .../globals.h
 >>    #define focus_follows_mouse
 >>      bool f_focus_follows_mouse;
 >>
 >> How am I supposed to proceed from here?
 >
 > It depends on what you mean by "proceed".  I understand that you want
 > to see the corresponding DEFVAR_* line, is that right?

Right.

 > Your wish has been granted in commit 8d69f38.

Thank you very much.

 >> That it goes to the definition of Vmake_pointer_invisible in frame.c
 >> (and never to that in globals.h).
 >
 > It will now offer both.  (I don't think I agree with that "never"
 > part, because it is important, and exactly what the user wants, in
 > some use cases, although evidently not in yours.)

OK with me (as long as the frame.c definition comes first).


I never worked with 'make TAGS' before.  Currently it's some sort of
pain to use for me, probably because of this

## FIXME? In out-of-tree builds, should TAGS be generated in srcdir?

(line 601 of Makefile.in).  Couldn't I direct it somehow to create the
TAGS file in the src directory?

Also, is there a way to tell it to not make emacs itself, that is, just
build the TAGS files and do nothing else?

OTOH what would my corresponding --regex argument to etags be now when
calling etags via shell-command?

martin

Re: xref-find-matches and stuff

[Eli Zaretskii]

> Date: Thu, 14 May 2015 12:13:19 +0200
> From: martin rudalics <rudalics@gmx.at>
> CC: pot@gnu.org, emacs-devel@gnu.org
> 
>  >> That it goes to the definition of Vmake_pointer_invisible in frame.c
>  >> (and never to that in globals.h).
>  >
>  > It will now offer both.  (I don't think I agree with that "never"
>  > part, because it is important, and exactly what the user wants, in
>  > some use cases, although evidently not in yours.)
> 
> OK with me (as long as the frame.c definition comes first).

AFAICS, the order is defined by the order of files submitted to etags
(i.e. random, from your POV).

> I never worked with 'make TAGS' before.  Currently it's some sort of
> pain to use for me, probably because of this
> 
> ## FIXME? In out-of-tree builds, should TAGS be generated in srcdir?
> 
> (line 601 of Makefile.in).  Couldn't I direct it somehow to create the
> TAGS file in the src directory?

I never build outside of the source tree.  Why can't you just move the
files back into the sources, or invoke the command from within the
source tree?

> Also, is there a way to tell it to not make emacs itself, that is, just
> build the TAGS files and do nothing else?

Does it still build Emacs if you invoke the command from src/?

> OTOH what would my corresponding --regex argument to etags be now when
> calling etags via shell-command?

With what shell?

Re: xref-find-matches and stuff

[martin rudalics]

 >> OK with me (as long as the frame.c definition comes first).
 >
 > AFAICS, the order is defined by the order of files submitted to etags
 > (i.e. random, from your POV).

You're right.  I was trying with an out-of-tree build where globals.h
does not get into my way because it's in the build's src directory ;-)

 >> I never worked with 'make TAGS' before.  Currently it's some sort of
 >> pain to use for me, probably because of this
 >>
 >> ## FIXME? In out-of-tree builds, should TAGS be generated in srcdir?
 >>
 >> (line 601 of Makefile.in).  Couldn't I direct it somehow to create the
 >> TAGS file in the src directory?
 >
 > I never build outside of the source tree.  Why can't you just move the
 > files back into the sources,

I already did that but it's a slight pain because I also have to remove
all those ../../src/ prefixes from the TAGS file.  But I obviously can
automatize that process if there's no better way.

 > or invoke the command from within the
 > source tree?

I'm not sure what you mean.  The source tree's src directory has no
Makefile so I get make: *** No rule to make target `TAGS'.  Stop.  If I
simply invoke make TAGS from its parent directory make TAGS builds Emacs
in-the-tree which is an even greater pain.

 >> Also, is there a way to tell it to not make emacs itself, that is, just
 >> build the TAGS files and do nothing else?
 >
 > Does it still build Emacs if you invoke the command from src/?

See above.

 >> OTOH what would my corresponding --regex argument to etags be now when
 >> calling etags via shell-command?
 >
 > With what shell?

On Windows cmd.exe.  Ever since I created my Emacs tags via

(defun my-shell-etags ()
   (interactive)
   (let ((etags (concat exec-directory "etags.exe *")))
     (let ((default-directory (concat default-directory "src/")))
       (shell-command etags))
     (let ((default-directory (concat default-directory "doc/lispref/")))
       (shell-command etags))
     (let ((default-directory (concat default-directory "doc/emacs/")))
       (shell-command etags))))

from the respective source directory.

martin

Re: xref-find-matches and stuff

[Eli Zaretskii]

> Date: Fri, 15 May 2015 18:44:47 +0200
> From: martin rudalics <rudalics@gmx.at>
> CC: pot@gnu.org, emacs-devel@gnu.org
> 
>  > or invoke the command from within the
>  > source tree?
> 
> I'm not sure what you mean.  The source tree's src directory has no
> Makefile so I get make: *** No rule to make target `TAGS'.  Stop.

You can configure the source tree, and only use that for "make TAGS".

>  >> OTOH what would my corresponding --regex argument to etags be now when
>  >> calling etags via shell-command?
>  >
>  > With what shell?
> 
> On Windows cmd.exe.

You will need to change the quoting style to use only "..", and escape
any embedded quotes.

Re: xref-find-matches and stuff

[Francesco Potortì]

>Francesco,
>
>while you're here: Could you pretty please make etags recognize
>DEFVAR_BOOL, DEFVAR_LISP, DEFVAR_KBOARD and friends?

The state machinery of Etags has problems with those, as is (tersely)
noted in etags.c.  That's why they are identified through --regex when
etags is called from the makefile in the Emacs sources.  So, if you
generate the tags file using make, there should be no problem.

Re: xref-find-matches and stuff

[Vitalie Spinu]

 >>> Dmitry Gutov on Fri, 8 May 2015 15:47:37 +0300 wrote:

 >> Yes. That can be. But I am setting xref-prompt-for-identifier /and
 >> recommend it widely ;)/. That implies a merge.

 > I don't understand. How does it imply that?

If xref-prompt-for-identifier is non-nil, do you propose to show only
the completions from the first back-end? If not, then you would need to
implement some kind of a merging strategy.

  Vitalie

Re: xref-find-matches and stuff

[Dmitry Gutov]

On 05/08/2015 07:44 PM, Vitalie Spinu wrote:

> If xref-prompt-for-identifier is non-nil, do you propose to show only
> the completions from the first back-end?

Yup. That's exactly what the "first one wins" strategy implies.

Re: xref-find-matches and stuff

[Vitalie Spinu]

 >>> Dmitry Gutov on Fri, 8 May 2015 19:47:46 +0300 wrote:

 > On 05/08/2015 07:44 PM, Vitalie Spinu wrote:
 >> If xref-prompt-for-identifier is non-nil, do you propose to show only
 >> the completions from the first back-end?

 > Yup. That's exactly what the "first one wins" strategy implies.

Ok. Then that's not something that I would be happy with. If I want a
symbol which is in the 2nd backed but nor in the first one, then there
is no way I can access that symbol.

I also don't think that backends that provide more symbols should be
given priority over backends which produce fewer but more targeted
candidates.

  Vitalie

Re: xref-find-matches and stuff

[Dmitry Gutov]

On 05/08/2015 09:42 PM, Vitalie Spinu wrote:

> Ok. Then that's not something that I would be happy with. If I want a
> symbol which is in the 2nd backed but nor in the first one, then there
> is no way I can access that symbol.

And my point is, the backends should be made well enough that you 
generally won't want to use a symbol from the 2nd backend that isn't in 
the first one. Ideally because there will be no such symbols.

I'm sure there can be situations when people might still want to merge 
backends, but imenu+etags+major mode, so far, doesn't make a convincing 
case.

> I also don't think that backends that provide more symbols should be
> given priority over backends which produce fewer but more targeted
> candidates.

We won't prioritize more vs fewer. Instead, the elements in 
xref-backends will be positioned in the order of priority. The first 
(thus, higher priority) backend that returns a non-nil completion table, 
will be used.

Re: xref-find-matches and stuff

[Vitalie Spinu]

 >>> Dmitry Gutov on Fri, 8 May 2015 22:42:35 +0300 wrote:


 > And my point is, the backends should be made well enough that you
 > generally won't want to use a symbol from the 2nd backend that isn't
 > in the first one. Ideally because there will be no such symbols.

I am afraid this is an impossible task.

 > but imenu+etags+major mode, so far, doesn't make a convincing case.

These tools live and work in different contexts. They are overlapping in
purpose but each does an unique job in situations in which other 2 are
not even defined. There is only one way to merge them into one "super
tool" - write one that does each of them in in their own application
context. The way I see it, that universal "super tool" is precisely
xref.

Anyways, I am summarizing my point in a final attempt to "make a case";)

  - IMENU operates on what I work, is always up-to date, works with
    non-programming modes (org, latex etc), non-visiting buffers
    (*scratch*) and any project-less file/language what I might open for
    whatever cursory reason. Stuff works out of the box - no tagging, no
    REPL is required.

  - REPLs are good, but work only on loaded files. Usually these cannot
    work with multiple projects or multi-language projects. Cross
    references support is lacking for many REPLs.

  - TAGs can support multiple projects but cannot do what Imenu can and
    will never be as smart as REPLs can be. No support for less common
    languages. Limited UI in emacs is a hindrance for common users (I
    cannot comment about ggtags though).

The above is also the order that I would like to have in my backend
list.

  Vitalie

Re: xref-find-matches and stuff

[Dmitry Gutov]

Sorry, we seem to be talking past each other. The bit about etags and 
unames (which makes having merge a part of the core behavior difficult) 
felt particularly relevant to me.

The question I'm trying to answer is what behavior would suit an average 
user best, and that doesn't involve manual management of backends, IMO. 
You can still implement a merging backend on top of it.

Using one backend at a time would mirror the mechanics of 
completion-at-point-function, which have been easy to comprehend for 
major mode authors thus far (just use `add-hook'). Asking a major mode 
to make a decision whether it will augment their results with etags, 
seems improper. But else to do that, if the value is a list?

I can enumerate use cases where the lack of availability of grouped 
elements in completion-at-point-functions was a real problem (basically, 
yasnippet, which isn't really completion, and file path completion, 
which has peculiarities that should ideally be tweaked on a 
per-language-and-project basis; both of these would be nice to have, 
though). I'm sure there will be corresponding cases with xref, but we 
haven't touched on those.

On 05/09/2015 12:52 AM, Vitalie Spinu wrote:

> I am afraid this is an impossible task.

I think we can get close enough. At least as far as most users are 
concerned; and offer additional customization for the rest.

> These tools live and work in different contexts. They are overlapping in
> purpose but each does an unique job in situations in which other 2 are
> not even defined. There is only one way to merge them into one "super
> tool" - write one that does each of them in in their own application
> context.

That description fits the "first one wins" approach as well. Even more 
so since you wrote "different contexts". An argument against would be 
stating that there are contexts where we really can't pick which tool to 
use. Contexts in which they all return equally useful, but different, 
results together.

> Anyways, I am summarizing my point in a final attempt to "make a case";)
>
>    - IMENU operates on what I work, is always up-to date, works with
>      non-programming modes (org, latex etc), non-visiting buffers
>      (*scratch*) and any project-less file/language what I might open for
>      whatever cursory reason. Stuff works out of the box - no tagging, no
>      REPL is required.

The quality of Imenu indexing is usually pretty low (lots of duplicates 
due to methods with the same name), so I'd put it on the last position: 
use only when we can't find a better alternative.

>    - REPLs are good, but work only on loaded files. Usually these cannot
>      work with multiple projects or multi-language projects. Cross
>      references support is lacking for many REPLs.

Multi-language projects are an interesting example. But if there are no 
references between the source file in different languages (or if there 
are, how would you handle them?), this case devolves into the situation 
of having several projects, semantically related.

Personally, I'd just make a "switch to another project" a separate step 
here.

>    - TAGs can support multiple projects but cannot do what Imenu can and
>      will never be as smart as REPLs can be. No support for less common
>      languages. Limited UI in emacs is a hindrance for common users (I
>      cannot comment about ggtags though).

It's not ideal, no argument here. However, being unavailable in buffers 
without TAGS, or in some languages, means it can be safely skipped in 
those circumstances. No need to merge.

> The above is also the order that I would like to have in my backend
> list.

I'm pretty sure Imenu can't be the default backend to use. And you were 
proposing to show only the results from the first backend in the list, 
right? Until the user asks for the rest?

Re: xref-find-matches and stuff

[Vitalie Spinu]

 >>> Dmitry Gutov on Sat, 9 May 2015 03:09:00 +0300 wrote:


 >and that doesn't involve manual management of backends, IMO.

I also hope not.

 > Using one backend at a time would mirror the mechanics of
 > completion-at-point-function,

It works for completion because you can reliably disambiguate from
context which completion to use. C in C, Python in py, files in strings
with "/" in them etc. With xref context is often unrelated to where you
want to jump and, thus, is orthogonal to the selection of the "best"
backend.

 > major mode authors thus far (just use `add-hook'). Asking a major
 > mode to make a decision whether it will augment their results with
 > etags, seems improper. 

No. Major mode just defines one backend. The end user defines a list of
backends to be used simultaneously.

 > The quality of Imenu indexing is usually pretty low

True, but it does a good job for me.

 > Multi-language projects are an interesting example. But if there are
 > no references between the source file in different languages

Yet another argument for prompting on M-. by default ;)

 > Personally, I'd just make a "switch to another project" a separate
 > step here.

That's the same as picking a backend just before the jump. Could be, but
I hope for a better UI.

 > It's not ideal, no argument here. However, being unavailable in
 > buffers without TAGS, or in some languages, means it can be safely
 > skipped in those circumstances.

No, it cannot. That's the main point. If I am in a non-tagged file I
still want to be able to jump to my tags.

 >> The above is also the order that I would like to have in my backend
 >> list.

 > I'm pretty sure Imenu can't be the default backend to use. And you were
 > proposing to show only the results from the first backend in the list, right?

As an non-ideal alternative, yes. I might be quite happy with being able
to rotate backends during the minibuffer completion.


  Vitalie

Re: xref-find-matches and stuff

[Dmitry Gutov]

On 05/09/2015 03:58 PM, Vitalie Spinu wrote:

> It works for completion because you can reliably disambiguate from
> context which completion to use. C in C, Python in py, files in strings
> with "/" in them etc.

It's not always so simple:

- In some contexts, file paths can appear in the buffer outside of 
string literals. In sh-mode and eshell-mode, for instance, that is the 
rule rather than exception.

- Depending on the context, file paths may have different quoting rules. 
See `completion-table-with-quoting' and its uses for more context (I 
don't understand that area well myself).

Point is, while it's possible to provide the basic functionality without 
specializing, a perfect file completion is language and 
environment-dependent. So it's hard to serve with just one completion 
function.

 > With xref context is often unrelated to where you
> want to jump and, thus, is orthogonal to the selection of the "best"
> backend.

Not that often. 95% of the time, I either jump to the thing at point (in 
the current project), or to a thing at least somewhat related to it: a 
caller, a callee, or simply a laterally related function in the same 
project or in one of its dependencies or dependants. The current project 
and its dependencies are usually possible to determine automatically.

> No. Major mode just defines one backend. The end user defines a list of
> backends to be used simultaneously.

Would the user even know which backends are possible to use, if the 
major mode doesn't add it to `xref-backends'?

By the way, the above is an example of "manual management of backends", 
which we've ostensibly agreed is undesirable. It may be good for a power 
user, but let's try to make things "just work" more often, for the 
average user.

>   > Multi-language projects are an interesting example. But if there are
>   > no references between the source file in different languages
>
> Yet another argument for prompting on M-. by default ;)

And prompt for the project to jump to, every time?

>   > Personally, I'd just make a "switch to another project" a separate
>   > step here.
>
> That's the same as picking a backend just before the jump. Could be, but
> I hope for a better UI.

Selecting from a list of project might be a more transparent choice for 
a user, rather than picking from a list of backends.

>   > It's not ideal, no argument here. However, being unavailable in
>   > buffers without TAGS, or in some languages, means it can be safely
>   > skipped in those circumstances.
>
> No, it cannot. That's the main point. If I am in a non-tagged file I
> still want to be able to jump to my tags.

This use case still could be served by an extra "switch to another 
project" step, I think.

> As an non-ideal alternative, yes. I might be quite happy with being able
> to rotate backends during the minibuffer completion.

To have this kind of capability could be fun, but I think the current 
completing-read interface is ill-suited for this. IOW, you'd have to 
start with writing a new completing-read-function (Helm might be a 
somewhat viable alternative, since it can already use several sources at 
a time, and displays the name of each in its buffer).

Re: xref-find-matches and stuff

[Eli Zaretskii]

> From: Vitalie Spinu <spinuvit@gmail.com>
> Date: Fri, 08 May 2015 23:52:30 +0200
> Cc: Helmut Eller <eller.helmut@gmail.com>, emacs-devel <emacs-devel@gnu.org>
> 
>  >>> Dmitry Gutov on Fri, 8 May 2015 22:42:35 +0300 wrote:
> 
> 
>  > And my point is, the backends should be made well enough that you
>  > generally won't want to use a symbol from the 2nd backend that isn't
>  > in the first one. Ideally because there will be no such symbols.
> 
> I am afraid this is an impossible task.

Can you tell why?

> Anyways, I am summarizing my point in a final attempt to "make a case";)
> 
>   - IMENU operates on what I work, is always up-to date, works with
>     non-programming modes (org, latex etc), non-visiting buffers
>     (*scratch*) and any project-less file/language what I might open for
>     whatever cursory reason. Stuff works out of the box - no tagging, no
>     REPL is required.
> 
>   - REPLs are good, but work only on loaded files. Usually these cannot
>     work with multiple projects or multi-language projects. Cross
>     references support is lacking for many REPLs.
> 
>   - TAGs can support multiple projects but cannot do what Imenu can and
>     will never be as smart as REPLs can be. No support for less common
>     languages. Limited UI in emacs is a hindrance for common users (I
>     cannot comment about ggtags though).

To me, this means we need to have a single back-end for each
situation, and ideally also a mechanism to choose that single back-end
automatically.  I see no reasons to conclude that we need more than
one in each use case.

Re: xref-find-matches and stuff

[Vitalie Spinu]

[-- Attachment #1: Type: text/plain, Size: 2124 bytes --]

 >>> Eli Zaretskii on Sat, 09 May 2015 11:05:31 +0300 wrote:

 >> > And my point is, the backends should be made well enough that you
 >> > generally won't want to use a symbol from the 2nd backend that isn't
 >> > in the first one. Ideally because there will be no such symbols.
 >> 
 >> I am afraid this is an impossible task.

 > Can you tell why?

In an ideal world you can imagine a tag-like backend that would know all
possible language and would be aware of all the intricacies of multiple
projects that you are working on simultaneously. It would also sniff
through all open files and tag them automatically.

Then, indeed, you won't need Imenu, nor REPL backends. Till such an
universal tool is there, xref itself comes very close to it if imenu,
tags and REPL backends were automatically merged.


 >> - IMENU operates on what I work, is always up-to date, works with
 >> non-programming modes (org, latex etc), non-visiting buffers
 >> (*scratch*) and any project-less file/language what I might open for
 >> whatever cursory reason. Stuff works out of the box - no tagging, no
 >> REPL is required.
 >> 
 >> - REPLs are good, but work only on loaded files. Usually these cannot
 >> work with multiple projects or multi-language projects. Cross
 >> references support is lacking for many REPLs.
 >> 
 >> - TAGs can support multiple projects but cannot do what Imenu can and
 >> will never be as smart as REPLs can be. No support for less common
 >> languages. Limited UI in emacs is a hindrance for common users (I
 >> cannot comment about ggtags though).

 > To me, this means we need to have a single back-end for each
 > situation, and ideally also a mechanism to choose that single
 > back-end automatically.  I see no reasons to conclude that we need
 > more than one in each use case.

Assume a high level language project with no good support for tags (R
for example's sake) which uses C++. REPL backends can give you
references only for loaded code in REPL, tags give you references only
for C++. Imenu gives you references for all open C++ and R files. Your
task is to jump from A to B (plot attached).


[-- Attachment #2: xref.png --]
[-- Type: image/png, Size: 6111 bytes --]

[-- Attachment #3: Type: text/plain, Size: 1382 bytes --]


You and other people seem to assume that given the context of A you can
infer which backend to use in order to jump to B. Unless you you have an
unique symbol at point in A matching B, that's impossible.

If I am in an C++ tutorial which is not tagged (but has Imenu). Assume
no symbol at point. How can xref know if I want to jump to a symbol in
this example file (imenu backend) or a C++ file in my current project
(tags backend)?

Dmitry goes as far as to suggest that even on C-u M-. only one backend
should be completed upon. I just don't see that working at all. With
multi-language projects it's common to have a high level wrapper (foo)
for low level (C_foo) and you would want to jump back and forth from
`foo` to `C_foo`. `C_foo` is probably unique, `foo` is not, and you just
cannot disambiguate the backend only by current context. In R projects I
can even have two different objects with the same name, one in C and one
in R.


There are basically three ways out

  1) merge backends
  2) select the backend before jump
  3) assign each backend to a separate key

IMO, the 3rd is better than the 2nd and I guess it's the current
situation for loads of people.

I have C-. for elisp-xref, M-. for whatever is the default, M-o for
imenu-anywhere and I am seriously thinking about the forth key for etags
because inconsistency with M-. started jarring on me.


  Vitalie

Re: xref-find-matches and stuff

[Eli Zaretskii]

> From: Vitalie Spinu <spinuvit@gmail.com>
> Cc: dgutov@yandex.ru,  eller.helmut@gmail.com,  emacs-devel@gnu.org
> Date: Sat, 09 May 2015 14:32:03 +0200
> 
>  >> > And my point is, the backends should be made well enough that you
>  >> > generally won't want to use a symbol from the 2nd backend that isn't
>  >> > in the first one. Ideally because there will be no such symbols.
>  >> 
>  >> I am afraid this is an impossible task.
> 
>  > Can you tell why?
> 
> In an ideal world you can imagine a tag-like backend that would know all
> possible language and would be aware of all the intricacies of multiple
> projects that you are working on simultaneously. It would also sniff
> through all open files and tag them automatically.
> 
> Then, indeed, you won't need Imenu, nor REPL backends. Till such an
> universal tool is there, xref itself comes very close to it if imenu,
> tags and REPL backends were automatically merged.

If by "merged" you mean a union of all the symbols returned by all the
back-ends, then you have just described a single back-end that will
satisfy our needs.

>  > To me, this means we need to have a single back-end for each
>  > situation, and ideally also a mechanism to choose that single
>  > back-end automatically.  I see no reasons to conclude that we need
>  > more than one in each use case.
> 
> Assume a high level language project with no good support for tags (R
> for example's sake) which uses C++. REPL backends can give you
> references only for loaded code in REPL, tags give you references only
> for C++. Imenu gives you references for all open C++ and R files. Your
> task is to jump from A to B (plot attached).

See above.

> You and other people seem to assume that given the context of A you can
> infer which backend to use in order to jump to B. Unless you you have an
> unique symbol at point in A matching B, that's impossible.

I'm sorry, but you didn't say what are A and B here, and what is the
context of A.

> If I am in an C++ tutorial which is not tagged (but has Imenu). Assume
> no symbol at point. How can xref know if I want to jump to a symbol in
> this example file (imenu backend) or a C++ file in my current project
> (tags backend)?

Who or what _can_ know that, except the user herself?  And the user
then has to set up this session so that the _single_ relevant back-end
is to be used.

And anyway, are the majority of the projects we work on tutorials?  I
sincerely doubt that.

> Dmitry goes as far as to suggest that even on C-u M-. only one backend
> should be completed upon. I just don't see that working at all. With
> multi-language projects it's common to have a high level wrapper (foo)
> for low level (C_foo) and you would want to jump back and forth from
> `foo` to `C_foo`. `C_foo` is probably unique, `foo` is not, and you just
> cannot disambiguate the backend only by current context. In R projects I
> can even have two different objects with the same name, one in C and one
> in R.

The difficulty of making the decision which back-end to use does not
imply that we should avoid making the decision.  It just implies we
need better methods of making that decision.  Avoiding the decision
makes no sense, it will just bring more chaos and confusion.

>   1) merge backends
>   2) select the backend before jump
>   3) assign each backend to a separate key
> 
> IMO, the 3rd is better than the 2nd and I guess it's the current
> situation for loads of people.
> 
> I have C-. for elisp-xref, M-. for whatever is the default, M-o for
> imenu-anywhere and I am seriously thinking about the forth key for etags
> because inconsistency with M-. started jarring on me.

There's a saying that a person with one watch always knows the time,
even if her idea of the time is a bit biased; whereas a person with
more than one watch always confused wrt what time it is.

Re: xref-find-matches and stuff

[Vitalie Spinu]

 >>> Eli Zaretskii on Sat, 09 May 2015 16:19:36 +0300 wrote:
 >> 
 >> Then, indeed, you won't need Imenu, nor REPL backends. Till such an
 >> universal tool is there, xref itself comes very close to it if imenu,
 >> tags and REPL backends were automatically merged.

 > If by "merged" you mean a union of all the symbols returned by all
 > the back-ends, then you have just described a single back-end that
 > will satisfy our needs.

That's exactly my point. Thank you!

Full union is a bit too much, you can easily do better than that.

 >> You and other people seem to assume that given the context of A you
 >> can infer which backend to use in order to jump to B. Unless you you
 >> have an unique symbol at point in A matching B, that's impossible.

 > I'm sorry, but you didn't say what are A and B here, and what is the
 > context of A.

That was in the picture. I hope you received it.

The exact context at point of A is irrelevant (it can be C++ or R) the
only thing relevant is that to A you can jump only with Imenu backend to
B only with TAGS backend.

 >> If I am in an C++ tutorial which is not tagged (but has Imenu). Assume
 >> no symbol at point. How can xref know if I want to jump to a symbol in
 >> this example file (imenu backend) or a C++ file in my current project
 >> (tags backend)?

 > Who or what _can_ know that, except the user herself?  

Exactly my point!

 > And the user then has to set up this session so that the _single_
 > relevant back-end is to be used.

??? In my example backends are non-overlapping. You need all of them
simultaneously.

 > And anyway, are the majority of the projects we work on tutorials?  I
 > sincerely doubt that.

Common. That's just an example. Don't you create scratch buffers to try
stuff? Don't you open other projects/languages to see how stuff is
implemented when developing your primary stuff?

 > The difficulty of making the decision which back-end to use does not
 > imply that we should avoid making the decision.  

You can leave the decision to the users but that doesn't mean it will
get any easier. Either you can try to solve it for all, or hundreds of
thousands will be solving for themselves.

 > It just implies we need better methods of making that decision.

You seem to be still convinced that disambiguation the backend from the
current context is a manageable task.

 >> I have C-. for elisp-xref, M-. for whatever is the default, M-o for
 >> imenu-anywhere and I am seriously thinking about the forth key for etags
 >> because inconsistency with M-. started jarring on me.

 > There's a saying that a person with one watch always knows the time,
 > even if her idea of the time is a bit biased; whereas a person with
 > more than one watch always confused wrt what time it is.

I think of it more like watch + scale + ruler ...


  Vitalie

Re: xref-find-matches and stuff

[Eli Zaretskii]

> From: Vitalie Spinu <spinuvit@gmail.com>
> Cc: dgutov@yandex.ru,  eller.helmut@gmail.com,  emacs-devel@gnu.org
> Date: Sat, 09 May 2015 15:46:00 +0200
> 
>  >>> Eli Zaretskii on Sat, 09 May 2015 16:19:36 +0300 wrote:
>  >> 
>  >> Then, indeed, you won't need Imenu, nor REPL backends. Till such an
>  >> universal tool is there, xref itself comes very close to it if imenu,
>  >> tags and REPL backends were automatically merged.
> 
>  > If by "merged" you mean a union of all the symbols returned by all
>  > the back-ends, then you have just described a single back-end that
>  > will satisfy our needs.
> 
> That's exactly my point. Thank you!
> 
> Full union is a bit too much, you can easily do better than that.

I don't see why bother.

>  >> can infer which backend to use in order to jump to B. Unless you you
>  >> have an unique symbol at point in A matching B, that's impossible.
> 
>  > I'm sorry, but you didn't say what are A and B here, and what is the
>  > context of A.
> 
> That was in the picture. I hope you received it.

I received the picture, but all it shows is a few intersecting
rectangles with labels.  It doesn't explain what is A or B.

> The exact context at point of A is irrelevant (it can be C++ or R) the
> only thing relevant is that to A you can jump only with Imenu backend to
> B only with TAGS backend.

The notion of "context" is still not clear, sorry.

Also, I don't understand why would TAGS not know about A.  It doesn't
miss definitions, IME.

>  > And the user then has to set up this session so that the _single_
>  > relevant back-end is to be used.
> 
> ??? In my example backends are non-overlapping. You need all of them
> simultaneously.

No, I need only one in each particular use case.

>  > And anyway, are the majority of the projects we work on tutorials?  I
>  > sincerely doubt that.
> 
> Common. That's just an example. Don't you create scratch buffers to try
> stuff? Don't you open other projects/languages to see how stuff is
> implemented when developing your primary stuff?

Not every open file or buffer needs this facility.  In a buffer you
can simply search.

>  > The difficulty of making the decision which back-end to use does not
>  > imply that we should avoid making the decision.  
> 
> You can leave the decision to the users but that doesn't mean it will
> get any easier. Either you can try to solve it for all, or hundreds of
> thousands will be solving for themselves.
> 
>  > It just implies we need better methods of making that decision.
> 
> You seem to be still convinced that disambiguation the backend from the
> current context is a manageable task.

Of course it's manageable!

>  >> I have C-. for elisp-xref, M-. for whatever is the default, M-o for
>  >> imenu-anywhere and I am seriously thinking about the forth key for etags
>  >> because inconsistency with M-. started jarring on me.
> 
>  > There's a saying that a person with one watch always knows the time,
>  > even if her idea of the time is a bit biased; whereas a person with
>  > more than one watch always confused wrt what time it is.
> 
> I think of it more like watch + scale + ruler ...

Scale and rules are worthless when I need to know the time.

Re: xref-find-matches and stuff

[Vitalie Spinu]

 >>> Eli Zaretskii on Sat, 09 May 2015 17:03:23 +0300 wrote:

 > rectangles with labels.  It doesn't explain what is A or B.

If rectangle is a file, then A and B are physical positions in that
file.

 >> The exact context at point of A is irrelevant (it can be C++ or R) the
 >> only thing relevant is that to A you can jump only with Imenu backend to
 >> B only with TAGS backend.

 > The notion of "context" is still not clear, sorry.

Anything what you can infer from around your current location is
context. Symbol at point is context, current major mode is context,
whether the buffer is visiting a file is context.

The xref initiative is essentially to make a supernatural M-. that would
be able to infer where to go (B) from the context around current point
(A). People seem to think that xref can achieve that by choosing the
backend first and then rely on that backend to give you B. But choosing
the backend based on the context is a meaningless task.

 > Also, I don't understand why would TAGS not know about A.  It doesn't
 > miss definitions, IME.

REPLs knows about dynamically created objects about which TAGS have no
idea.

Go to *scratch* buffer and write (defvar foo 34). Imenu now knows about
'foo', TAGS don't. Each time you open an untagged file (related project,
example etc), imenu will know about symbols in it, TAGS won't.

 >> > There's a saying that a person with one watch always knows the time,
 >> > even if her idea of the time is a bit biased; whereas a person with
 >> > more than one watch always confused wrt what time it is.
 >> 
 >> I think of it more like watch + scale + ruler ...

 > Scale and rules are worthless when I need to know the time.

If all you have is a watch, you cannot measure your weight.

Re: xref-find-matches and stuff

[Eli Zaretskii]

> From: Vitalie Spinu <spinuvit@gmail.com>
> Cc: dgutov@yandex.ru,  eller.helmut@gmail.com,  emacs-devel@gnu.org
> Date: Sat, 09 May 2015 20:23:49 +0200
> 
>  >>> Eli Zaretskii on Sat, 09 May 2015 17:03:23 +0300 wrote:
> 
>  > rectangles with labels.  It doesn't explain what is A or B.
> 
> If rectangle is a file, then A and B are physical positions in that
> file.

Files don't intersect; your rectangles did.  So I still don't
understand what that picture was supposed to illustrate.

>  >> The exact context at point of A is irrelevant (it can be C++ or R) the
>  >> only thing relevant is that to A you can jump only with Imenu backend to
>  >> B only with TAGS backend.
> 
>  > The notion of "context" is still not clear, sorry.
> 
> Anything what you can infer from around your current location is
> context. Symbol at point is context, current major mode is context,
> whether the buffer is visiting a file is context.

OK, but how's that relevant to the issue at hand?

> The xref initiative is essentially to make a supernatural M-. that would
> be able to infer where to go (B) from the context around current point
> (A). People seem to think that xref can achieve that by choosing the
> backend first and then rely on that backend to give you B. But choosing
> the backend based on the context is a meaningless task.

Who said a back-end is selected solely based on context?

>  > Also, I don't understand why would TAGS not know about A.  It doesn't
>  > miss definitions, IME.
> 
> REPLs knows about dynamically created objects about which TAGS have no
> idea.

Why would I need to "jump" to such an object?  I've just created it,
so it's right here, where I did that.

xref is for objects you don't see, because you don't know where to
look.

> Go to *scratch* buffer and write (defvar foo 34). Imenu now knows about
> 'foo', TAGS don't. Each time you open an untagged file (related project,
> example etc), imenu will know about symbols in it, TAGS won't.

I already know about foo, so I need no help in finding it.

>  >> > There's a saying that a person with one watch always knows the time,
>  >> > even if her idea of the time is a bit biased; whereas a person with
>  >> > more than one watch always confused wrt what time it is.
>  >> 
>  >> I think of it more like watch + scale + ruler ...
> 
>  > Scale and rules are worthless when I need to know the time.
> 
> If all you have is a watch, you cannot measure your weight.

I don't need to, I just need to know the time.

Re: xref-find-matches and stuff

[Vitalie Spinu]

 >>> Eli Zaretskii on Sat, 09 May 2015 21:49:07 +0300 wrote:

 > Files don't intersect; your rectangles did.  So I still don't
 > understand what that picture was supposed to illustrate.

Let me try it again. A rectangle is a set of all places where each
backend collects its locations. For Imenu set of all open buffers. For
tags all files tagged. For repls all loaded files. Would you be happy
with this definition?

 >> Anything what you can infer from around your current location is
 >> context. Symbol at point is context, current major mode is context,
 >> whether the buffer is visiting a file is context.

 > OK, but how's that relevant to the issue at hand?

You asked what context means. The relevant part is that you cannot infer
backend on the context.

 >> The xref initiative is essentially to make a supernatural M-. that would
 >> be able to infer where to go (B) from the context around current point
 >> (A). People seem to think that xref can achieve that by choosing the
 >> backend first and then rely on that backend to give you B. But choosing
 >> the backend based on the context is a meaningless task.

 > Who said a back-end is selected solely based on context?

People want M-. to operate without additional user's input, if that's
what you mean.

 >> > Also, I don't understand why would TAGS not know about A.  It doesn't
 >> > miss definitions, IME.
 >> 
 >> REPLs knows about dynamically created objects about which TAGS have no
 >> idea.

 > Why would I need to "jump" to such an object?  I've just created it,
 > so it's right here, where I did that.

Jump to a tag and them jump to "foo". Of course you want to jump to
"foo", it's the most jumpable symbol around.

Re: xref-find-matches and stuff

[Eli Zaretskii]

> From: Vitalie Spinu <spinuvit@gmail.com>
> Date: Fri, 08 May 2015 20:42:59 +0200
> Cc: Helmut Eller <eller.helmut@gmail.com>, emacs-devel <emacs-devel@gnu.org>
> 
> If I want a symbol which is in the 2nd backed but nor in the first
> one, then there is no way I can access that symbol.
> 
> I also don't think that backends that provide more symbols should be
> given priority over backends which produce fewer but more targeted
> candidates.

Back-ends that mis symbols make no sense, IMO.  They should be fixed
not to miss them.

Re: xref-find-matches and stuff

[Helmut Eller]

On Sat, May 09 2015, Eli Zaretskii wrote:

> Back-ends that mis symbols make no sense, IMO.  They should be fixed
> not to miss them.

There are many situations where it's desirable to show fewer than more
symbols; irrelevant symbols should be filtered out.  In other words:
most people prefer high Precision[*] over high Recall[*].  Ideally we
would like to have both high Precision and high Recall, but in practice
a trade-off is needed.  E.g. the etags backend has relatively high
Recall but very low Precision.  IMO a bad trade-off compared to the much
better Precision of the elisp/find-func backend.

Also I think that the generic/naive merging of backends as proposed so
far will (always?) increase Recall but usually decrease Precision.

Helmut

[*]http://en.wikipedia.org/wiki/Precision_recall

Re: xref-find-matches and stuff

[Eli Zaretskii]

> From: Helmut Eller <eller.helmut@gmail.com>
> Cc: Vitalie Spinu <spinuvit@gmail.com>,  dgutov@yandex.ru,  emacs-devel@gnu.org
> Date: Sat, 09 May 2015 11:06:45 +0200
> 
> On Sat, May 09 2015, Eli Zaretskii wrote:
> 
> > Back-ends that mis symbols make no sense, IMO.  They should be fixed
> > not to miss them.
> 
> There are many situations where it's desirable to show fewer than more
> symbols; irrelevant symbols should be filtered out.  In other words:
> most people prefer high Precision[*] over high Recall[*].

Yes, but it's UI's task to tell the back-end which of these to prefer.
A back-end that misses symbols without being told so should be fixed.

It's possible that preferring precision over recall should be the
default, but that's not to mean the back-end cannot honor the opposite
preference when told so.

It also makes no sense to have back-ends with different defaults in
this respect; that will produce the same confusing user experience as
I reported.

> Ideally we would like to have both high Precision and high Recall,
> but in practice a trade-off is needed.

The trade off is up to the user and/or UI, not up to the back-end.

> E.g. the etags backend has relatively high Recall but very low
> Precision.

I thought I already debunked this fallacy.  The etags back-end
provides a very fine-grained control over its preferences, it's just
that we didn't use that feature, and instead always told it to return
as many symbols as it could.  It's small wonder you regard the results
as "very low-precision".

> IMO a bad trade-off compared to the much better Precision of the
> elisp/find-func backend.

This trade-off is the blame of the UI: it didn't instruct etags to be
more selective.

> Also I think that the generic/naive merging of backends as proposed so
> far will (always?) increase Recall but usually decrease Precision.

If the user wants that, it's their (users') funeral; we should honor
that.

Re: xref-find-matches and stuff

[Helmut Eller]

On Sat, May 09 2015, Eli Zaretskii wrote:

>> > Back-ends that mis symbols make no sense, IMO.  They should be fixed
>> > not to miss them.
>> 
>> There are many situations where it's desirable to show fewer than more
>> symbols; irrelevant symbols should be filtered out.  In other words:
>> most people prefer high Precision[*] over high Recall[*].
> Yes, but it's UI's task to tell the back-end which of these to prefer.

A language specific backend is in a much better position to decide what
is relevant than the language agnostic UI.

> A back-end that misses symbols without being told so should be fixed.

There may be many good reasons why a backend may decide to "miss"
symbols.  Ignoring code that's not loaded into Emacs is a very good
strategy most of the time and doesn't need to be "fixed".

> It's possible that preferring precision over recall should be the
> default, but that's not to mean the back-end cannot honor the opposite
> preference when told so.
>
> It also makes no sense to have back-ends with different defaults in
> this respect; that will produce the same confusing user experience as
> I reported.

I see no reason why different backends shouldn't have different
precision/recall trade-offs.

[...]
>> E.g. the etags backend has relatively high Recall but very low
>> Precision.
>
> I thought I already debunked this fallacy.  The etags back-end
> provides a very fine-grained control over its preferences, it's just
> that we didn't use that feature, and instead always told it to return
> as many symbols as it could.  It's small wonder you regard the results
> as "very low-precision".

Let's use the src/TAGS file and then do C-u M-. Lisp_Object RET in
src/alloc.c.  That returns 1000 results.  Only the 2 typedefs in lisp.h
are relevant; everything else is irrelevant.

Recall: 2/2 = 100%
Precision: 2/1000 = 0.2%

Yes indeed, I consider 0.2% as "very low-precision".
Feel free to disagree.

Helmut

Re: xref-find-matches and stuff

[Eli Zaretskii]

> From: Helmut Eller <eller.helmut@gmail.com>
> Cc: emacs-devel@gnu.org
> Date: Sat, 09 May 2015 13:43:24 +0200
> 
> On Sat, May 09 2015, Eli Zaretskii wrote:
> 
> >> > Back-ends that mis symbols make no sense, IMO.  They should be fixed
> >> > not to miss them.
> >> 
> >> There are many situations where it's desirable to show fewer than more
> >> symbols; irrelevant symbols should be filtered out.  In other words:
> >> most people prefer high Precision[*] over high Recall[*].
> > Yes, but it's UI's task to tell the back-end which of these to prefer.
> 
> A language specific backend is in a much better position to decide what
> is relevant than the language agnostic UI.

How would the back-end be able to know what the user-level feature
wants?  It can't.

The API for the invocation of the back-end must specify that in terms
that are abstractions of language-specific features.  Any language has
variables and functions, some have classes and macros, etc.  I see no
problems here.

> > A back-end that misses symbols without being told so should be fixed.
> 
> There may be many good reasons why a backend may decide to "miss"
> symbols.

A back-end should not decide anything, it should perform the job as
instructed.  Otherwise, it's not a back-end, but a large part of
application-level knowledge.

> Ignoring code that's not loaded into Emacs is a very good
> strategy most of the time and doesn't need to be "fixed".

Then tell the back-end when to ignore it and when not to ignore.  It's
not up to the back-end to decide.

> > It's possible that preferring precision over recall should be the
> > default, but that's not to mean the back-end cannot honor the opposite
> > preference when told so.
> >
> > It also makes no sense to have back-ends with different defaults in
> > this respect; that will produce the same confusing user experience as
> > I reported.
> 
> I see no reason why different backends shouldn't have different
> precision/recall trade-offs.

Sigh...  I described what that does in terms of user experience.

> >> E.g. the etags backend has relatively high Recall but very low
> >> Precision.
> >
> > I thought I already debunked this fallacy.  The etags back-end
> > provides a very fine-grained control over its preferences, it's just
> > that we didn't use that feature, and instead always told it to return
> > as many symbols as it could.  It's small wonder you regard the results
> > as "very low-precision".
> 
> Let's use the src/TAGS file and then do C-u M-. Lisp_Object RET in
> src/alloc.c.  That returns 1000 results.  Only the 2 typedefs in lisp.h
> are relevant; everything else is irrelevant.
> 
> Recall: 2/2 = 100%
> Precision: 2/1000 = 0.2%
> 
> Yes indeed, I consider 0.2% as "very low-precision".
> Feel free to disagree.

Feel free to actually look at the code which does that.  There, you
will see that this is what we ask etags to do.

Then try your example again, after this small change:

  (setq etags-xref-find-definitions-tag-order
        '(tag-exact-match-p tag-implicit-name-match-p))

Feel free to disagree with yourself now.

Re: xref-find-matches and stuff

[Dmitry Gutov]

On 05/09/2015 03:20 PM, Eli Zaretskii wrote:

>    (setq etags-xref-find-definitions-tag-order
>          '(tag-exact-match-p tag-implicit-name-match-p))

Any objections to making this the default value?

Re: xref-find-matches and stuff

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sat, 9 May 2015 15:31:20 +0300
> 
> On 05/09/2015 03:20 PM, Eli Zaretskii wrote:
> 
> >    (setq etags-xref-find-definitions-tag-order
> >          '(tag-exact-match-p tag-implicit-name-match-p))
> 
> Any objections to making this the default value?

None here.

Re: xref-find-matches and stuff

[Dmitry Gutov]

On 05/09/2015 03:20 PM, Eli Zaretskii wrote:

> Then try your example again, after this small change:
>
>    (setq etags-xref-find-definitions-tag-order
>          '(tag-exact-match-p tag-implicit-name-match-p))
>
> Feel free to disagree with yourself now.

By the way, it still gives a false positive:

1473:     word_size = sizeof (Lisp_Object)

Re: xref-find-matches and stuff

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sat, 9 May 2015 15:58:44 +0300
> 
> On 05/09/2015 03:20 PM, Eli Zaretskii wrote:
> 
> > Then try your example again, after this small change:
> >
> >    (setq etags-xref-find-definitions-tag-order
> >          '(tag-exact-match-p tag-implicit-name-match-p))
> >
> > Feel free to disagree with yourself now.
> 
> By the way, it still gives a false positive:
> 
> 1473:     word_size = sizeof (Lisp_Object)

C is a hard language to parse with ad-hoc parsers.  If you know of a
better tool that doesn't get confused by the source at that point of
lisp.h (did you look at it?), name it.

Re: xref-find-matches and stuff

[Dmitry Gutov]

On 05/09/2015 04:22 PM, Eli Zaretskii wrote:

> C is a hard language to parse with ad-hoc parsers.  If you know of a
> better tool that doesn't get confused by the source at that point of
> lisp.h (did you look at it?), name it.

GNU Global doesn't, at least not in this example:

~/v/e/src> global -x Lisp_Object
Lisp_Object 567 lisp.h typedef struct { EMACS_INT i; } Lisp_Object;
Lisp_Object 577 lisp.h typedef EMACS_INT Lisp_Object;

Re: xref-find-matches and stuff

[Eli Zaretskii]

> Cc: eller.helmut@gmail.com, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sun, 10 May 2015 00:56:36 +0300
> 
> On 05/09/2015 04:22 PM, Eli Zaretskii wrote:
> 
> > C is a hard language to parse with ad-hoc parsers.  If you know of a
> > better tool that doesn't get confused by the source at that point of
> > lisp.h (did you look at it?), name it.
> 
> GNU Global doesn't, at least not in this example:
> 
> ~/v/e/src> global -x Lisp_Object
> Lisp_Object 567 lisp.h typedef struct { EMACS_INT i; } Lisp_Object;
> Lisp_Object 577 lisp.h typedef EMACS_INT Lisp_Object;

With the patch below, neither will etags.

Francesco, do you see any problems with it?

--- lib-src/etags.c~0	2015-03-29 08:32:03 +0300
+++ lib-src/etags.c	2015-05-10 12:15:23 +0300
@@ -2862,7 +2862,10 @@ consider_token (char *str, int len, int
      case st_none:
        if (constantypedefs
 	   && structdef == snone
-	   && structtype == st_C_enum && bracelev > structbracelev)
+	   && structtype == st_C_enum && bracelev > structbracelev
+	   /* Don't tag tokens in expressions that assign values to enum
+	      constants.  */
+	   && fvdef != vignore)
 	 return true;		/* enum constant */
        switch (fvdef)
 	 {
@@ -3176,7 +3179,19 @@ C_entries (int c_ext, FILE *inf)
 		      cpptoken = false;
 		  }
 	      if (cpptoken)
-		definedef = dsharpseen;
+		{
+		  definedef = dsharpseen;
+		  /* This is needed for tagging enum values: when there are
+		     preprocessor conditionals inside the enum, we need to
+		     reset the value of fvdef so that the next enum value is
+		     tagged even though the one before it did not end in a
+		     comma.  */
+		  if (fvdef == vignore && instruct && parlev == 0)
+		    {
+		      if (strneq (cp, "#if", 3) || strneq (cp, "#el", 3))
+			fvdef = fvnone;
+		    }
+		}
 	    } /* if (definedef == dnone) */
 	  continue;
 	case '[':
@@ -3507,7 +3522,10 @@ C_entries (int c_ext, FILE *inf)
 	    case fstartlist:
 	    case finlist:
 	    case fignore:
+	      break;
 	    case vignore:
+	      if (instruct && parlev == 0)
+		fvdef = fvnone;
 	      break;
 	    case fdefunname:
 	      fvdef = fignore;

maintaining etags

[Francesco Potortì]

>> > C is a hard language to parse with ad-hoc parsers.  If you know of a
>> > better tool that doesn't get confused by the source at that point of
>> > lisp.h (did you look at it?), name it.
>> 
>> GNU Global doesn't, at least not in this example:
>> 
>> ~/v/e/src> global -x Lisp_Object
>> Lisp_Object 567 lisp.h typedef struct { EMACS_INT i; } Lisp_Object;
>> Lisp_Object 577 lisp.h typedef EMACS_INT Lisp_Object;
>
>With the patch below, neither will etags.
>
>Francesco, do you see any problems with it?

Eli, I see that you move around Etags's state machinery without problems.

In the last many years, my main job has gradually moved away from
programming, so every time I try to do something with Etags' sources I
must invest some significant amount of time even to just download the
latest sources and look at it.  To be true, I am not even sure what sort
of code revision backend Emacs developers currently use...

Long story short, my role as Etags maintainer is, to be mild, obsolete,
and maybe someone else (you? Stefan?) should take over :)

Anyway, when I used to patch Etags, I always run a regression test,
then updated the regression test with the case the patch was meant to
address.  The tests that I used can be found in the tree at
<http://fly.isti.cnr.it/pub/software/unix/etags-regression-test.tar>,
together with some garbage.  The tests are invoked from the Makefile as
the default target.  If, as I hope, someone will care to install the
regression tests in the Emacs tree I can assist with answers to
questions. 

The patch you propose is the sort of patch that should be run against a
regression test, as it is not always obvious what will happen in the
general case.

>--- lib-src/etags.c~0	2015-03-29 08:32:03 +0300
>+++ lib-src/etags.c	2015-05-10 12:15:23 +0300
>@@ -2862,7 +2862,10 @@ consider_token (char *str, int len, int
>      case st_none:
>        if (constantypedefs
> 	   && structdef == snone
>-	   && structtype == st_C_enum && bracelev > structbracelev)
>+	   && structtype == st_C_enum && bracelev > structbracelev
>+	   /* Don't tag tokens in expressions that assign values to enum
>+	      constants.  */
>+	   && fvdef != vignore)
> 	 return true;		/* enum constant */
>        switch (fvdef)
> 	 {
>@@ -3176,7 +3179,19 @@ C_entries (int c_ext, FILE *inf)
> 		      cpptoken = false;
> 		  }
> 	      if (cpptoken)
>-		definedef = dsharpseen;
>+		{
>+		  definedef = dsharpseen;
>+		  /* This is needed for tagging enum values: when there are
>+		     preprocessor conditionals inside the enum, we need to
>+		     reset the value of fvdef so that the next enum value is
>+		     tagged even though the one before it did not end in a
>+		     comma.  */
>+		  if (fvdef == vignore && instruct && parlev == 0)
>+		    {
>+		      if (strneq (cp, "#if", 3) || strneq (cp, "#el", 3))
>+			fvdef = fvnone;
>+		    }
>+		}
> 	    } /* if (definedef == dnone) */
> 	  continue;
> 	case '[':
>@@ -3507,7 +3522,10 @@ C_entries (int c_ext, FILE *inf)
> 	    case fstartlist:
> 	    case finlist:
> 	    case fignore:
>+	      break;
> 	    case vignore:
>+	      if (instruct && parlev == 0)
>+		fvdef = fvnone;
> 	      break;
> 	    case fdefunname:
> 	      fvdef = fignore;
>
>

Re: maintaining etags

[Eli Zaretskii]

> Date: Mon, 11 May 2015 17:37:22 +0200
> From: Francesco Potortì <pot@gnu.org>
> Cc: emacs-devel@gnu.org, eller.helmut@gmail.com,
>  Dmitry Gutov <dgutov@yandex.ru>, Stefan Monnier <monnier@IRO.UMontreal.CA>
> 
> Eli, I see that you move around Etags's state machinery without problems.

I wouldn't say "without problems", it took me a while staring at the
state machines to pick up enough to be able to think of a solution.
That part of etags is not easily penetrated (the parts that deal with
other languages are easier). 

> Anyway, when I used to patch Etags, I always run a regression test,
> then updated the regression test with the case the patch was meant to
> address.  The tests that I used can be found in the tree at
> <http://fly.isti.cnr.it/pub/software/unix/etags-regression-test.tar>,
> together with some garbage.  The tests are invoked from the Makefile as
> the default target.  If, as I hope, someone will care to install the
> regression tests in the Emacs tree I can assist with answers to
> questions. 

Thanks, I will definitely run the test suite, I didn't know about it.
I will also look intyo adding it to the Emacs regression suite.

> The patch you propose is the sort of patch that should be run against a
> regression test, as it is not always obvious what will happen in the
> general case.

Agreed.  (I did run it on the entire Emacs tree and examined the
differences one by one against the previous version, but the more the
better, of course.)

Re: maintaining etags

[Francesco Potortì]

>> Eli, I see that you move around Etags's state machinery without problems.
>
>I wouldn't say "without problems", it took me a while staring at the
>state machines to pick up enough to be able to think of a solution.
>That part of etags is not easily penetrated (the parts that deal with
>other languages are easier). 

Yes, C and C-like languages have a complex parser.  I gradually rewrote
it entirely maybe twice, trying to make it readable...

>> Anyway, when I used to patch Etags, I always run a regression test,
>> then updated the regression test with the case the patch was meant to
>> address.  The tests that I used can be found in the tree at
>> <http://fly.isti.cnr.it/pub/software/unix/etags-regression-test.tar>,
>> together with some garbage.  The tests are invoked from the Makefile as
>> the default target.  If, as I hope, someone will care to install the
>> regression tests in the Emacs tree I can assist with answers to
>> questions. 
>
>Thanks, I will definitely run the test suite, I didn't know about it.
>I will also look intyo adding it to the Emacs regression suite.

I had mentioned it to Stefan too, but I did not get down to preparing it
until I stumbled upon this thread.

It works simply by running the just compiled etags on a series of source
code snippets, and sometimes on whole source files, and compare the
generated tags file with the one generated by a stable version.

It would be great if it were incorporated in the Emacs test suite or
anyway made available to Emacs developers.

>> The patch you propose is the sort of patch that should be run against a
>> regression test, as it is not always obvious what will happen in the
>> general case.
>
>Agreed.  (I did run it on the entire Emacs tree and examined the
>differences one by one against the previous version, but the more the
>better, of course.)

The Emacs tree is a good test.  The advantage of the regression tests I
provide is that they should exercise all the bug squashings and the
improvements that were made to etags in the past, and do so on a small
code base.  Also, etags is compiled with debugging enabled when running
the regression test, which enables internal tests.

Re: maintaining etags

[Eli Zaretskii]

> Date: Tue, 12 May 2015 13:17:06 +0200
> From: Francesco Potortì <pot@gnu.org>
> Cc: monnier@IRO.UMontreal.CA, dgutov@yandex.ru, eller.helmut@gmail.com,
>  emacs-devel@gnu.org
> 
> >> Anyway, when I used to patch Etags, I always run a regression test,
> >> then updated the regression test with the case the patch was meant to
> >> address.  The tests that I used can be found in the tree at
> >> <http://fly.isti.cnr.it/pub/software/unix/etags-regression-test.tar>,
> >> together with some garbage.  The tests are invoked from the Makefile as
> >> the default target.  If, as I hope, someone will care to install the
> >> regression tests in the Emacs tree I can assist with answers to
> >> questions. 
> >
> >Thanks, I will definitely run the test suite, I didn't know about it.
> >I will also look intyo adding it to the Emacs regression suite.
> 
> I had mentioned it to Stefan too, but I did not get down to preparing it
> until I stumbled upon this thread.
> 
> It works simply by running the just compiled etags on a series of source
> code snippets, and sometimes on whole source files, and compare the
> generated tags file with the one generated by a stable version.
> 
> It would be great if it were incorporated in the Emacs test suite or
> anyway made available to Emacs developers.

I've run it (no regressions ;-), and prepared a stand-alone test suite
extracted from that tarball.  I will commit it to the Emacs repository
soon.

Thanks.

Re: maintaining etags

[Eli Zaretskii]

> Date: Tue, 12 May 2015 19:21:36 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: emacs-devel@gnu.org, eller.helmut@gmail.com, monnier@IRO.UMontreal.CA,
> 	dgutov@yandex.ru
> 
> > It would be great if it were incorporated in the Emacs test suite or
> > anyway made available to Emacs developers.
> 
> I've run it (no regressions ;-), and prepared a stand-alone test suite
> extracted from that tarball.  I will commit it to the Emacs repository
> soon.

Done.

Re: xref-find-matches and stuff

[Eli Zaretskii]

> Date: Sun, 10 May 2015 17:48:49 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: eller.helmut@gmail.com, emacs-devel@gnu.org
> 
> > Cc: eller.helmut@gmail.com, emacs-devel@gnu.org
> > From: Dmitry Gutov <dgutov@yandex.ru>
> > Date: Sun, 10 May 2015 00:56:36 +0300
> > 
> > On 05/09/2015 04:22 PM, Eli Zaretskii wrote:
> > 
> > > C is a hard language to parse with ad-hoc parsers.  If you know of a
> > > better tool that doesn't get confused by the source at that point of
> > > lisp.h (did you look at it?), name it.
> > 
> > GNU Global doesn't, at least not in this example:
> > 
> > ~/v/e/src> global -x Lisp_Object
> > Lisp_Object 567 lisp.h typedef struct { EMACS_INT i; } Lisp_Object;
> > Lisp_Object 577 lisp.h typedef EMACS_INT Lisp_Object;
> 
> With the patch below, neither will etags.

Installed on master.

Re: xref-find-matches and stuff

[Stefan Monnier]

> Back-ends that mis symbols make no sense, IMO.  They should be fixed
> not to miss them.

I mostly agree.  Nevertheless, as mentioned in an earlier example, etags
misses symbols.  Typically, all but one of the symbols created by
"complex" declarations such `cl-defstruct' or `define-derived-mode'.
xref/elisp does get those right, but misses other.


        Stefan

Re: xref-find-matches and stuff

[Dmitry Gutov]

On 05/11/2015 04:01 AM, Stefan Monnier wrote:

> I mostly agree.  Nevertheless, as mentioned in an earlier example, etags
> misses symbols.  Typically, all but one of the symbols created by
> "complex" declarations such `cl-defstruct' or `define-derived-mode'.
> xref/elisp does get those right, but misses other.

For cl-defstruct accessors and predicates (and even constructors), 
find-func knows which file to find them in, but reliably fails to find 
them inside that file.

Though that should be easier to fix than to make etags recognize them.

Re: xref-find-matches and stuff

[Stefan Monnier]

> For cl-defstruct accessors and predicates (and even constructors), find-func
> knows which file to find them in, but reliably fails to find them inside
> that file.
> Though that should be easier to fix than to make etags recognize them.

Maybe, maybe not, the fact is that the coverage is never 100%.


        Stefan

Re: xref-find-matches and stuff

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Eli Zaretskii <eliz@gnu.org>,  Vitalie Spinu <spinuvit@gmail.com>,  emacs-devel@gnu.org,  eller.helmut@gmail.com
> Date: Mon, 11 May 2015 09:59:22 -0400
> 
> > For cl-defstruct accessors and predicates (and even constructors), find-func
> > knows which file to find them in, but reliably fails to find them inside
> > that file.
> > Though that should be easier to fix than to make etags recognize them.
> 
> Maybe, maybe not, the fact is that the coverage is never 100%.

My point was that we should strive to get there, and for some
languages we are pretty close.  For other languages I really am not
sure there is a satisfactory solution.

Re: xref-find-matches and stuff

[Eli Zaretskii]

> Cc: Vitalie Spinu <spinuvit@gmail.com>, emacs-devel@gnu.org,
>  eller.helmut@gmail.com
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Mon, 11 May 2015 04:46:43 +0300
> 
> On 05/11/2015 04:01 AM, Stefan Monnier wrote:
> 
> > I mostly agree.  Nevertheless, as mentioned in an earlier example, etags
> > misses symbols.  Typically, all but one of the symbols created by
> > "complex" declarations such `cl-defstruct' or `define-derived-mode'.
> > xref/elisp does get those right, but misses other.
> 
> For cl-defstruct accessors and predicates (and even constructors), 
> find-func knows which file to find them in, but reliably fails to find 
> them inside that file.
> 
> Though that should be easier to fix than to make etags recognize them.

Making etags recognize them is almost trivial.  The question is, do we
want such ad-hocery in etags.c?

Re: xref-find-matches and stuff

[Dmitry Gutov]

On 05/11/2015 05:56 PM, Eli Zaretskii wrote:

> Making etags recognize them is almost trivial.

cl-defstruct accessors and predicates? I wouldn't say it's trivial, but 
it should be straightforward to implement. In Elisp, at least. In C, 
it's bound to be an order of magnitude longer.

 > The question is, do we want such ad-hocery in etags.c?

I guess not, but that's the point: within the restrictions imposed by 
etags's design and common sense, you're not going to write an indexer 
that takes into account the many different ways Elisp can define 
functions implicitly. And there's no way to specify those with --regex 
either.

Re: xref-find-matches and stuff

[Eli Zaretskii]

> Cc: monnier@iro.umontreal.ca, spinuvit@gmail.com, emacs-devel@gnu.org,
>  eller.helmut@gmail.com
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Mon, 11 May 2015 23:24:10 +0300
> 
> On 05/11/2015 05:56 PM, Eli Zaretskii wrote:
> 
> > Making etags recognize them is almost trivial.
> 
> cl-defstruct accessors and predicates? I wouldn't say it's trivial, but 
> it should be straightforward to implement. In Elisp, at least. In C, 
> it's bound to be an order of magnitude longer.

I don't see the difficulty.  Can you explain why you think it's more
than a few lines of code needed to produce the tags for accessors and
predicates "out of thin air", given just the name of the defstruct,
immediately after we produce the tag for the defstruct itself?

>  > The question is, do we want such ad-hocery in etags.c?
> 
> I guess not, but that's the point: within the restrictions imposed by 
> etags's design and common sense, you're not going to write an indexer 
> that takes into account the many different ways Elisp can define 
> functions implicitly.

If we decide that's what we want, then why won't we write that?  The
number of constructs that need to be handled might be large, but it's
finite.

Re: xref-find-matches and stuff

[Dmitry Gutov]

On 05/12/2015 05:36 AM, Eli Zaretskii wrote:

> I don't see the difficulty.  Can you explain why you think it's more
> than a few lines of code needed to produce the tags for accessors and
> predicates "out of thin air", given just the name of the defstruct,
> immediately after we produce the tag for the defstruct itself?

Upon encountering cl-defstruct foo, you'll need to collect the list of 
its slots, and create a tag for a function named foo-slotname for each.

Is there a sexp parser in etags already?

Also handle constructors, but those could be taken care of with a 
regexp, I think, without a lot of false positives.

> If we decide that's what we want, then why won't we write that?  The
> number of constructs that need to be handled might be large, but it's
> finite.

Maybe because it's not really finite. Anyone can define a new construct 
like this, with different syntax.

Re: xref-find-matches and stuff

[Eli Zaretskii]

> Cc: monnier@iro.umontreal.ca, spinuvit@gmail.com, emacs-devel@gnu.org,
>  eller.helmut@gmail.com
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Tue, 12 May 2015 14:08:20 +0300
> 
>     I don't see the difficulty.  Can you explain why you think it's more
>     than a few lines of code needed to produce the tags for accessors and
>     predicates "out of thin air", given just the name of the defstruct,
>     immediately after we produce the tag for the defstruct itself?
> 
> Upon encountering cl-defstruct foo, you'll need to collect the list of its slots, and create a tag for a function named foo-slotname for each.

If the symbols generated by a macro are highly dynamic and there are a
lot of them, and if we indeed want to tag all the slots, then it might
be a bit more complex, indeed.  But the alternative is no better,
since AFAICT it only supports interned symbols, i.e. only loaded
packages.

> Is there a sexp parser in etags already?

There's no need for a full-fledged sexp parser, I think, just for
something that can find and extract values of a fixed list of
keywords from a parenthesized list.  Hardly rocket science.

> Also handle constructors, but those could be taken care of with a regexp, I think, without a lot of false positives.

Rules for creating names of constructors and other fixed symbols could
be simply known to etags, so it could generate them given the name of
the defstruct.

>     If we decide that's what we want, then why won't we write that?  The
>     number of constructs that need to be handled might be large, but it's
>     finite.
>
> Maybe because it's not really finite. Anyone can define a new construct like this, with different syntax.

I see no problem here, we already have a few Emacs-specific tricks in
etags, a few more won't hurt.  The question is do we want that?  If
most people are happy with having access only to symbols they have
interned, then I guess it isn't worth our while to make etags
especially smart in this regard.

One thing we could do is provide a way for the user to ask for looser
matches if the exact ones failed.  Then foo-mode will be considered a
potential match for foo-mode-abbrev-table, which is exactly what is
needed in these cases.

Re: xref-find-matches and stuff

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Sun, 10 May 2015 21:01:42 -0400
> Cc: Vitalie Spinu <spinuvit@gmail.com>, eller.helmut@gmail.com,
> 	dgutov@yandex.ru, emacs-devel@gnu.org
> 
> > Back-ends that miss symbols make no sense, IMO.  They should be fixed
> > not to miss them.
> 
> I mostly agree.  Nevertheless, as mentioned in an earlier example, etags
> misses symbols.  Typically, all but one of the symbols created by
> "complex" declarations such `cl-defstruct' or `define-derived-mode'.

Do we really want/expect etags to fully support languages that produce
symbols dynamically?  It's not up to the job.  We could add some
ad-hoc rules for declarations such as the above, though.  E.g., each
use of define-derived-mode would automagically "define" MODE-map,
MODE-hook, MODE-syntax-table, and MODE-abbrev-table.  If that's what
we want, it shouldn't be hard to add that to etags.c.  Or we could
teach xref to look for MODE if MODE-abbrev-table is not found.  But
the result will still be only a partial coverage.

These languages need a different approach, IMO.  And I'm not sure any
approach will yield better results.