Re: [Emacs-diffs] emacs-25 f8208b6: Document the user-level features of the Xref package

Re: [Emacs-diffs] emacs-25 f8208b6: Document the user-level features of the Xref package

[Dmitry Gutov]

Hi Eli,

Thanks for doing this.

I've noticed some problems, though, described below:

On 01/09/2016 10:14 PM, Eli Zaretskii wrote:

> +Emacs provides a unified user interface to these tools, called
> +@samp{xref}.  The tools supported by @samp{xref} include:
> +...
> +@acronym{GNU} GLOBAL, the source code tagging system, which provides
> ...
> +Cscope (@uref{http://cscope.sourceforge.net/}, a tool for browsing
> ...
> +@acronym{GNU} IDUtils, a package for generating databases of
> ...
> +Grep, the venerable program that searches files for lines matching
> ...
> +Additional tools could be supported as they become available, or as
> +user extensions.  Each such tool is used as a @dfn{backend} by
> +commands described in this section.  Each command detects which
> +backends are available for the current major mode, and uses the most
> +capable of the available backends, with Grep generally serving as the
> +fall-back backend.

Sorry, this is not true, unfortunately, as things currently stand.

We don't use GNU Global, Cscope, ID Utils, or Grep as xref backends. We 
can't "find definitions" using any of them, for example, even though the 
user might expect to be able to do so.

The only place where the aforementioned tools are used, is in the 
_default_ implementation of the generic function 
xref-backend-references. IOW, the implementation that gets used when the 
current backend doesn't provide its own.

> +@cindex tag
> +The commands described here are useful for finding references in major
> +modes other than those defined to support programming languages.  For
> +example, chapters, sections, appendices, etc. of a text or a @TeX{}
> +document can be treated as identifiers as well.  In this chapter, we
> +collectively refer to a reference that specifies the name of the file
> +where the corresponding subunit is defined, and the position of the
> +subunit's definition in that file, as a @dfn{tag}.  We refer to the
> +backends used by @code{xref} as @dfn{tagging backends}.

"tagging backends"? Why not call them "Xref backends", like referred to 
in the Xref commentary and the docstrings?

I wouldn't say that the current terminology in Xref is perfect (the name 
itself encroaches on info-xref.el, for instance), but introducing terms 
that are only used in the manual, won't help clarity.

Why not submit a patch for xref.el first?

> +@node Identifier Search
> +@subsubsection Searching and Replacing with Identifiers
> +@cindex search and replace in multiple files
> +@cindex multiple-file search and replace
> +
> +  The commands in this section visit and search all the files listed
> +in the @code{xref} backend's database, one by one.  For these
> +commands, the database serves only to specify a sequence of files to
> +search.

That may be true for tags-search, but that's never true for 
xref-find-references.

If the current backend defines xref-backend-references, then we just ask 
it, and show the results.

If it doesn't, we delegate to the first available "CEDET tool", but they 
do perform the regexp search, not just list the files. Then, 
xref-collect-references takes the list of matches and verified, for each 
one, whether the match begins and ends on a symbol boundary (Grep can't 
check that).

> These commands scan all the databases starting with the first
> +one (if any) that describes the current file, proceed from there to
> +the end of the list, and then scan from the beginning of the list
> +until they have covered all the databases in the list.

What list? xref definitely uses one database at a time (per project 
root, if we're talking about the default implementation).

> +@findex tags-search
> +  @kbd{M-x tags-search} reads a regexp using the minibuffer, then
> +searches for matches in all the files in the selected tags table, one
> +file at a time.  It displays the name of the file being searched so
> +you can follow its progress.  As soon as it finds an occurrence,
> +@code{tags-search} returns.  This command works only with the etags
> +backend, and requires tags tables to be available (@pxref{Tags
> +Tables}).

The last sentence makes it seem like it goes through the xref machinery, 
whereas it doesn't.

> +@findex tags-loop-continue
> +  Having found one match, you probably want to find all the rest.
> +Type @kbd{M-x tags-loop-continue}) to resume the @code{tags-search},
> +finding one more match.  This searches the rest of the current buffer,
> +followed by the remaining files of the tags table.

This makes it seem like tags-loop-continue is applicable for use after 
xref-find-* commands as well. But it isn't.

Maybe this command, as well as tags-query-replace, the other etags-only 
commands, and the related etags variables, should still remain in a 
separate section?

> +@node List Identifiers
> +@subsubsection Identifier Inquiries
> +
> +@table @kbd
> +@item C-M-i
> +@itemx M-@key{TAB}
> +Perform completion on the text around point, using the @code{xref}
> +backend if one is available (@code{completion-at-point}).

If I understand this sentence right, it's inaccurate: xref backends 
don't define completion tables for buffer text. It is still governed by 
completion-at-point-functions.

xref-backend-identifier-completion-table is only used for completion in 
the prompt when reading an identifier, by xref commands.

> +@c Sadly, the new-and-improved Xref feature doesn't provide anything
> +@c close to the described below features of the now-obsoleted
> +@c tags-apropos.  I'm leaving this here to encourage enhancements to
> +@c xref.el.

Writing this complaint in the manual, expecting it to be read by someone 
likely to produce enhancements to xref.el, seems rather optimistic.

Please submit a feature request (or several, if they're distinct). 
Displaying the tags files names should be possible after the xref 
rendering logic becomes more flexible (and e.g. could be supplied by the 
backend).

>   mode is enabled, it tries to use the Semantic parser data for
>   completion (@pxref{Semantic}).  If Semantic mode is not enabled or
>   fails at performing completion, it tries to complete using the
> -selected tags table (@pxref{Tags}).  If in Emacs Lisp mode, it
> +available @code{xref} backend (@pxref{Xref}).

Also not true. Like mentioned, we don't use xref backends for 
completion. Unifying in-buffer completion and xref identifier completion 
might be beneficial, but it's not easy to do.

Re: [Emacs-diffs] emacs-25 f8208b6: Document the user-level features of the Xref package

[Eli Zaretskii]

> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sun, 10 Jan 2016 06:02:42 +0300
> 
> Thanks for doing this.
> 
> I've noticed some problems, though, described below:

Thanks for reviewing the changes.

> On 01/09/2016 10:14 PM, Eli Zaretskii wrote:

>     +Emacs provides a unified user interface to these tools, called
>     +@samp{xref}.  The tools supported by @samp{xref} include:
>     +...
>     +@acronym{GNU} GLOBAL, the source code tagging system, which provides
>     ...
>     +Cscope (@uref{http://cscope.sourceforge.net/}, a tool for browsing
>     ...
>     +@acronym{GNU} IDUtils, a package for generating databases of
>     ...
>     +Grep, the venerable program that searches files for lines matching
>     ...
>     +Additional tools could be supported as they become available, or as
>     +user extensions.  Each such tool is used as a @dfn{backend} by
>     +commands described in this section.  Each command detects which
>     +backends are available for the current major mode, and uses the most
>     +capable of the available backends, with Grep generally serving as the
>     +fall-back backend.
> 
> Sorry, this is not true, unfortunately, as things currently stand.
> 
> We don't use GNU Global, Cscope, ID Utils, or Grep as xref backends. We can't "find definitions" using any of them, for example, even though the user might expect to be able to do so.
> 
> The only place where the aforementioned tools are used, is in the _default_ implementation of the generic function xref-backend-references. IOW, the implementation that gets used when the current backend doesn't provide its own.

Sorry, I don't understand: AFAIK (and you seem to confirm) we do use
these tools in some situations, so the text is correct in saying that
they are used when appropriate.  So what is the nature of the problem
you have with this text?  Is it that I used "backend" in a meaning
that's different from what the xref implementation uses?  If so,
that's deliberate: this is a user manual, and that was the only
reasonable way I could think of to convey the idea of several backends
without talking about technicalities like xref-backend-references,
what symref does, when do we use the fallbacks, what are generic
functions, etc.

The idea the user will hopefully take out of this is that we use
several potential helpers under the hood some of which are based on
the mentioned tools that are not part of Emacs proper.  The tools are
mentioned to encourage the users to install them, so that their Emacs
environment is more complete and capable.

>     +The commands described here are useful for finding references in major
>     +modes other than those defined to support programming languages.  For
>     +example, chapters, sections, appendices, etc. of a text or a @TeX{}
>     +document can be treated as identifiers as well.  In this chapter, we
>     +collectively refer to a reference that specifies the name of the file
>     +where the corresponding subunit is defined, and the position of the
>     +subunit's definition in that file, as a @dfn{tag}.  We refer to the
>     +backends used by @code{xref} as @dfn{tagging backends}.
> 
> "tagging backends"? Why not call them "Xref backends", like referred to in the Xref commentary and the docstrings?

I ended up not using this term, so I will just delete it, to avoid
confusion.

>     +@node Identifier Search
>     +@subsubsection Searching and Replacing with Identifiers
>     +@cindex search and replace in multiple files
>     +@cindex multiple-file search and replace
>     +
>     +  The commands in this section visit and search all the files listed
>     +in the @code{xref} backend's database, one by one.  For these
>     +commands, the database serves only to specify a sequence of files to
>     +search.
> 
> That may be true for tags-search, but that's never true for xref-find-references.

It's true when xref-find-references uses IDUtils, for example.  And
even when it doesn't, the command does use _some_ database, even if
that database is made up of doc strings loaded into memory plus
load-history.

Once again, look at this from the user POV, not from the POV of
someone who knows how the implementation works.  Users need some
mental model of what's going on to make some sense and order out of a
heap of potentially unrelated features.  The above should provide
users with a conceptual framework that facilitates coming up with such
a mental model.  At least I hope it will.

> If the current backend defines xref-backend-references, then we just ask it, and show the results.
> 
> If it doesn't, we delegate to the first available "CEDET tool", but they do perform the regexp search, not just list the files. Then, xref-collect-references takes the list of matches and verified, for each one, whether the match begins and ends on a symbol boundary (Grep can't check that).

I'm not describing xref alone, I'm describing what xref _and_ its
backend do together.  Differentiating between the two parts of the job
is not useful for the purposes of the user manual.

In any case, what you describe above is not something that can be
reasonably said in a user manual, can it?  You describe the (current)
implementation, whereas the user manual has quite different goals.
You are saying that the description I came up with loses some
accuracy, which is something I'm aware of, and which is inevitable
when describing complex software made up of several independent
modules whose features only partially overlap.

If you have suggestions for how to convey these ideas without going
into implementation, I'm all ears, of course.

>       These commands scan all the databases starting with the first
>     +one (if any) that describes the current file, proceed from there to
>     +the end of the list, and then scan from the beginning of the list
>     +until they have covered all the databases in the list.
> 
> What list? xref definitely uses one database at a time (per project root, if we're talking about the default implementation).

Most of the commands described in the section from which this text
comes are in etags.el, for which the description is accurate.  The
only other command is M-? (xref-query-replace), for which this is
somewhat less accurate, but not entirely incorrect: a single database
is still a special case of "a list of all relevant databases".

Once again, feel free to suggest more useful ways of conveying the
ideas such that they make sense to users.  The above was the best I
could come up with, and I'll be happy to hear about possible
improvements, if they exist.

>     +@findex tags-search
>     +  @kbd{M-x tags-search} reads a regexp using the minibuffer, then
>     +searches for matches in all the files in the selected tags table, one
>     +file at a time.  It displays the name of the file being searched so
>     +you can follow its progress.  As soon as it finds an occurrence,
>     address@hidden returns.  This command works only with the etags
>     +backend, and requires tags tables to be available (@pxref{Tags
>     +Tables}).
> 
> The last sentence makes it seem like it goes through the xref machinery, whereas it doesn't.

The last sentence mentions backends because the parent section is
built upon that concept: that these commands use the appropriate
backend as they see fit.  That concept was introduced as an attempt to
make sense of the fact that xref implemented only part of what's
available with etags.  I do indeed think that IWBNI xref provided a
front-end for these features as well (it would have made my job of
documenting it a whole lot easier, if nothing else), but the text as
it stands stops short of saying explicitly that xref is involved.

OTOH, I don't see much harm if users somehow understand that xref is
involved (although the name of the command should tell otherwise).

>     +@findex tags-loop-continue
>     +  Having found one match, you probably want to find all the rest.
>     +Type @kbd{M-x tags-loop-continue}) to resume the @code{tags-search},
>     +finding one more match.  This searches the rest of the current buffer,
>     +followed by the remaining files of the tags table.
> 
> This makes it seem like tags-loop-continue is applicable for use after xref-find-* commands as well. But it isn't.

Well, it explicitly references tags-search and comes directly after
the description of that command.  Would it be better if the first
sentence said that explicitly, like below?

  Having found one match with @kbd{M-x tags-search}, you probably
  want to find all the rest.

> Maybe this command, as well as tags-query-replace, the other etags-only commands, and the related etags variables, should still remain in a separate section?

No, I don't think so.  It would make two sections out of one, which
would mean less order in the description.

>     +@table @kbd
>     +@item C-M-i
>     +@itemx M-@key{TAB}
>     +Perform completion on the text around point, using the @code{xref}
>     +backend if one is available (@code{completion-at-point}).
> 
> If I understand this sentence right, it's inaccurate: xref backends don't define completion tables for buffer text. It is still governed by completion-at-point-functions.

Is xref entirely not involved in completion-at-point-functions?  Do we
still use etags there?  The original text mentioned etags, and I
imagined that was replaced by xref now.  (I couldn't trace the code to
see what was it actually doing.)

>     @c Sadly, the new-and-improved Xref feature doesn't provide anything
>     @c close to the described below features of the now-obsoleted
>     @c tags-apropos.  I'm leaving this here to encourage enhancements to
>     @c xref.el.
> 
> Writing this complaint in the manual, expecting it to be read by someone likely to produce enhancements to xref.el, seems rather optimistic.

I'm an optimist, yes.

> Please submit a feature request (or several, if they're distinct). Displaying the tags files names should be possible after the xref rendering logic becomes more flexible (and e.g. could be supplied by the backend).

Will do when I have time.

>       mode is enabled, it tries to use the Semantic parser data for
>       completion (@pxref{Semantic}).  If Semantic mode is not enabled or
>       fails at performing completion, it tries to complete using the
>     -selected tags table (@pxref{Tags}).  If in Emacs Lisp mode, it
>     +available @code{xref} backend (@pxref{Xref}).
> 
> Also not true. Like mentioned, we don't use xref backends for completion. Unifying in-buffer completion and xref identifier completion might be beneficial, but it's not easy to do.

I only changed the cross-reference, since there's no node named "Tags"
anymore; and etags is one of xref backends.  Are you saying that
completion-at-point has been changed not to use the tags table when
the Semantic mode is not enabled?  Or are you saying that this text
was incorrect even before my changes, and we didn't use tags tables
even before xref was added?

Thanks.

Re: [Emacs-diffs] emacs-25 f8208b6: Document the user-level features of the Xref package

[Dmitry Gutov]

On 01/10/2016 06:50 PM, Eli Zaretskii wrote:

> Sorry, I don't understand: AFAIK (and you seem to confirm) we do use
> these tools in some situations, so the text is correct in saying that
> they are used when appropriate.

No, these are programs that we just use in some circumstances.

The text says: "Each such tool is used as a @dfn{backend} by commands 
described in this section". This is untrue, they're not used as 
"backends" in Xref terms. And most of them are only used by one command 
(singular, not plural): xref-find-references, and only when an actual 
backend doesn't provide a replacement logic.

> Is it that I used "backend" in a meaning
> that's different from what the xref implementation uses?

Yes. You can call them "programs", or "external tools" like CEDET's docs 
prefer to call them.

This part also makes using the term "backend" for them inaccurate: "Each 
command detects which backends are available for the current major mode".

Xref backends *do* depend on the current major mode. Or can depend, at 
least. But whether we use ggtags, cscope, or grep, doesn't depend on the 
current major mode at all, just on whether there is an index file 
generated by the respective external program, in some root directory.

semantic-symref-filepattern-alist uses the major-mode value to derive 
the file wildcards to search for, but not, again the tools to use.

> If so,
> that's deliberate: this is a user manual, and that was the only
> reasonable way I could think of to convey the idea of several backends
> without talking about technicalities like xref-backend-references,
> what symref does, when do we use the fallbacks, what are generic
> functions, etc.

A backend is a value that can be returned by xref-find-backend. By 
diluting the term, you all but make sure that the same users will become 
confused later, when they peek under the covers, to try to fix 
something, or use xref in their own programs, etc.

If you think the users are fine with only superficial understanding, 
let's keep the description at that level, and not include misleading 
specifics.

> The idea the user will hopefully take out of this is that we use
> several potential helpers under the hood some of which are based on
> the mentioned tools that are not part of Emacs proper.  The tools are
> mentioned to encourage the users to install them, so that their Emacs
> environment is more complete and capable.

Let's call them "external tools", then.

>>      +  The commands in this section visit and search all the files listed
>>      +in the @code{xref} backend's database, one by one.  For these
>>      +commands, the database serves only to specify a sequence of files to
>>      +search.
>>
>> That may be true for tags-search, but that's never true for xref-find-references.
>
> It's true when xref-find-references uses IDUtils, for example.  And
> even when it doesn't, the command does use _some_ database, even if
> that database is made up of doc strings loaded into memory plus
> load-history.

The specific sentence that isn't true is: "For these commands, the 
database serves only to specify a sequence of files to search."

No, not only. It serves for more, IIUC.

Again, here you present unnecessary specifics (backend has a database, 
database is only a list of files), which aren't really true in all cases.

> Once again, look at this from the user POV, not from the POV of
> someone who knows how the implementation works.  Users need some
> mental model of what's going on to make some sense and order out of a
> heap of potentially unrelated features.  The above should provide
> users with a conceptual framework that facilitates coming up with such
> a mental model.  At least I hope it will.

I'd rather the imaginary framework didn't stray too far from reality.

>> If the current backend defines xref-backend-references, then we just ask it, and show the results.
>>
>> If it doesn't, we delegate to the first available "CEDET tool", but they do perform the regexp search, not just list the files. Then, xref-collect-references takes the list of matches and verified, for each one, whether the match begins and ends on a symbol boundary (Grep can't check that).
>
> I'm not describing xref alone, I'm describing what xref _and_ its
> backend do together.  Differentiating between the two parts of the job
> is not useful for the purposes of the user manual.

You _don't know_ what an arbitrary backend does. Xref just asks it, and 
the backend returns a list. You can describe what elisp and etags 
backends do, but that's not what's happening here.

> In any case, what you describe above is not something that can be
> reasonably said in a user manual, can it?  You describe the (current)
> implementation, whereas the user manual has quite different goals.

No, it's you who seem to be describing the implementation, and end up 
doing it not accurately. I'd be entirely fine with not describing the 
implementation at all.

> You are saying that the description I came up with loses some
> accuracy, which is something I'm aware of, and which is inevitable
> when describing complex software made up of several independent
> modules whose features only partially overlap.

I think we should either describe things accurately, or omit describing 
what we don't know. "Magic happens" is fine, the user just needs to know 
the initial requirements and the end result.

On the other hand, I haven't seem some crucial information mentioned:

- The the databases used by these tools can get out of date, and it's 
the user's responsibility to update them. (Hopefully, we will solve this 
problem sometime).

- Each of the external tools, except for Grep, supports only a fixed set 
of programming languages. If, in a multi-language project, we open a 
file using an exotic-ish language, and ggtags database is present in the 
project root, xref-find-references will return an empty result, erroneously.

> If you have suggestions for how to convey these ideas without going
> into implementation, I'm all ears, of course.

I'd prefer something like:

Xref has a set of commands, which ask for data from the currently active 
Xref backend. And use its identifier-at-point and 
identifier-completion-table logic as well.

And there's this special command called xref-find-references. We wanted 
functionality like that very much, but turns out that both built-in 
backends (elisp and etags), aren't good at this sort of thing.

So we've written a default implementation for xref-backend-references, 
which is used when a backend can't give us better data. That 
implementation uses the "external tools" feature borrowed from CEDET, 
that uses one of the tools from the list, provided the necessary program 
is present on the computer, and a database file has been generated, 
falling back to Grep. We could always use Grep here, but ggtags and 
friends are faster. On the flip side, their databases don't support all 
languages, and can get out of date, we're very sorry about that.

Oh, and by the way, that default implementation also uses the Project 
package, to determine which directories to scan.

>> What list? xref definitely uses one database at a time (per project root, if we're talking about the default implementation).
>
> Most of the commands described in the section from which this text
> comes are in etags.el, for which the description is accurate.  The
> only other command is M-? (xref-query-replace), for which this is

M-? is xref-find-references.

> somewhat less accurate, but not entirely incorrect: a single database
> is still a special case of "a list of all relevant databases".

This looks confusing to me.

> Once again, feel free to suggest more useful ways of conveying the
> ideas such that they make sense to users.  The above was the best I
> could come up with, and I'll be happy to hear about possible
> improvements, if they exist.

In this case, I'm not entirely sure why that piece of text exists.

>>      +@findex tags-search
...
>> The last sentence makes it seem like it goes through the xref machinery, whereas it doesn't.
>
> The last sentence mentions backends because the parent section is
> built upon that concept: that these commands use the appropriate
> backend as they see fit.

And this is blatantly inaccurate in this case. I think that tells us 
that the sections' organization should be different.

> That concept was introduced as an attempt to
> make sense of the fact that xref implemented only part of what's
> available with etags.  I do indeed think that IWBNI xref provided a
> front-end for these features as well

But then, it kind of already does, doesn't it? You can initiate a regexp 
search with project-find-regexp (and even choose the directory with 
C-u), and perform the query-replace from the results buffer.

It would be easy to create dired-specific version of this command, that 
will use the marked directories, if we want to. Just let me know what to 
call it.

>> This makes it seem like tags-loop-continue is applicable for use after xref-find-* commands as well. But it isn't.
>
> Well, it explicitly references tags-search and comes directly after
> the description of that command.  Would it be better if the first
> sentence said that explicitly, like below?

Probably.

>    Having found one match with @kbd{M-x tags-search}, you probably
>    want to find all the rest.
>
>> Maybe this command, as well as tags-query-replace, the other etags-only commands, and the related etags variables, should still remain in a separate section?
>
> No, I don't think so.  It would make two sections out of one, which
> would mean less order in the description.

I really think there should be separation. etags-specific commands, and 
commands that _can_ dispatch to etags (but not necessarily do), and go 
through Xref, are very different beasts. They provide different UI, too.

> Is xref entirely not involved in completion-at-point-functions?

Entirely, yes. But I expect some major and minor modes will 
simultaneously add elements to completion-at-point-functions and 
xref-backend-functions, and those elements could dispatch to the same 
underlying logic under the covers. But the APIs are separate.

> Do we
> still use etags there?

etags by default, elisp-completion-at-point in emacs-lisp-buffers. Other 
major and minor modes set their completion functions as well.

> The original text mentioned etags, and I
> imagined that was replaced by xref now.  (I couldn't trace the code to
> see what was it actually doing.)

You can look at the value of completion-at-point-functions in a c-mode, 
and an emacs-lisp-mode buffers.

>>        mode is enabled, it tries to use the Semantic parser data for
>>        completion (@pxref{Semantic}).  If Semantic mode is not enabled or
>>        fails at performing completion, it tries to complete using the
>>      -selected tags table (@pxref{Tags}).  If in Emacs Lisp mode, it
>>      +available @code{xref} backend (@pxref{Xref}).
>>
>> Also not true. Like mentioned, we don't use xref backends for completion. Unifying in-buffer completion and xref identifier completion might be beneficial, but it's not easy to do.
>
> I only changed the cross-reference, since there's no node named "Tags"
> anymore; and etags is one of xref backends.

etags contains an Xref backend (which is mostly a set of functions that 
have both "xref" and "etags" in their names), but I wouldn't call the 
whole etags package an Xref backend.

In particular, tags-completion-at-point-function is not used by the 
Etags Xref backend. They both delegate to tags-lazy-completion-table, 
however.

So I think this sentence confuses things by implying that Xref backends 
are responsible for in-buffer completion. I.e., in practical terms, 
implying that one could register a new Xref backend (by adding an 
element to xref-backend-functions), and that would take care of 
completion support. Which isn't true.

> Are you saying that
> completion-at-point has been changed not to use the tags table when
> the Semantic mode is not enabled?  Or are you saying that this text
> was incorrect even before my changes, and we didn't use tags tables
> even before xref was added?

I wouldn't call the text entirely accurate before (since it only lists 
Emacs Lisp mode as an exception, implying all other modes use etags for 
completion). But it's the new text that I'm really worried about.

Re: [Emacs-diffs] emacs-25 f8208b6: Document the user-level features of the Xref package

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sun, 10 Jan 2016 21:05:35 +0300
> 
> The text says: "Each such tool is used as a @dfn{backend} by commands 
> described in this section". This is untrue, they're not used as 
> "backends" in Xref terms. And most of them are only used by one command 
> (singular, not plural): xref-find-references, and only when an actual 
> backend doesn't provide a replacement logic.

AFAIU, that's not entirely true, because at least some of those can
produce etags-compatible TAGS tables.  It's hard to be 100% accurate
with this, as you see.

> > Is it that I used "backend" in a meaning
> > that's different from what the xref implementation uses?
> 
> Yes. You can call them "programs", or "external tools" like CEDET's docs 
> prefer to call them.

But some of them aren't programs or external tools at all, like the
elisp backend.  So that, too, is inaccurate.

> This part also makes using the term "backend" for them inaccurate: "Each 
> command detects which backends are available for the current major mode".
> 
> Xref backends *do* depend on the current major mode. Or can depend, at 
> least. But whether we use ggtags, cscope, or grep, doesn't depend on the 
> current major mode at all, just on whether there is an index file 
> generated by the respective external program, in some root directory.
> 
> semantic-symref-filepattern-alist uses the major-mode value to derive 
> the file wildcards to search for, but not, again the tools to use.

These are implementation details that are not relevant at this level,
IMO.

> > If so,
> > that's deliberate: this is a user manual, and that was the only
> > reasonable way I could think of to convey the idea of several backends
> > without talking about technicalities like xref-backend-references,
> > what symref does, when do we use the fallbacks, what are generic
> > functions, etc.
> 
> A backend is a value that can be returned by xref-find-backend.

The text uses "backend" in a different meaning than xref.el does.  We
are using the same word to refer to 2 different, though related,
beasts.

> > The idea the user will hopefully take out of this is that we use
> > several potential helpers under the hood some of which are based on
> > the mentioned tools that are not part of Emacs proper.  The tools are
> > mentioned to encourage the users to install them, so that their Emacs
> > environment is more complete and capable.
> 
> Let's call them "external tools", then.

They are not necessarily external, see above.

Believe me, I've been through all that already.  There are no easy
ways out here, at least I didn't find one.  Suggestions are welcome,
but what you suggest above is not better, IMO, it's just a different
type of inaccuracy.

> >>      +  The commands in this section visit and search all the files listed
> >>      +in the @code{xref} backend's database, one by one.  For these
> >>      +commands, the database serves only to specify a sequence of files to
> >>      +search.
> >>
> >> That may be true for tags-search, but that's never true for xref-find-references.
> >
> > It's true when xref-find-references uses IDUtils, for example.  And
> > even when it doesn't, the command does use _some_ database, even if
> > that database is made up of doc strings loaded into memory plus
> > load-history.
> 
> The specific sentence that isn't true is: "For these commands, the 
> database serves only to specify a sequence of files to search."
> 
> No, not only. It serves for more, IIUC.

What else does it serve for?

> > Once again, look at this from the user POV, not from the POV of
> > someone who knows how the implementation works.  Users need some
> > mental model of what's going on to make some sense and order out of a
> > heap of potentially unrelated features.  The above should provide
> > users with a conceptual framework that facilitates coming up with such
> > a mental model.  At least I hope it will.
> 
> I'd rather the imaginary framework didn't stray too far from reality.

I agree.  It doesn't.

> >> If the current backend defines xref-backend-references, then we just ask it, and show the results.
> >>
> >> If it doesn't, we delegate to the first available "CEDET tool", but they do perform the regexp search, not just list the files. Then, xref-collect-references takes the list of matches and verified, for each one, whether the match begins and ends on a symbol boundary (Grep can't check that).
> >
> > I'm not describing xref alone, I'm describing what xref _and_ its
> > backend do together.  Differentiating between the two parts of the job
> > is not useful for the purposes of the user manual.
> 
> You _don't know_ what an arbitrary backend does.

Of course, I do: I have the code of the existing backends, don't I?

> >> This makes it seem like tags-loop-continue is applicable for use after xref-find-* commands as well. But it isn't.
> >
> > Well, it explicitly references tags-search and comes directly after
> > the description of that command.  Would it be better if the first
> > sentence said that explicitly, like below?
> 
> Probably.

OK, I will make that change.

> > Is xref entirely not involved in completion-at-point-functions?
> 
> Entirely, yes. But I expect some major and minor modes will 
> simultaneously add elements to completion-at-point-functions and 
> xref-backend-functions, and those elements could dispatch to the same 
> underlying logic under the covers. But the APIs are separate.
> 
> > Do we
> > still use etags there?
> 
> etags by default, elisp-completion-at-point in emacs-lisp-buffers. Other 
> major and minor modes set their completion functions as well.

Thanks, I will fix that inaccuracy.

> >>        mode is enabled, it tries to use the Semantic parser data for
> >>        completion (@pxref{Semantic}).  If Semantic mode is not enabled or
> >>        fails at performing completion, it tries to complete using the
> >>      -selected tags table (@pxref{Tags}).  If in Emacs Lisp mode, it
> >>      +available @code{xref} backend (@pxref{Xref}).
> >>
> I wouldn't call the text entirely accurate before (since it only lists 
> Emacs Lisp mode as an exception, implying all other modes use etags for 
> completion). But it's the new text that I'm really worried about.

So would you say that this:

  If Semantic mode is not enabled or fails at performing completion,
  it tries to complete using the selected tags table

is accurate enough?  Or does it need any further fixes?

Thanks.

Re: [Emacs-diffs] emacs-25 f8208b6: Document the user-level features of the Xref package

[Dmitry Gutov]

On 01/10/2016 09:59 PM, Eli Zaretskii wrote:

> AFAIU, that's not entirely true, because at least some of those can
> produce etags-compatible TAGS tables.  It's hard to be 100% accurate
> with this, as you see.

That's an exotic use case, which can be described separately. But even 
if the user does produce a TAGS file through one of these tools, it's 
the Etags Xref backend that will be used to obtain information from it.

>> Yes. You can call them "programs", or "external tools" like CEDET's docs
>> prefer to call them.
>
> But some of them aren't programs or external tools at all, like the
> elisp backend.  So that, too, is inaccurate.

Elisp backend is one of the two Xref backends we have. The rest of the 
"backends" you've described, are "external tools", which are used in a 
different way than "proper" Xref backends.

So I want the Elisp Xref backend to continue to be called an Xref 
backend, but I don't want GNU Global called that, until we actually have 
an Xref backend for it.

>> This part also makes using the term "backend" for them inaccurate: "Each
>> command detects which backends are available for the current major mode".
>>
>> Xref backends *do* depend on the current major mode. Or can depend, at
>> ...
>
> These are implementation details that are not relevant at this level,
> IMO.

Yes. I'm just describing, in detail, how the current manual text is wrong.

> The text uses "backend" in a different meaning than xref.el does.  We
> are using the same word to refer to 2 different, though related,
> beasts.

We shouldn't, they're really different.

>> Let's call them "external tools", then.
>
> They are not necessarily external, see above.

They are. I'm not proposing to call the Elisp backend an external tool.

> Believe me, I've been through all that already.  There are no easy
> ways out here, at least I didn't find one.  Suggestions are welcome,
> but what you suggest above is not better, IMO, it's just a different
> type of inaccuracy.

I don't see how.

>> The specific sentence that isn't true is: "For these commands, the
>> database serves only to specify a sequence of files to search."
>>
>> No, not only. It serves for more, IIUC.
>
> What else does it serve for?

When we use the GNU Global external tool to produce all references to a 
symbol, it consults its database, and returns the actual references to 
the symbol, not just the list of files.

>>> I'm not describing xref alone, I'm describing what xref _and_ its
>>> backend do together.  Differentiating between the two parts of the job
>>> is not useful for the purposes of the user manual.

My point is, a large part of this description will be inaccurate if the 
current Xref backend does implement xref-backend-references.

>> You _don't know_ what an arbitrary backend does.
>
> Of course, I do: I have the code of the existing backends, don't I?

It's better to maintain the levels of abstraction: if we're describing 
Xref commands, we should describe what the commands are for, and what 
output the user should expect, but treat the backends as black boxes as 
much as possible.

And separately (maybe nearby, maybe in a different section), describe 
the actual backends that come with Emacs. What Xref commands they 
support, and what commands they don't support. What data sources they 
use (say, load-history for Elisp backend, and TAGS files for etags backend).

Going back to what this part of the subthread started from: "The 
commands in this section visit and search all the files listed in the 
@code{xref} backend's database, one by one.  For these commands, the 
database serves only to specify a sequence of files to search."

This isn't true. It's halfway true when the current Xref backend does 
_not_ implement xref-backend-references. And even then, like I mentioned 
above, "search all the files listed in the @code{xref} backend's 
database, one by one" is a highly questionable description.

> So would you say that this:
>
>    If Semantic mode is not enabled or fails at performing completion,
>    it tries to complete using the selected tags table
>
> is accurate enough?  Or does it need any further fixes?

It's better.

I feel that this entire mode of description is broken (instead, we 
should say that it completes using whatever 
completion-at-point-functions says to use; and its elements can be set 
by Semantic, etags, major modes, minor modes, etc), but that would 
require more changes.

And if the previous version of the manual didn't elicit any complaints, 
perhaps it was good enough.

Re: [Emacs-diffs] emacs-25 f8208b6: Document the user-level features of the Xref package

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sun, 10 Jan 2016 22:32:48 +0300
> 
> Elisp backend is one of the two Xref backends we have. The rest of the 
> "backends" you've described, are "external tools", which are used in a 
> different way than "proper" Xref backends.
> 
> So I want the Elisp Xref backend to continue to be called an Xref 
> backend, but I don't want GNU Global called that, until we actually have 
> an Xref backend for it.

Can you propose an alternative conceptual framework for the user
manual, wrt to backends and the features described in this section?

Here are the problems I bumped into when I worked on this section:

 . It is impractical to try to ignore the existence of backends: many
   functions and commands mention them in their names and doc strings,
   so the fact of their existence is pretty much into the user's face.
   I couldn't ignore them.

 . There are only 2 proper backends, but then there are "fallbacks",
   like symref, Grep, etc. that are used in important commands.  What
   do we call those "non-backends", and how do we describe them
   without confusing the heck out of the readers?

 . How to describe a bunch of related features, of which only part is
   implemented by Xref, the rest are still (and probably always will
   be) in Etags, Semantic, etc.?  How to describe commands whose
   results could be very different depending on the backend and on the
   major mode?

 . How to make all of this reasonably future-proof, given that the
   functionality implemented today covers only a small portion of the
   turf it was supposed to?

Faced with these challenges, I came up with the solution that is now
before your eyes.  What alternative do you suggest?  Can you present a
coherent conceptual framework for describing these features, and a
structure of the section to go with that framework?

It is futile to argue about minor details if we disagree about the
basics.  If you can propose a coherent alternative for describing
these features, maybe we can make some progress.  Otherwise, I will
fix the few specific minor aspects that you've pointed out, and then I
will quickly try to forget about this ungrateful job I took upon
myself.

Thanks for the other comments, I will get to them in due time.

Re: [Emacs-diffs] emacs-25 f8208b6: Document the user-level features of the Xref package

[Dmitry Gutov]

On 01/10/2016 11:51 PM, Eli Zaretskii wrote:

> Can you propose an alternative conceptual framework for the user
> manual, wrt to backends and the features described in this section?

Before I try answering this, please confirm that you've already read my 
brief attempt at doing it, here:

http://lists.gnu.org/archive/html/emacs-devel/2016-01/msg00695.html

Starting with "I'd prefer something like". Is it at all relevant to what 
you want to see in response?

You haven't replied to the "dired-specific version of this command" part 
of that email either.

Re: [Emacs-diffs] emacs-25 f8208b6: Document the user-level features of the Xref package

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Mon, 11 Jan 2016 00:11:57 +0300
> 
> On 01/10/2016 11:51 PM, Eli Zaretskii wrote:
> 
> > Can you propose an alternative conceptual framework for the user
> > manual, wrt to backends and the features described in this section?
> 
> Before I try answering this, please confirm that you've already read my 
> brief attempt at doing it, here:
> 
> http://lists.gnu.org/archive/html/emacs-devel/2016-01/msg00695.html
> 
> Starting with "I'd prefer something like".

Of course, I read it.

> Is it at all relevant to what you want to see in response?

It talks about stuff way below the level that I'm talking about.  It
describes the details of the implementation, something that doesn't
help at all at the conceptual level.

> You haven't replied to the "dired-specific version of this command" part 
> of that email either.

Any work towards making xref more complete is welcome.  But that's
tangential to the main issue at hand here.

Re: [Emacs-diffs] emacs-25 f8208b6: Document the user-level features of the Xref package

[Dmitry Gutov]

On 01/11/2016 06:33 AM, Eli Zaretskii wrote:

> Any work towards making xref more complete is welcome.  But that's
> tangential to the main issue at hand here.

Not if we can remove the word "Etags" from this sentence: "the rest are 
still (and probably always will be) in Etags, Semantic, etc", at least 
as far as important features are concerned.

(I don't really consider tags-apropos-additional-actions to be one: it 
would be nice to have, but it's not essential, and we don't have to 
mention it in the manual either way).

Re: [Emacs-diffs] emacs-25 f8208b6: Document the user-level features of the Xref package

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Mon, 11 Jan 2016 19:55:50 +0300
> 
> On 01/11/2016 06:33 AM, Eli Zaretskii wrote:
> 
> > Any work towards making xref more complete is welcome.  But that's
> > tangential to the main issue at hand here.
> 
> Not if we can remove the word "Etags" from this sentence: "the rest are 
> still (and probably always will be) in Etags, Semantic, etc", at least 
> as far as important features are concerned.

That comment is tangential to the main issue at point: how to document
the features we were discussing in the user manual.

Re: [Emacs-diffs] emacs-25 f8208b6: Document the user-level features of the Xref package

[Dmitry Gutov]

On 01/10/2016 11:51 PM, Eli Zaretskii wrote:

> Here are the problems I bumped into when I worked on this section:

These don't seem too complex to me. I think I've already raised many of 
the below points (though not the "let's not document" one).

>   . It is impractical to try to ignore the existence of backends: many
>     functions and commands mention them in their names and doc strings,
>     so the fact of their existence is pretty much into the user's face.
>     I couldn't ignore them.

When the xref functions and commands mention the word "backend", they 
refer to xref backends. And I don't suggest we ignore them.

>   . There are only 2 proper backends, but then there are "fallbacks",
>     like symref, Grep, etc. that are used in important commands.  What
>     do we call those "non-backends", and how do we describe them
>     without confusing the heck out of the readers?

Again, they're only used in one command. And the way they're used, is 
unlikely to be extended to other commands.

In the future, we may have either individual backends based on these 
tools, or maybe one aggregated "external tools" backend that will 
dispatch to the appropriate tool, if they provide more or less the same 
functionality.

Now, using the same word, backend, for both actual Xref backends and the 
"external tools" feature we've borrowed from CEDET is my main complaint. 
Even if we only deal with that, I'll be much happier.

First and foremost, do not list Etags, GNU GLOBAL, Cscope, IDUtils and 
Grep together. As much as we want it to be true, Emacs does not really 
provide a "unified user interface to these tools" (but hopefully, will 
sometime).

I see, basically, two options:

- Do not mention the external tools at all here. Only list Etags and 
Emacs Lisp as Xref backends. As a consequence, do not document 
xref-find-references in the manual. Since it doesn't replace any 
particular pre-existing feature, I think we're allowed to do that.

- Only mention the "external tools" in the documentation of 
xref-find-refrerences. Maybe add a reference there to some other node. 
That node might have to be created, and it would list them, and contain 
some other guidance, like "if you're using any of those except Grep, 
refreshing the database is up to you".

Maybe you can choose to simply refer to (info "(semantic) SymRef") 
there, which mentions them (and calls them "symbol reference tools"). I 
think that description is too short, however.

>   . How to describe a bunch of related features, of which only part is
>     implemented by Xref, the rest are still (and probably always will
>     be) in Etags, Semantic, etc.?  How to describe commands whose
>     results could be very different depending on the backend and on the
>     major mode?

The features implemented by Semantic should be described separately--it 
has its own manual. I'm not sure which feature of Semantic in particular 
you feel should be documented with that of Etags.

If you mean completion-at-point, which is enhanced a bit when 
semantic-mode is on, then I don't see why you have to mention it. Yes, 
it might improve accuracy, but the user interface is the same. You can 
add a reference to a node in Semantic's manual that describes the 
benefits it adds to completion-at-point. Or could, if that node existed.

Regarding the features implemented by Etags, some of which are not 
covered by Xref: let's make a list of the essential omissions, and try 
to cover them.

For instance, tags-query-replace has an alternative in 
project[-or-external]-find-regexp, followed by xref-query-replace.

dired-do-search, likewise, could be substituted for a new 
dired-find-regexp command which uses xref's presentation.

dired-do-query-replace-regexp--with using the same command followed by 
xref-query-replace.

I'm not saying they will be perfect, but they *will* essentially cover 
the same functionality.

And maybe someone somewhere will be surprised to know that 
find-grep-dired + dired-do-query-replace-regexp is not the only way to 
perform replacements across all files in a directory.

>   . How to make all of this reasonably future-proof, given that the
>     functionality implemented today covers only a small portion of the
>     turf it was supposed to?

I know it covers a small portion, but why is that a problem? We can 
document some existing xref commands (thus promising to maintain them), 
and we'll get to the rest of the functionality when it arrives.

> Faced with these challenges, I came up with the solution that is now
> before your eyes.  What alternative do you suggest?  Can you present a
> coherent conceptual framework for describing these features, and a
> structure of the section to go with that framework?

I can try.

Re: [Emacs-diffs] emacs-25 f8208b6: Document the user-level features of the Xref package

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Mon, 11 Jan 2016 22:09:46 +0300
> 
> - Do not mention the external tools at all here. Only list Etags and 
> Emacs Lisp as Xref backends. As a consequence, do not document 
> xref-find-references in the manual. Since it doesn't replace any 
> particular pre-existing feature, I think we're allowed to do that.

Doesn't sound right to me.  This is an important command, and a good
addition to the Emacs features.  Leaving it out of the manual just
because we cannot find a good place for it, or a good way to squeeze
it into the chapter, sounds... unwise.  However, ...

> - Only mention the "external tools" in the documentation of 
> xref-find-refrerences. Maybe add a reference there to some other node. 
> That node might have to be created, and it would list them, and contain 
> some other guidance, like "if you're using any of those except Grep, 
> refreshing the database is up to you".

... xref-find-refrerences was the least of my problems when I worked
on this.  It's just one command.

What I needed is a coherent framework that would bring some order to
this group of commands and features.

> > Faced with these challenges, I came up with the solution that is now
> > before your eyes.  What alternative do you suggest?  Can you present a
> > coherent conceptual framework for describing these features, and a
> > structure of the section to go with that framework?
> 
> I can try.

Thanks, that's what I need.  If the concepts and the overall idea are
good, the details will fall in place by themselves.

Re: [Emacs-diffs] emacs-25 f8208b6: Document the user-level features of the Xref package

[Dmitry Gutov]

On 01/11/2016 10:31 PM, Eli Zaretskii wrote:

> Doesn't sound right to me.  This is an important command, and a good
> addition to the Emacs features.  Leaving it out of the manual just
> because we cannot find a good place for it, or a good way to squeeze
> it into the chapter, sounds... unwise.

Also because it relies on a very scarcely documented facility, which 
comes with its own gotchas.

> ... xref-find-refrerences was the least of my problems when I worked
> on this.  It's just one command.

The "external tools" are only used in it, however. And my main complaint 
is against calling them "backends".

> What I needed is a coherent framework that would bring some order to
> this group of commands and features.

You might want to make the requirements more specific.

> Thanks, that's what I need.  If the concepts and the overall idea are
> good, the details will fall in place by themselves.

So, um, what exactly is missing in my previous response? I feel like 
I've answered all questions.

Aside from writing the "structure of the section", that is, because that 
sounds about as easy as rewriting the manual myself.

Re: [Emacs-diffs] emacs-25 f8208b6: Document the user-level features of the Xref package

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Mon, 11 Jan 2016 22:41:18 +0300
> 
> > What I needed is a coherent framework that would bring some order to
> > this group of commands and features.
> 
> You might want to make the requirements more specific.

I thought I did, but I must have failed.

> > Thanks, that's what I need.  If the concepts and the overall idea are
> > good, the details will fall in place by themselves.
> 
> So, um, what exactly is missing in my previous response? I feel like 
> I've answered all questions.
> 
> Aside from writing the "structure of the section", that is, because that 
> sounds about as easy as rewriting the manual myself.

In that case, thanks.  I will have to work this out myself somehow.

Re: [Emacs-diffs] emacs-25 f8208b6: Document the user-level features of the Xref package

[Dmitry Gutov]

On 01/11/2016 11:14 PM, Eli Zaretskii wrote:

>> Aside from writing the "structure of the section", that is, because that
>> sounds about as easy as rewriting the manual myself.
>
> In that case, thanks.  I will have to work this out myself somehow.

Hmm. If that turns out to be a problem, I can try, if you tell me what 
section, and how I can view its current structure.

So far, I've just been commenting on a diff.

Re: [Emacs-diffs] emacs-25 f8208b6: Document the user-level features of the Xref package

[Eli Zaretskii]

I tried to improve the manual according to your comments, please have
a look.  (I suggest to read the Info manual, not look at the diffs.)

Thanks.

Re: [Emacs-diffs] emacs-25 f8208b6: Document the user-level features of the Xref package

[Dmitry Gutov]

On 01/18/2016 08:31 PM, Eli Zaretskii wrote:
> I tried to improve the manual according to your comments, please have

Thank you, Eli. I think it's good now, but see a couple of nits below.

> a look.  (I suggest to read the Info manual, not look at the diffs.)

I did that too, after some usual (for me) flailing around, to find the 
needed node in the manual. The diff turned out to be more useful, 
though, because I was able to quickly see what was removed, and what was 
changed.

The promised nits:

 > If there is a tags table loaded, this command can use it to
+generate completion candidates more intelligently.

That implies that we have some "dumber" completion sources than etags in 
Emacs. I don't think we do, currently. tags-completion-at-point-function 
is the default value of completion-at-point-functions, and we use it as 
the last resort if the major mode (or any minor mode) don't provide any 
specialized completion functions. Then, if no tags table is loaded, we 
don't provide any completions at all, not even stupid ones. I'd suggest 
to simply remove "more intelligently".

 > +  A @dfn{tag} is a synonym for identifier reference.  @xref{Xref}.

Maybe that's technically true, but as employed by Emacs usually, tags 
are elements of TAGS file, generated by the 'etags' program, and they 
only reference identifier definitions. This is the reason we had to 
resort to other means to implement xref-find-references anyway.

And in CEDET, there is a notion of "current tag", which is the function 
definition around point. Not variable reference or the method called at 
point, as one might expect to be the default input for `semantic-symref'.

Call it an "identifier definition reference", maybe? Or "identifier 
definition", or "definition reference".

Re: [Emacs-diffs] emacs-25 f8208b6: Document the user-level features of the Xref package

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Tue, 19 Jan 2016 01:18:20 +0300
> 
> On 01/18/2016 08:31 PM, Eli Zaretskii wrote:
> > I tried to improve the manual according to your comments, please have
> 
> Thank you, Eli. I think it's good now, but see a couple of nits below.
> 
> > a look.  (I suggest to read the Info manual, not look at the diffs.)
> 
> I did that too, after some usual (for me) flailing around, to find the 
> needed node in the manual. The diff turned out to be more useful, 
> though, because I was able to quickly see what was removed, and what was 
> changed.

Some of the discussion we had before was about the structure of this
section.  You cannot see the structure from the diffs, which is why I
suggested to read the formatted manual.

>  > If there is a tags table loaded, this command can use it to
> +generate completion candidates more intelligently.
> 
> That implies that we have some "dumber" completion sources than etags in 
> Emacs. I don't think we do, currently. tags-completion-at-point-function 
> is the default value of completion-at-point-functions, and we use it as 
> the last resort if the major mode (or any minor mode) don't provide any 
> specialized completion functions. Then, if no tags table is loaded, we 
> don't provide any completions at all, not even stupid ones. I'd suggest 
> to simply remove "more intelligently".

Thanks, I tweaked that text some more.

>  > +  A @dfn{tag} is a synonym for identifier reference.  @xref{Xref}.
> 
> Maybe that's technically true, but as employed by Emacs usually, tags 
> are elements of TAGS file, generated by the 'etags' program, and they 
> only reference identifier definitions.

The manual in its previous form said:

  A "tag" is a reference to a subunit in a program or in a document.

This is now replaced with

  An "identifier" is a name of a syntactical subunit of the program...

So in this regard, a "tag" is indeed a synonym for "identifier".
There's nothing that can prevent recording references in tags table
format.

In any case, changing terminology in that subsection means a serious
rewrite of it in its entirety.  I see no reason for such an effort.  I
moved the offending sentence to a footnote (with some explanation why
we change terminology in this subsection), so people who already know
what "tag" is will need not stumble upon this.

Thanks.

Re: [Emacs-diffs] emacs-25 f8208b6: Document the user-level features of the Xref package

[Dmitry Gutov]

On 01/19/2016 08:41 PM, Eli Zaretskii wrote:

> Some of the discussion we had before was about the structure of this
> section.  You cannot see the structure from the diffs, which is why I
> suggested to read the formatted manual.

Yes, but I didn't offer any particular opinion on that. It seems good 
enough, although I'd remove, or move out, the references to etags 
commands: the section is named Xref, tags-* commands are not part of it.

With the two new additions, we could feel more confident doing that.

There's no direct counterpart to tags-search, but I think it can be 
adequately replaced with dired-do-find-regexp or 
project-or-external-find-regexp.

> In any case, changing terminology in that subsection means a serious
> rewrite of it in its entirety.  I see no reason for such an effort.  I
> moved the offending sentence to a footnote (with some explanation why
> we change terminology in this subsection), so people who already know
> what "tag" is will need not stumble upon this.

Fair enough. Thanks for considering it.

Re: [Emacs-diffs] emacs-25 f8208b6: Document the user-level features of the Xref package

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Wed, 20 Jan 2016 00:53:19 +0300
> 
> On 01/19/2016 08:41 PM, Eli Zaretskii wrote:
> 
> > Some of the discussion we had before was about the structure of this
> > section.  You cannot see the structure from the diffs, which is why I
> > suggested to read the formatted manual.
> 
> Yes, but I didn't offer any particular opinion on that. It seems good 
> enough, although I'd remove, or move out, the references to etags 
> commands: the section is named Xref, tags-* commands are not part of it.

"Xref" is the name of the node, not of the section.  And the node's
name does not mean it describes xref the package; this is user-level
documentation.  If you have a better suggestion for a short name of a
node which aims at describing features most of which have "xref-" in
their names, please tell.

> With the two new additions, we could feel more confident doing that.

AFAICT, your additions are not directly related to this section; they
are not mentioned in it.

> There's no direct counterpart to tags-search, but I think it can be 
> adequately replaced with dired-do-find-regexp or 
> project-or-external-find-regexp.

No, dired-do-find-regexp cannot replace it, because it looks through
all the files in the directory, whereas tags-search looks in the files
recorded in the tags tables.  We should have an xref-based replacement
for tags-search and tags-query-replace, which would similarly search
through the "relevant" files.  These are currently the only 2
etags-based commands in that section that directly invoke etags.el.
Once we solve that problem (and I think we should do this before v25.1
is out), there will be no reason for splitting the description more
than it already is (the tags tables and how to build them are already
described in a separate subsection).

(Btw, I think replacing tags-search and tags-query-replace with
commands that start with "project-" is not a good idea, for the same
reason that having "tags-*" commands there is.  They should be
"xref-*" commands.)

Re: [Emacs-diffs] emacs-25 f8208b6: Document the user-level features of the Xref package

[John Wiegley]

>>>>> Eli Zaretskii <eliz@gnu.org> writes:

> No, dired-do-find-regexp cannot replace it, because it looks through all the
> files in the directory, whereas tags-search looks in the files recorded in
> the tags tables. We should have an xref-based replacement for tags-search
> and tags-query-replace, which would similarly search through the "relevant"
> files. These are currently the only 2 etags-based commands in that section
> that directly invoke etags.el. Once we solve that problem (and I think we
> should do this before v25.1 is out), there will be no reason for splitting
> the description more than it already is (the tags tables and how to build
> them are already described in a separate subsection).

Thanks, Eli, you wrote exactly what I had wanted to after reading that
comment. Yes, and yes.

> (Btw, I think replacing tags-search and tags-query-replace with commands
> that start with "project-" is not a good idea, for the same reason that
> having "tags-*" commands there is. They should be "xref-*" commands.)

And yes.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: [Emacs-diffs] emacs-25 f8208b6: Document the user-level features of the Xref package

[Dmitry Gutov]

On 01/20/2016 10:28 AM, John Wiegley wrote:

>> (Btw, I think replacing tags-search and tags-query-replace with commands
>> that start with "project-" is not a good idea, for the same reason that
>> having "tags-*" commands there is. They should be "xref-*" commands.)
>
> And yes.

Are you both proposing that we rename all project- commands to xref-? 
Despite the fact that the former act on the current project, using 
simple operations (Grep files or list files), and the existing xref- 
commands all use data provided by the current xref backend?

The reason I want to replace the tags- commands is not really because of 
their names, but because they use a different UI, and because they keep 
tags-loop-continue relevant.

Re: [Emacs-diffs] emacs-25 f8208b6: Document the user-level features of the Xref package

[John Wiegley]

>>>>> Dmitry Gutov <dgutov@yandex.ru> writes:

> Are you both proposing that we rename all project- commands to xref-?
> Despite the fact that the former act on the current project, using simple
> operations (Grep files or list files), and the existing xref- commands all
> use data provided by the current xref backend?

Think of a user who isn't aware of the distinction between project and xref.
Which command would make the most sense to them at the M-x prompt?

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: [Emacs-diffs] emacs-25 f8208b6: Document the user-level features of the Xref package

[Dmitry Gutov]

On 01/21/2016 12:03 AM, John Wiegley wrote:

> Think of a user who isn't aware of the distinction between project and xref.
> Which command would make the most sense to them at the M-x prompt?

project-*? That word at least means something, for a random new user.

Re: [Emacs-diffs] emacs-25 f8208b6: Document the user-level features of the Xref package

[Stephen Leake]

John Wiegley <jwiegley@gmail.com> writes:

>>>>>> Dmitry Gutov <dgutov@yandex.ru> writes:
>
>> Are you both proposing that we rename all project- commands to xref-?
>> Despite the fact that the former act on the current project, using simple
>> operations (Grep files or list files), and the existing xref- commands all
>> use data provided by the current xref backend?
>
> Think of a user who isn't aware of the distinction between project and xref.
> Which command would make the most sense to them at the M-x prompt?

First response is; tell them to read the relevant manual sections, this
stuff is _not_ trivial.

Second response; that depends on what they want to accomplish.

If they want to perform a find-and-replace of a string on all the files
in the current project, then "project" makes the most sense.

If they've just used xref to produce a list of references, and now want
to perform a find-and-replace on those, "xref" makes the most sense.

There is a confusion between xref and project; xref must operate on some
set of files, and one job of project is provide a set of files. But
xref does not rely solely on project for the set, hence the confusion.

I think we should move more towards relying on "project" commands for
operations on sets of files; xref is then mostly a tool that project
commands use.

Another way to look at this is that any xref backend should also provide
a project backend, so the two sets of commands always operate on the
same set of files.

--
-- Stephe

Re: [Emacs-diffs] emacs-25 f8208b6: Document the user-level features of the Xref package

[Dmitry Gutov]

On 01/21/2016 10:01 PM, Stephen Leake wrote:

> I think we should move more towards relying on "project" commands for
> operations on sets of files; xref is then mostly a tool that project
> commands use.
>
> Another way to look at this is that any xref backend should also provide
> a project backend, so the two sets of commands always operate on the
> same set of files.

I would state it like this: most projects that provide an xref backend, 
should provide a project backend as well, or at least set 
project-vc-external-roots-function. The reverse is probably also true.

The main exceptions are simplistic backends, for languages without 
strong tooling.

I thought about having xref.el depend on project.el (by, say, passing 
the current project to xref-backend-functions), but I'm less confident 
in the user-facing parts of project.el than in xref commands.

So it would make a more-stable facility depend on a less-stable one, 
unfortunately.

Re: [Emacs-diffs] emacs-25 f8208b6: Document the user-level features of the Xref package

[Eli Zaretskii]

> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Wed, 20 Jan 2016 23:53:24 +0300
> 
> On 01/20/2016 10:28 AM, John Wiegley wrote:
> 
> >> (Btw, I think replacing tags-search and tags-query-replace with commands
> >> that start with "project-" is not a good idea, for the same reason that
> >> having "tags-*" commands there is. They should be "xref-*" commands.)
> >
> > And yes.
> 
> Are you both proposing that we rename all project- commands to xref-? 

No, of course not.  Rather, I'm saying that commands which replace
these specific two tags-* commands with similar functionality that
presents an xref UI are better called xref-SOMETHING.

> The reason I want to replace the tags- commands is not really because of 
> their names, but because they use a different UI, and because they keep 
> tags-loop-continue relevant.

No argument here.

Re: [Emacs-diffs] emacs-25 f8208b6: Document the user-level features of the Xref package

[Dmitry Gutov]

On 01/20/2016 07:43 AM, Eli Zaretskii wrote:

> "Xref" is the name of the node, not of the section.  And the node's
> name does not mean it describes xref the package; this is user-level
> documentation.  If you have a better suggestion for a short name of a
> node which aims at describing features most of which have "xref-" in
> their names, please tell.

Cross-Referencing? A native speaker might have a better suggestion.

> AFAICT, your additions are not directly related to this section; they
> are not mentioned in it.

Hmm. When I asked what we needed to obsolete tags-loop-continue, you 
only mentioned Dired commands.

http://lists.gnu.org/archive/html/emacs-devel/2016-01/msg00842.html

> No, dired-do-find-regexp cannot replace it, because it looks through
> all the files in the directory, whereas tags-search looks in the files
> recorded in the tags tables.

Ok, dired-do-find-regexp cannot. What about project-or-external-find-regexp?

By default, when the VC project backend is used, and 
project-vc-external-roots-function is at its default value, 
project-or-external-find-regexp will search the current project, as well 
as the directories of the tags tables, if they are outside of it.

Or, in emacs-lisp-mode, it will search the load-path directories.

> We should have an xref-based replacement
> for tags-search and tags-query-replace, which would similarly search
> through the "relevant" files.

tags-query-replace? We have xref-query-replace already, and it's more 
flexible: depending on which results it's called from, it'll perform 
replacements in simple regexp matches, or in "smarter" list of 
references (or other similar kinds of results).

Back to tags-search.

Like described above, we have a command that goes further that it 
already. So what would be the goal of having xref-find-regexp? Just to 
tick off a box and simplify updating the manual?

It will require adding a new method to xref backends, with clear 
semantics and purpose. Backend authors will want to know why they 
implement it, and why a user might want to use xref-find-regexp, instead 
of the aforementioned alternative.

Suppose we're in emacs-lisp-mode. What will xref-find-regexp do? Will it 
search the load-path elements, but skip the current project, however 
it's defined, if it's not in load-path? Will it only search *.el files 
inside load-path directories, and ignore files with all other extensions?

> (Btw, I think replacing tags-search and tags-query-replace with
> commands that start with "project-" is not a good idea, for the same

I definitely never suggested replacing tags-query-replace with project- 
anything. The new "replace" workflow is two-step: two the search (using 
some command that creates an xref buffer), then call xref-query-replace.

Re: [Emacs-diffs] emacs-25 f8208b6: Document the user-level features of the Xref package

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Thu, 21 Jan 2016 06:28:51 +0300
> 
> On 01/20/2016 07:43 AM, Eli Zaretskii wrote:
> 
> > "Xref" is the name of the node, not of the section.  And the node's
> > name does not mean it describes xref the package; this is user-level
> > documentation.  If you have a better suggestion for a short name of a
> > node which aims at describing features most of which have "xref-" in
> > their names, please tell.
> 
> Cross-Referencing? A native speaker might have a better suggestion.

Cross-Referencing doesn't fit, IMO, not if you consider the user-level
functionality.  I've tried to find a name that somehow expressed the
functionality, but came up empty-handed.  "Xref" is a compromise: it's
not a word that means something, it's just the name of the package
(but then so was "Tags").

> > AFAICT, your additions are not directly related to this section; they
> > are not mentioned in it.
> 
> Hmm. When I asked what we needed to obsolete tags-loop-continue, you 
> only mentioned Dired commands.
> 
> http://lists.gnu.org/archive/html/emacs-devel/2016-01/msg00842.html

Yes.  But I think it would be good to replace these last two as well.

> > No, dired-do-find-regexp cannot replace it, because it looks through
> > all the files in the directory, whereas tags-search looks in the files
> > recorded in the tags tables.
> 
> Ok, dired-do-find-regexp cannot. What about project-or-external-find-regexp?

It's close.  But AFAIU it requires something called "roots", which is
an additional input.  What I had in mind was just what tags-search
does, but with the xref-style UI, i.e. via the *xref* buffer showing
the matches.  Same as you did in Dired, but looking in files returned
by the backend -- etags will return the files in TAGS, elisp will
return the files in load-path (or wherever else it gets the files),
etc.

> By default, when the VC project backend is used, and 
> project-vc-external-roots-function is at its default value, 
> project-or-external-find-regexp will search the current project, as well 
> as the directories of the tags tables, if they are outside of it.
> 
> Or, in emacs-lisp-mode, it will search the load-path directories.

So how do the "roots", whatever that is, come into play?  IOW, why is
this command in project.el and not in xref.el?  I suppose the reason
is somehow related to the purpose of project.el, which is different
from xref.el.

> > We should have an xref-based replacement
> > for tags-search and tags-query-replace, which would similarly search
> > through the "relevant" files.
> 
> tags-query-replace? We have xref-query-replace already

xref-query-replace is (confusingly) different: it makes replacements
in the names of the matching identifiers, not in matches found in the
files returned by the backend.

> Back to tags-search.
> 
> Like described above, we have a command that goes further that it 
> already. So what would be the goal of having xref-find-regexp? Just to 
> tick off a box and simplify updating the manual?

If project-or-external-find-regexp is exactly equivalent, then perhaps
a simple alias would be enough.

And don't underestimate the power of consistent names and of array of
commands whose names and descriptions in the manual make sense.
Confusing documentation makes it much harder to grasp the features and
their internal logic.

> Suppose we're in emacs-lisp-mode. What will xref-find-regexp do? Will it 
> search the load-path elements, but skip the current project, however 
> it's defined, if it's not in load-path? Will it only search *.el files 
> inside load-path directories, and ignore files with all other extensions?

Why don't those questions arise when we invoke xref-find-references?
How do you know in what files to search then?  I think the same answer
will do for the replacements of tags-search and tags-query-replace.

> > (Btw, I think replacing tags-search and tags-query-replace with
> > commands that start with "project-" is not a good idea, for the same
> 
> I definitely never suggested replacing tags-query-replace with project- 
> anything. The new "replace" workflow is two-step: two the search (using 
> some command that creates an xref buffer), then call xref-query-replace.

xref-query-replace does this:

  This command interactively replaces FROM with TO in the names of the
  references displayed in the current *xref* buffer.

That's not what tags-query-replace does.

Re: [Emacs-diffs] emacs-25 f8208b6: Document the user-level features of the Xref package

[Dmitry Gutov]

On 01/21/2016 08:29 PM, Eli Zaretskii wrote:

> Cross-Referencing doesn't fit, IMO, not if you consider the user-level
> functionality.  I've tried to find a name that somehow expressed the
> functionality, but came up empty-handed.  "Xref" is a compromise: it's
> not a word that means something, it's just the name of the package
> (but then so was "Tags").

I don't know if that's true. "Tags" is a more meaningful word. You even 
define it in the manual. We don't define the word "xref".

> What I had in mind was just what tags-search
> does, but with the xref-style UI, i.e. via the *xref* buffer showing
> the matches.  Same as you did in Dired, but looking in files returned
> by the backend -- etags will return the files in TAGS, elisp will
> return the files in load-path (or wherever else it gets the files),
> etc.

Backends don't return files. They return lists of references. Which 
could be "definitions", "references", "apropos matches"... and, I 
suppose, "regexp matches", if we really want to.

Initially, I created the xref-find-regexp command, but then moved it to 
the project package, because always including the current project root 
turned out to be more useful, to me.

> So how do the "roots", whatever that is, come into play?  IOW, why is
> this command in project.el and not in xref.el?

The what and why of "project roots" are described inside project.el.

As a practical example, xref-find-regexp, with etags backend, in the 
Emacs project, would only search the files inside src, lisp and lwlib, 
whereas project-find-regexp searches inside the whole emacs/ directory.

And project-or-external-find-regexp would search outside of it as well, 
if any of the currently used TAGS files resided outside of emacs/. Each 
such TAGS file would create an "external root" to be searched.

And if you were to use xref-find-regexp to search for occurrences of 
`tags-loop-continue', it would only find them in the sources. Whereas 
both project- commands would also find its occurrences inside 
doc/**/*.texi, etc/NEWS*, top-level ChangeLog files and inside test/.

That looks distinctly more useful to me. Especially note how tags-search 
skips the test/ directory.

> I suppose the reason
> is somehow related to the purpose of project.el, which is different
> from xref.el.

Yes. Projects are groups of different kind of files.

Xref deals with information about source code: where a function was 
defined, where it was referenced, and which functions we have.

> xref-query-replace is (confusingly) different: it makes replacements
> in the names of the matching identifiers, not in matches found in the
> files returned by the backend.

The fact that we had tags-query-replace doing things a certain way, 
doesn't mean that the replacement should work exactly the same.

We can give it a different name, to appraise the user of the difference. 
How about xref-query-replace-in-matches?

> If project-or-external-find-regexp is exactly equivalent, then perhaps
> a simple alias would be enough.

That would've been too easy.

> And don't underestimate the power of consistent names and of array of
> commands whose names and descriptions in the manual make sense.
> Confusing documentation makes it much harder to grasp the features and
> their internal logic.

Sure. But that a matter of proper naming and documentation. That doesn't 
necessarily imply making the new commands do exactly what the previous 
ones did.

>> Suppose we're in emacs-lisp-mode. What will xref-find-regexp do? Will it
>> search the load-path elements, but skip the current project, however
>> it's defined, if it's not in load-path? Will it only search *.el files
>> inside load-path directories, and ignore files with all other extensions?
>
> Why don't those questions arise when we invoke xref-find-references?
> How do you know in what files to search then?  I think the same answer
> will do for the replacements of tags-search and tags-query-replace.

For xref-find-references, we generally leave it up to the backend. But 
that query is easier to make sense of: we usually only look for 
references in source files, so it can disregard the rest.

When doing a regexp search, we expect to see the matches in all kinds of 
related files, including README at the top-level of the project. At 
least I do. xref-find-regexp, as I imagine you want it to be defined, 
doesn't fit any of my usual workflows. Hence the questions.

> xref-query-replace does this:
>
>    This command interactively replaces FROM with TO in the names of the
>    references displayed in the current *xref* buffer.
>
> That's not what tags-query-replace does.

I never suggested making tags-query-replace an "obsolete alias" to 
xref-query-replace.

Re: [Emacs-diffs] emacs-25 f8208b6: Document the user-level features of the Xref package

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Thu, 21 Jan 2016 21:46:19 +0300
> 
> On 01/21/2016 08:29 PM, Eli Zaretskii wrote:
> 
> > Cross-Referencing doesn't fit, IMO, not if you consider the user-level
> > functionality.  I've tried to find a name that somehow expressed the
> > functionality, but came up empty-handed.  "Xref" is a compromise: it's
> > not a word that means something, it's just the name of the package
> > (but then so was "Tags").
> 
> I don't know if that's true. "Tags" is a more meaningful word. You even 
> define it in the manual.

That's just history: someone invented a definition that would make
sense.

> We don't define the word "xref".

We could, if we wanted to.  I think for now just pretending the name
has no semantic meaning is good enough.

> > What I had in mind was just what tags-search
> > does, but with the xref-style UI, i.e. via the *xref* buffer showing
> > the matches.  Same as you did in Dired, but looking in files returned
> > by the backend -- etags will return the files in TAGS, elisp will
> > return the files in load-path (or wherever else it gets the files),
> > etc.
> 
> Backends don't return files. They return lists of references. Which 
> could be "definitions", "references", "apropos matches"... and, I 
> suppose, "regexp matches", if we really want to.

But a reference includes a file name, doesn't it?  If so, why not
search through those files?

> Initially, I created the xref-find-regexp command, but then moved it to 
> the project package, because always including the current project root 
> turned out to be more useful, to me.

It probably is for some use cases.  But in others, users might want
the other kind.

> As a practical example, xref-find-regexp, with etags backend, in the 
> Emacs project, would only search the files inside src, lisp and lwlib, 
> whereas project-find-regexp searches inside the whole emacs/ directory.
> 
> And project-or-external-find-regexp would search outside of it as well, 
> if any of the currently used TAGS files resided outside of emacs/. Each 
> such TAGS file would create an "external root" to be searched.

That's what I thought.  So a command that would only search what's in
TAGS does have a place, IMO.  It would also be a good replacement for
tags-search.

> And if you were to use xref-find-regexp to search for occurrences of 
> `tags-loop-continue', it would only find them in the sources. Whereas 
> both project- commands would also find its occurrences inside 
> doc/**/*.texi, etc/NEWS*, top-level ChangeLog files and inside test/.
> 
> That looks distinctly more useful to me. Especially note how tags-search 
> skips the test/ directory.

I think both use cases are legitimate.

> How about xref-query-replace-in-matches?

Fine with me.

> >> Suppose we're in emacs-lisp-mode. What will xref-find-regexp do? Will it
> >> search the load-path elements, but skip the current project, however
> >> it's defined, if it's not in load-path? Will it only search *.el files
> >> inside load-path directories, and ignore files with all other extensions?
> >
> > Why don't those questions arise when we invoke xref-find-references?
> > How do you know in what files to search then?  I think the same answer
> > will do for the replacements of tags-search and tags-query-replace.
> 
> For xref-find-references, we generally leave it up to the backend. But 
> that query is easier to make sense of: we usually only look for 
> references in source files, so it can disregard the rest.

It would be OK for xref-find-regexp to do the same.  Users who want
the project.el way of doing that can invoke the commands there.  (They
should probably be documented in the manual.)

Re: [Emacs-diffs] emacs-25 f8208b6: Document the user-level features of the Xref package

[Dmitry Gutov]

On 01/21/2016 10:00 PM, Eli Zaretskii wrote:

> But a reference includes a file name, doesn't it?  If so, why not
> search through those files?

My core objection is that "find regexp" is not a semantic query. Whereas 
all current xref queries are semantic (i.e. require language analysis 
for best results).

So really want some clear justification, rather than "etags has command 
that does this", and "a user could want this". Without the 
justification, it would be hard to document the new command anyway.

We _couldn't_ just copy "Search through all files listed in tags table" 
from tags-search docstring.

> It probably is for some use cases.  But in others, users might want
> the other kind.

Such as?

>> How about xref-query-replace-in-matches?
>
> Fine with me.

You'd agree that tags-query-replace doesn't require a direct replacement 
then?

> It would be OK for xref-find-regexp to do the same.  Users who want
> the project.el way of doing that can invoke the commands there.  (They
> should probably be documented in the manual.)

We should worry that a user might invoke xref-find-regexp while 
expecting it to behave like project-find-regexp.

Because in most other editors regexp search works more like the latter: 
it's either in the current file, or across the current project.

Re: [Emacs-diffs] emacs-25 f8208b6: Document the user-level features of the Xref package

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Thu, 21 Jan 2016 22:46:03 +0300
> 
> On 01/21/2016 10:00 PM, Eli Zaretskii wrote:
> 
> > But a reference includes a file name, doesn't it?  If so, why not
> > search through those files?
> 
> My core objection is that "find regexp" is not a semantic query. Whereas 
> all current xref queries are semantic (i.e. require language analysis 
> for best results).
> 
> So really want some clear justification, rather than "etags has command 
> that does this", and "a user could want this". Without the 
> justification, it would be hard to document the new command anyway.

Same justification as for Dired's command you just coded.  Except that
in this case, I want to search and/or replace in files that belong to
some coherent group that is not a directory hierarchy.

> >> How about xref-query-replace-in-matches?
> >
> > Fine with me.
> 
> You'd agree that tags-query-replace doesn't require a direct replacement 
> then?

I thought that was what xref-query-replace-in-matches will do.

> > It would be OK for xref-find-regexp to do the same.  Users who want
> > the project.el way of doing that can invoke the commands there.  (They
> > should probably be documented in the manual.)
> 
> We should worry that a user might invoke xref-find-regexp while 
> expecting it to behave like project-find-regexp.

I don't.

> Because in most other editors regexp search works more like the latter: 
> it's either in the current file, or across the current project.

I'm not lobbying for supplanting the project.el command.  It has its
uses.

Re: [Emacs-diffs] emacs-25 f8208b6: Document the user-level features of the Xref package

[Dmitry Gutov]

On 01/21/2016 11:02 PM, Eli Zaretskii wrote:

>> So really want some clear justification, rather than "etags has command
>> that does this", and "a user could want this". Without the
>> justification, it would be hard to document the new command anyway.
>
> Same justification as for Dired's command you just coded.  Except that
> in this case, I want to search and/or replace in files that belong to
> some coherent group that is not a directory hierarchy.

Dired's command makes sense: when there's no project, or we want to 
search some dirs specifically, we can fire up Dired, select the 
directories and files one by one, and do the search.

When you call xref-find-regexp, you won't get to tell exactly what you 
want. So, the command needs to have some useful, clear semantics.

>>>> How about xref-query-replace-in-matches?
>>>
>>> Fine with me.
>>
>> You'd agree that tags-query-replace doesn't require a direct replacement
>> then?
>
> I thought that was what xref-query-replace-in-matches will do.

xref-query-replace-in-matches will be the new name for the current 
xref-query-replace. The changed name should signify that it replaces, 
indeed, in the already-present list of matches.

Since you said that having xref-query-replace called almost the same as 
tags-query-replace is confusing.

Re: [Emacs-diffs] emacs-25 f8208b6: Document the user-level features of the Xref package

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Thu, 21 Jan 2016 23:10:52 +0300
> 
> On 01/21/2016 11:02 PM, Eli Zaretskii wrote:
> 
> >> So really want some clear justification, rather than "etags has command
> >> that does this", and "a user could want this". Without the
> >> justification, it would be hard to document the new command anyway.
> >
> > Same justification as for Dired's command you just coded.  Except that
> > in this case, I want to search and/or replace in files that belong to
> > some coherent group that is not a directory hierarchy.
> 
> Dired's command makes sense: when there's no project, or we want to 
> search some dirs specifically, we can fire up Dired, select the 
> directories and files one by one, and do the search.

The same with a backend: it selects the files to search.

> When you call xref-find-regexp, you won't get to tell exactly what you 
> want. So, the command needs to have some useful, clear semantics.

It does.  With the etags backend, the TAGS file tells it to, and the
user controls what goes into that file.  I presume other backends
could do similar stuff.

> >>>> How about xref-query-replace-in-matches?
> >>>
> >>> Fine with me.
> >>
> >> You'd agree that tags-query-replace doesn't require a direct replacement
> >> then?
> >
> > I thought that was what xref-query-replace-in-matches will do.
> 
> xref-query-replace-in-matches will be the new name for the current 
> xref-query-replace. The changed name should signify that it replaces, 
> indeed, in the already-present list of matches.
> 
> Since you said that having xref-query-replace called almost the same as 
> tags-query-replace is confusing.

Ah, okay.  That'd be good, but we still should try to find a good xref
replacement for tags-query-replace, IMO.

Re: [Emacs-diffs] emacs-25 f8208b6: Document the user-level features of the Xref package

[Dmitry Gutov]

On 01/21/2016 11:32 PM, Eli Zaretskii wrote:

>> Dired's command makes sense: when there's no project, or we want to
>> search some dirs specifically, we can fire up Dired, select the
>> directories and files one by one, and do the search.
>
> The same with a backend: it selects the files to search.

We wouldn't want to use a command that searches an arbitrary set of 
files, which depends only on the mood of the person who wrote a 
particular backend. Stricter semantics are needed.

Furhermore, Dired's commands we're replacing had default bindings. 
tags-search only has a menu entry.

>> When you call xref-find-regexp, you won't get to tell exactly what you
>> want. So, the command needs to have some useful, clear semantics.
>
> It does.  With the etags backend, the TAGS file tells it to, and the
> user controls what goes into that file.  I presume other backends
> could do similar stuff.

That doesn't look general enough. What do I write in the docstring? 
"Searches where the TAGS files tell it to, or does similar stuff"?

>> xref-query-replace-in-matches will be the new name for the current
>> xref-query-replace. The changed name should signify that it replaces,
>> indeed, in the already-present list of matches.
>>
>> Since you said that having xref-query-replace called almost the same as
>> tags-query-replace is confusing.
>
> Ah, okay.  That'd be good, but we still should try to find a good xref
> replacement for tags-query-replace, IMO.

We should wait for more opinions, then. It won't be hard to add a 
command like that later.

Re: [Emacs-diffs] emacs-25 f8208b6: Document the user-level features of the Xref package

[Dmitry Gutov]

On 01/21/2016 11:32 PM, Eli Zaretskii wrote:

>> xref-query-replace-in-matches will be the new name for the current
>> xref-query-replace. The changed name should signify that it replaces,
>> indeed, in the already-present list of matches.
>>
>> Since you said that having xref-query-replace called almost the same as
>> tags-query-replace is confusing.
>
> Ah, okay.  That'd be good,

It's called xref-query-replace-in-results now. Less ambiguous.

Re: [Emacs-diffs] emacs-25 f8208b6: Document the user-level features of the Xref package

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sun, 24 Jan 2016 05:20:20 +0300
> 
> On 01/21/2016 11:32 PM, Eli Zaretskii wrote:
> 
> >> xref-query-replace-in-matches will be the new name for the current
> >> xref-query-replace. The changed name should signify that it replaces,
> >> indeed, in the already-present list of matches.
> >>
> >> Since you said that having xref-query-replace called almost the same as
> >> tags-query-replace is confusing.
> >
> > Ah, okay.  That'd be good,
> 
> It's called xref-query-replace-in-results now. Less ambiguous.

Thanks.

Re: [Emacs-diffs] emacs-25 f8208b6: Document the user-level features of the Xref package

[Ken Brown]

I have a couple of suggestions, based on my experience when I tried to 
use Xref for the first time.

First, paragraph b of the Xref info node talks about etags, with a 
reference to the subnode on creating tags tables.  It would have helped 
me if there had been a further sentence saying that a user has to give 
the command M-x visit-tags-table before Xref will use etags as the 
backend.  This sentence would be accompanied by a reference to the 
subnode on selecting a tags table.  That node is currently listed in the 
Menu, but it's not obvious (or at least wasn't obvious to me) what the 
relevance was, especially since the description of that node refers to 
*visiting* a tags table rather than telling Xref to using one.

Second, I suggest that Xref should offer to use etags as the backend 
when a TAGS file exists in the current directory.

I had the following experience in my first attempt to use Xref:

After reading the Xref node and glancing at the "Creating Tags Tables" 
node, I remembered that there was a 'tags' target in the top level 
Makefile of the emacs tree, so I gave the command 'make tags' to create 
the TAGS files.  Then I visited a C file in the src directory and tried 
'M-.' on an identifier.  The result was the error message 'Symbol’s 
function definition is void: etags--xref-backend'.  I didn't know what 
to do with that message, so I went back to the Xref node to see if it 
would tell me how to tell Xref to use etags as backend.  It didn't, but 
I finally tried M-x visit-tags-table, and it worked.

Ken

Re: [Emacs-diffs] emacs-25 f8208b6: Document the user-level features of the Xref package

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Ken Brown <kbrown@cornell.edu>
> Date: Mon, 25 Jan 2016 15:38:58 -0500
> 
> First, paragraph b of the Xref info node talks about etags, with a 
> reference to the subnode on creating tags tables.  It would have helped 
> me if there had been a further sentence saying that a user has to give 
> the command M-x visit-tags-table before Xref will use etags as the 
> backend.

That's not true, you don't need to invoke visit-tags-table manually.
Xref does that automatically (it asks you for confirmation).
Something isn't working correctly in your case.

> Second, I suggest that Xref should offer to use etags as the backend 
> when a TAGS file exists in the current directory.

It already does, at least in C mode.

> I had the following experience in my first attempt to use Xref:
> 
> After reading the Xref node and glancing at the "Creating Tags Tables" 
> node, I remembered that there was a 'tags' target in the top level 
> Makefile of the emacs tree, so I gave the command 'make tags' to create 
> the TAGS files.  Then I visited a C file in the src directory and tried 
> 'M-.' on an identifier.  The result was the error message 'Symbol’s 
> function definition is void: etags--xref-backend'.  I didn't know what 
> to do with that message, so I went back to the Xref node to see if it 
> would tell me how to tell Xref to use etags as backend.  It didn't, but 
> I finally tried M-x visit-tags-table, and it worked.

Very strange.  Do you have a TAGS file in the src directory?  You
should if you invoked "make TAGS" at top level.

If I visit some C file in src/ and type M-. I'm asked to confirm to
visit tags table in src, and then I'm presented with the definition of
the identifier (or with an *xref* buffer if there are more than one
candidate).  So there's some factor at work in your case that somehow
prevents this from working correctly.

Did you try the same in a .el file?  What happened?

Re: [Emacs-diffs] emacs-25 f8208b6: Document the user-level features of the Xref package

[Dmitry Gutov]

On 01/25/2016 11:38 PM, Ken Brown wrote:

> After reading the Xref node and glancing at the "Creating Tags Tables"
> node, I remembered that there was a 'tags' target in the top level
> Makefile of the emacs tree, so I gave the command 'make tags' to create
> the TAGS files.  Then I visited a C file in the src directory and tried
> 'M-.' on an identifier.  The result was the error message 'Symbol’s
> function definition is void: etags--xref-backend'.

That sounds like a bug. Did you try it in the current version in the 
branch emacs-25? I fixed something like that not too long ago (though 
more than a week ago, for sure).

If it's still there, please M-x report-emacs-bug with the exact steps.

Re: [Emacs-diffs] emacs-25 f8208b6: Document the user-level features of the Xref package

[Ken Brown]

On 1/25/2016 4:29 PM, Dmitry Gutov wrote:
> On 01/25/2016 11:38 PM, Ken Brown wrote:
>
>> After reading the Xref node and glancing at the "Creating Tags Tables"
>> node, I remembered that there was a 'tags' target in the top level
>> Makefile of the emacs tree, so I gave the command 'make tags' to create
>> the TAGS files.  Then I visited a C file in the src directory and tried
>> 'M-.' on an identifier.  The result was the error message 'Symbol’s
>> function definition is void: etags--xref-backend'.
>
> That sounds like a bug. Did you try it in the current version in the
> branch emacs-25? I fixed something like that not too long ago (though
> more than a week ago, for sure).

I just pulled and rebuilt, and the problem is gone.

Thanks.

Ken

Re: [Emacs-diffs] emacs-25 f8208b6: Document the user-level features of the Xref package

[Dmitry Gutov]

On 01/26/2016 05:44 AM, Ken Brown wrote:

> I just pulled and rebuilt, and the problem is gone.
>
> Thanks.

Thanks for checking.

Re: [Emacs-diffs] emacs-25 f8208b6: Document the user-level features of the Xref package

[Stephen Leake]

Eli Zaretskii <eliz@gnu.org> writes:

>> Cc: emacs-devel@gnu.org
>> From: Dmitry Gutov <dgutov@yandex.ru>
>> Date: Thu, 21 Jan 2016 06:28:51 +0300
>> 
>> On 01/20/2016 07:43 AM, Eli Zaretskii wrote:
>> 
>> > "Xref" is the name of the node, not of the section.  And the node's
>> > name does not mean it describes xref the package; this is user-level
>> > documentation.  If you have a better suggestion for a short name of a
>> > node which aims at describing features most of which have "xref-" in
>> > their names, please tell.
>> 
>> Cross-Referencing? A native speaker might have a better suggestion.
>
> Cross-Referencing doesn't fit, IMO, not if you consider the user-level
> functionality.  

"Cross-reference" was certainly the initial intent of the xref package.

If that name no longer fits, it means there is functionality in
xref that doesn't belong there.

What operations currently in xref don't fit "cross referencing"? I
searched for "interactive" in xref.el to get a list of user-level
commands. Most were clearly related to cross references; the following
could be questioned:

xref-pop-marker-stack, xref-push-marker-stack

    Marginal; the xref UI clearly needs this functionality, but so does
    any operation that produces a list of references (like compile,
    grep, etc). It was moved here from tags-*; better would have been to
    move it to a more neutral place.

xref-query-replace

    This operates on the current list of refs, so it makes sense in xref

xref-find-apropos

    This is just a search, not relying on any "cross reference"
    information.

    project-find-* provides the same functionality, so xref-find-apropos
    can be removed (assuming every xref backend provides a project
    backend).


A short phrase descibing xref could be "operations to produce and
operate on a list of cross-references", and "cross-reference" would be a
reasonable section name.

-- 
-- Stephe

Re: [Emacs-diffs] emacs-25 f8208b6: Document the user-level features of the Xref package

[Dmitry Gutov]

On 01/21/2016 10:19 PM, Stephen Leake wrote:

> xref-find-apropos
>
>      This is just a search, not relying on any "cross reference"
>      information.
>
>      project-find-* provides the same functionality, so xref-find-apropos
>      can be removed (assuming every xref backend provides a project
>      backend).

No, "apropos" lists the *definitions* matching a regexp. You won't get 
that by simply using regexp-search.

Re: [Emacs-diffs] emacs-25 f8208b6: Document the user-level features of the Xref package

[Stephen Leake]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 01/21/2016 10:19 PM, Stephen Leake wrote:
>
>> xref-find-apropos
>>
>>      This is just a search, not relying on any "cross reference"
>>      information.
>>
>>      project-find-* provides the same functionality, so xref-find-apropos
>>      can be removed (assuming every xref backend provides a project
>>      backend).
>
> No, "apropos" lists the *definitions* matching a regexp. You won't get
> that by simply using regexp-search.

The doc string says:

    Find all meaningful symbols that match PATTERN.
    The argument has the same meaning as in ‘apropos’.

I read that as "all occurances of the regexp"; I don't know what
"meaningful" is supposed to mean here.

If it is supposed to find only definitions, the doc string should say
so:

    Find the definitions of all symbols that match PATTERN.
    PATTERN has the same meaning as in `apropos'.

However, the etags implementation uses re-search-forward to find PATTERN
in the tags files; that won't handle word lists as `apropos' does. So
does that need to be fixed, or should we change the doc string further
to say:

    Find the definitions of all symbols that match PATTERN (a regexp).

-- 
-- Stephe

Re: [Emacs-diffs] emacs-25 f8208b6: Document the user-level features of the Xref package

[Eli Zaretskii]

> From: Stephen Leake <stephen_leake@stephe-leake.org>
> Cc: Eli Zaretskii <eliz@gnu.org>,  emacs-devel@gnu.org
> Date: Fri, 22 Jan 2016 01:40:04 -0600
> 
> Dmitry Gutov <dgutov@yandex.ru> writes:
> 
> > On 01/21/2016 10:19 PM, Stephen Leake wrote:
> >
> >> xref-find-apropos
> >>
> >>      This is just a search, not relying on any "cross reference"
> >>      information.
> >>
> >>      project-find-* provides the same functionality, so xref-find-apropos
> >>      can be removed (assuming every xref backend provides a project
> >>      backend).
> >
> > No, "apropos" lists the *definitions* matching a regexp. You won't get
> > that by simply using regexp-search.
> 
> The doc string says:
> 
>     Find all meaningful symbols that match PATTERN.
>     The argument has the same meaning as in ‘apropos’.

The doc string "needs work"™.  Welcome to the issue of doc strings of
quite a few commands introduced in Emacs 25.  Feel free to augment,
expand, improve, or even rewrite any of them, based on what the code
actually does.  Thanks in advance.

> I read that as "all occurances of the regexp"; I don't know what
> "meaningful" is supposed to mean here.
> 
> If it is supposed to find only definitions, the doc string should say
> so:
> 
>     Find the definitions of all symbols that match PATTERN.
>     PATTERN has the same meaning as in `apropos'.

The manual says:

     The command `C-M-.' (`xref-find-apropos') finds the definitions of
  one or more identifiers that match a specified regular expression.  It
  is just like `M-.' except that it does regexp matching of identifiers
  instead of matching symbol names as fixed strings.

> However, the etags implementation uses re-search-forward to find PATTERN
> in the tags files; that won't handle word lists as `apropos' does. So
> does that need to be fixed, or should we change the doc string further
> to say:
> 
>     Find the definitions of all symbols that match PATTERN (a regexp).

Good question.

Re: [Emacs-diffs] emacs-25 f8208b6: Document the user-level features of the Xref package

[Dmitry Gutov]

On 01/22/2016 11:21 AM, Eli Zaretskii wrote:

>> The doc string says:
>>
>>      Find all meaningful symbols that match PATTERN.
>>      The argument has the same meaning as in ‘apropos’.
>
> The doc string "needs work"™.

By the way, I (?) took the "meaningful symbols" phrase from apropos's 
docstring.

That one says "meaningful Lisp symbols", but naturally, I had to take 
the word "Lisp" out.

Re: [Emacs-diffs] emacs-25 f8208b6: Document the user-level features of the Xref package

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Fri, 22 Jan 2016 13:46:02 +0300
> 
> On 01/22/2016 11:21 AM, Eli Zaretskii wrote:
> 
> >> The doc string says:
> >>
> >>      Find all meaningful symbols that match PATTERN.
> >>      The argument has the same meaning as in ‘apropos’.
> >
> > The doc string "needs work"™.
> 
> By the way, I (?) took the "meaningful symbols" phrase from apropos's 
> docstring.

If what I said could be interpreted to mean the above is the _only_
doc string that needs work, then I apologize: it isn't, far from that.

Re: [Emacs-diffs] emacs-25 f8208b6: Document the user-level features of the Xref package

[Dmitry Gutov]

On 01/22/2016 10:40 AM, Stephen Leake wrote:

> The doc string says:
>
>      Find all meaningful symbols that match PATTERN.
>      The argument has the same meaning as in ‘apropos’.
>
> I read that as "all occurances of the regexp"; I don't know what
> "meaningful" is supposed to mean here.

Symbol that are "meaningful" in the current runtime. Functions, 
variables, etc.

> If it is supposed to find only definitions, the doc string should say
> so:
>
>      Find the definitions of all symbols that match PATTERN.
>      PATTERN has the same meaning as in `apropos'.

Feel free to change the docstring to this. However, "find the symbols" 
semantics might be preferable, since we might want to show some 
additional info there, not just locations of their definitions.

Just like M-x apropos does.

> However, the etags implementation uses re-search-forward to find PATTERN
> in the tags files; that won't handle word lists as `apropos' does. So
> does that need to be fixed, or should we change the doc string further
> to say:
>
>      Find the definitions of all symbols that match PATTERN (a regexp).

No. xref-find-apropos converts from PATTERN to a regexp itself. Just 
like `apropos'.

Re: [Emacs-diffs] emacs-25 f8208b6: Document the user-level features of the Xref package

[Eli Zaretskii]

> From: Stephen Leake <stephen_leake@stephe-leake.org>
> Cc: Dmitry Gutov <dgutov@yandex.ru>,  emacs-devel@gnu.org
> Date: Thu, 21 Jan 2016 13:19:45 -0600
> 
> > Cross-Referencing doesn't fit, IMO, not if you consider the user-level
> > functionality.  
> 
> "Cross-reference" was certainly the initial intent of the xref package.
> 
> If that name no longer fits, it means there is functionality in
> xref that doesn't belong there.

I don't think the decisions on what does and doesn't belong should be
based on names.  FWIW, I don't see in xref.el anything that doesn't
belong there.

> What operations currently in xref don't fit "cross referencing"?

None, IMO.

> xref-query-replace
> 
>     This operates on the current list of refs, so it makes sense in xref

"Reference" and "cross-reference" are two very different things.  I
use "reference" in that section, but even that is not always
appropriate.

> A short phrase descibing xref could be "operations to produce and
> operate on a list of cross-references", and "cross-reference" would be a
> reasonable section name.

They would both be wrong.

Anyway, the manual section that describes these commands is written,
and I don't intend investing any more significant effort on it for
this release.  It took enough of my resources already.  Let's move on.

Re: [Emacs-diffs] emacs-25 f8208b6: Document the user-level features of the Xref package

[John Yates]

[-- Attachment #1: Type: text/plain, Size: 1011 bytes --]

On Thu, Jan 21, 2016 at 3:09 PM, Eli Zaretskii <eliz@gnu.org> wrote:

> > From: Stephen Leake <stephen_leake@stephe-leake.org>
> > Cc: Dmitry Gutov <dgutov@yandex.ru>,  emacs-devel@gnu.org
> > Date: Thu, 21 Jan 2016 13:19:45 -0600
> >
> > > Cross-Referencing doesn't fit, IMO, not if you consider the user-level
> > > functionality.
> >
> > "Cross-reference" was certainly the initial intent of the xref package.
> >
> > If that name no longer fits, it means there is functionality in
> > xref that doesn't belong there.
>
> I don't think the decisions on what does and doesn't belong should be
> based on names.  FWIW, I don't see in xref.el anything that doesn't
> belong there.
>

My sense is that xref (and tags before it) suggests existence of an
index and that xref- operations are then driven by that index.  Not
by taking entries out of that index and extracting file paths.  That is
the role of project.  Project supplies only a set of files.  After that
regex-based algorithms seem entirely expected.

[-- Attachment #2: Type: text/html, Size: 1679 bytes --]

Re: [Emacs-diffs] emacs-25 f8208b6: Document the user-level features of the Xref package

[John Wiegley]

>>>>> Dmitry Gutov <dgutov@yandex.ru> writes:

> We don't use GNU Global, Cscope, ID Utils, or Grep as xref backends. We
> can't "find definitions" using any of them, for example, even though the
> user might expect to be able to do so.

Dmitry, "global --definition" will list all definitions from the GTAGS file.

Couldn't you find definitions with global that way?

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: [Emacs-diffs] emacs-25 f8208b6: Document the user-level features of the Xref package

[Dmitry Gutov]

On 01/19/2016 08:36 AM, John Wiegley wrote:

> Dmitry, "global --definition" will list all definitions from the GTAGS file.
>
> Couldn't you find definitions with global that way?

It's just code that needs to be written. Preferably while retaining the 
functionality from lisp/cedet/cedet-global.el, etc.