Project support and completions (Was: Re: bug#19466: 25.0.50; xref-find-def doesn't find C functions)

Project support and completions (Was: Re: bug#19466: 25.0.50; xref-find-def doesn't find C functions)

[Dmitry Gutov]

On 01/24/2015 11:40 AM, Eli Zaretskii wrote:

>> The easiest example is code completion.  In emacs-lisp-mode, it will
>> parse the current buffer context and offer completions from obarray,
>> somewhat filtered. If you have a README in the same directory as
>> foo.el (so we might consider them to be in the same project), when
>> you open it, you definitely won't get the same completions as in
>> emacs-lisp-mode. In fact, if there's no current tags file visited,
>> in the default Emacs configuration you won't get any completions at
>> all.
>>
>> Similarly for python-mode (python-shell-completion-at-point only works
>> in Python buffers, and only when an inferior shell is running). Likewise
>> for Octave, and I just haven't looked at Guile, Awk and GDB support.
>
> I'm not sure what completions are alluded to here.  Completions are
> context-dependent, i.e. their DWIM-ish nature should depend on what is
> being completed, and for what purposes.

Code completion, like the first sentence says. Or completions for text, 
if we're talking about that README file.

Since external completion tools are usually already wedded to specific 
notions of projects, let's consider "all words in buffers" (a la 
dabbrev). Arguably, such completion source should use the words from all 
buffers belonging to the same project, as well as some limited 
dictionary that's applicable everywhere.

elisp-completion-at-point only considers the local buffer contents and 
currently loaded symbols (which if fine by me, but that exactly follows 
the pattern).

tags-completion-at-point-function, which is the default, has a different 
problem: it will be used in any buffer, even those not within the 
current tag file(s)'s subtree(s).

> For example, I hope you'd
> agree that it would be bad in general for completion in "C-x C-f" to
> show only files from the current project.

That's neither here no there. `find-file' is a plainly different function.

> By contrast, a specialized
> command that is meant to find files only from the current project
> should probably limit the candidates to the project.

Yes, and that's one feature that would use the current project. When 
used with a completing-read-function that provides some kind of fuzzy 
matching, it can be very handy: type a few letters and have the file you 
were thinking of at your fingertips.

Not so with the default completing-read, though.

> Likewise with symbols -- when the user wants only symbols from the
> current project, or from the programming language of the current
> buffer, that's what Emacs should show.  But there are also situations
> where such limitations produce bad results.

Personally, I consider that handwavy thinking a problem. We can't 
support each and every approach and expect the user not to be 
overwhelmed trying to remember and be able to use all of them.

Even if situations where given limitations are bad can exist, as long as 
they are rare enough, we can sacrifice certain edge cases if what's left 
is a coherent, simple user interface.

An extreme example, I'd say, is Icicles (sorry, Drew). Just looking at 
the documentation makes my head spin. And even if there can be 
situations in my life where I could use each option, I'd rather do the 
work manually rather than look for it, or generally live with an 
interface that does so many things at once.

> The important point is we cannot limit the user to just one possible
> context, even if it is the most frequent one.  Such a feature would be
> too limited.

Ideally, we should provide extensible interfaces. The backend logic that 
xref uses is swappable, and an interested user can write their take on 
the indexing and navigation logic, then share their code, and other 
like-minded users will benefit. Some would be included in Emacs, and 
would be used by enabling a specific minor mode.

Re: Project support and completions (Was: Re: bug#19466: 25.0.50; xref-find-def doesn't find C functions)

[Eli Zaretskii]

> Date: Sat, 24 Jan 2015 20:43:02 +0200
> From: Dmitry Gutov <dgutov@yandex.ru>
> CC: emacs-devel <emacs-devel@gnu.org>
> 
> > Likewise with symbols -- when the user wants only symbols from the
> > current project, or from the programming language of the current
> > buffer, that's what Emacs should show.  But there are also situations
> > where such limitations produce bad results.
> 
> Personally, I consider that handwavy thinking a problem. We can't 
> support each and every approach and expect the user not to be 
> overwhelmed trying to remember and be able to use all of them.

If we succeed in parameterizing the space of possible approaches, and
let users specify what parameters they want, then yes, we can.

For example, it sounds to me that by having an "add project" and
"remove project" commands, we can give the user the ability to tell
which projects' databases of symbols are relevant to what she is doing
now.  Then Emacs can select completion candidates from the list of
projects currently in the active set.  Such a paradigm will not
overwhelm users, since knowing which packages one is working on at any
given time is something we all do anyway.

> Even if situations where given limitations are bad can exist, as long as 
> they are rare enough, we can sacrifice certain edge cases if what's left 
> is a coherent, simple user interface.

If we must.  But I don't think this situation is one of those cases.
I think simple yet powerful solutions are at hand here.

> > The important point is we cannot limit the user to just one possible
> > context, even if it is the most frequent one.  Such a feature would be
> > too limited.
> 
> Ideally, we should provide extensible interfaces.

Ideally, the popular extensions should already be there.  Working on
several packages at the same time is not an uncommon use case.

Re: Project support and completions (Was: Re: bug#19466: 25.0.50; xref-find-def doesn't find C functions)

[Dmitry Gutov]

On 01/25/2015 12:26 AM, Eli Zaretskii wrote:

> For example, it sounds to me that by having an "add project" and
> "remove project" commands, we can give the user the ability to tell

A specific project solution could provide those commands.

> which projects' databases of symbols are relevant to what she is doing
> now.  Then Emacs can select completion candidates from the list of
> projects currently in the active set.

This introduces a new constraint on the potential "project" API. While 
we want to be able to list files or buffers (and maybe do something with 
their contents) from all current projects, there are commands in 
Projectile that necessarily work just on one directory, such as:

- Visit project in Dired.
- Run a build tool, ag, grep, or any plain shell command in root.
- Run vc-dir in root.
- (Might or might not also fit the description) Regenerate tags.

So the project API either has to be able to designate one project as the 
"main one" while implementing "list all files" differently, or 
dynamically look up the real current project root, to use in those commands.

> Such a paradigm will not
> overwhelm users, since knowing which packages one is working on at any
> given time is something we all do anyway.

I believe some would still prefer just having it inferred.

> Ideally, the popular extensions should already be there.  Working on
> several packages at the same time is not an uncommon use case.

If the projects are unrelated, personally I'd rather keep them separate.

Or, in the example you've given before (application and a library it 
uses), at least with certain languages/environments the project solution 
could automatically infer all the libraries used by the current 
application, as well as their current sources, from the project file. 
That would make add-project and remove-project commands unnecessary, 
leaving only select-project.

Re: Project support and completions (Was: Re: bug#19466: 25.0.50; xref-find-def doesn't find C functions)

[John Yates]

[-- Attachment #1: Type: text/plain, Size: 766 bytes --]

On Sat, Jan 24, 2015 at 6:25 PM, Dmitry Gutov <dgutov@yandex.ru> wrote:

> On 01/25/2015 12:26 AM, Eli Zaretskii wrote:
>
>> For example, it sounds to me that by having an "add project" and
>> "remove project" commands, we can give the user the ability to tell
>>
>
Such a model is inherently stateful, hence problematic.  It makes
multiplexing work on multiple projects difficult and error-prone.

I believe some would still prefer just having it inferred.
>

Yes, please!  Absolutely.  For me a important recent elisp development was
introduction of locate-dominating-file.  Using it to perform just such
inference I have free myself of previously stateful, stressful mechanisms.

If the projects are unrelated, personally I'd rather keep them separate.


Me too.

[-- Attachment #2: Type: text/html, Size: 1783 bytes --]

Re: Project support and completions (Was: Re: bug#19466: 25.0.50; xref-find-def doesn't find C functions)

[Eli Zaretskii]

> Date: Sat, 24 Jan 2015 19:21:19 -0500
> From: John Yates <john@yates-sheets.org>
> Cc: Eli Zaretskii <eliz@gnu.org>, Emacs developers <emacs-devel@gnu.org>
> 
>     On 01/25/2015 12:26 AM, Eli Zaretskii wrote:

>         For example, it sounds to me that by having an "add project" and
>         "remove project" commands, we can give the user the ability to tell
> 
> Such a model is inherently stateful, hence problematic.  It makes multiplexing work on multiple projects difficult and error-prone.

Stateful, yes.  Working on a certain project or a set of projects is
indeed inherently stateful.

I don't see the problematic part in that, though.  Could you elaborate
on what practical problems you see with this?

>     I believe some would still prefer just having it inferred.
> 
> Yes, please!  Absolutely.

I thought I described a use case where automatic inference is
inherently limited and error-prone.

However, we could have both modes of operation, the automatically
inferred one for simple use cases, and the manual mode for more
complex ones.  There's no need to choose and support only one.

>     If the projects are unrelated, personally I'd rather keep them separate.
> 
> Me too.

Me too, but sometimes it makes sense not to.

Re: Project support and completions (Was: Re: bug#19466: 25.0.50; xref-find-def doesn't find C functions)

[John Yates]

[-- Attachment #1: Type: text/plain, Size: 913 bytes --]

EZ> For example, it sounds to me that by having an "add project" and
EZ> "remove project" commands, we can give the user the ability to tell

JY> Such a model is inherently stateful, hence problematic.  It makes
JY> multiplexing work on multiple projects difficult and error-prone.

EZ> Stateful, yes.  Working on a certain project or a set of projects is
EZ> indeed inherently stateful.
EZ>
EZ> I don't see the problematic part in that, though.  Could you elaborate
EZ> on what practical problems you see with this?

Eli,

To me your proposed model of "add project" / "remove project" seemed
just too reminiscent of etag's paradigm for managing tag tables.  Of late
I have been using git-new-workspace.  In that setting find-tag forever
seems to open the _wrong_ file.  More perniciously it often opens one
that looks right though in a different workspace, thereby slowing me down
and making me paranoid.

/john

[-- Attachment #2: Type: text/html, Size: 1517 bytes --]

Re: Project support and completions (Was: Re: bug#19466: 25.0.50; xref-find-def doesn't find C functions)

[Eli Zaretskii]

> Date: Thu, 29 Jan 2015 16:19:19 -0500
> From: John Yates <john@yates-sheets.org>
> Cc: Brief Busters <dgutov@yandex.ru>, Emacs developers <emacs-devel@gnu.org>
> 
> EZ> For example, it sounds to me that by having an "add project" and
> EZ> "remove project" commands, we can give the user the ability to tell
> 
> JY> Such a model is inherently stateful, hence problematic. It makes 
> JY> multiplexing work on multiple projects difficult and error-prone.
> 
> EZ> Stateful, yes. Working on a certain project or a set of projects is
> EZ> indeed inherently stateful.
> EZ>
> EZ> I don't see the problematic part in that, though. Could you elaborate
> EZ> on what practical problems you see with this?
> 
> Eli,
> 
> To me your proposed model of "add project" / "remove project" seemed
> just too reminiscent of etag's paradigm for managing tag tables.

There's nothing inherently wrong with that, is there?

> Of late I have been using git-new-workspace. In that setting
> find-tag forever seems to open the _wrong_ file. More perniciously
> it often opens one that looks right though in a different workspace,
> thereby slowing me down and making me paranoid.

I never used git-new-workspace, so I don't really understand what you
describe here.  In any case, this sounds like a problem with the
program that created the TAGS file, not with find-tag per se: the
latter just obeys what's in the database.

And I'd still want to know what are the problems with the stateful
approach I proposed.

Re: Project support and completions (Was: Re: bug#19466: 25.0.50; xref-find-def doesn't find C functions)

[John Yates]

[-- Attachment #1: Type: text/plain, Size: 3017 bytes --]

JY> To me your proposed model of "add project" / "remove project" seemed
JY> just too reminiscent of etag's paradigm for managing tag tables.

EZ> There's nothing inherently wrong with that, is there?

JY> Of late I have been using git-new-workspace. In that setting
JY> find-tag forever seems to open the _wrong_ file. More perniciously
JY> it often opens one that looks right though in a different workspace,
JY> thereby slowing me down and making me paranoid.

EZ> I never used git-new-workspace, so I don't really understand what you
EZ> describe here.  In any case, this sounds like a problem with the
EZ> program that created the TAGS file, not with find-tag per se: the
EZ> latter just obeys what's in the database.

EZ> And I'd still want to know what are the problems with the stateful
EZ> approach I proposed.

The historic way to use git to do work across different branches within a
single code base is to multiplex a single workspace.  This means that at
any instant one has available only a single branch and a single set of
(possibly out of date) tags.  Assume then that I have branches B1 and B2,
along with tag T defined in file F1 and referenced in file F2.  I generate
a tags database while I have branch B1 checked out.  I later switch to
branch B2 though I do not rebuild nor my tags database.  I open file F2
open and want to visit the definition of tag T.  My tags database, though
somewhat out of date (it describes branch B1), correctly opens B2's version
of file F1 (the only version available) and if I am lucky still gets me to
T's definition.

The git-new-workstation script automates setting up light weight per branch
workspaces.  (They are light weight primarily in the sense that they share
a single history database stored in a common parent workspace.)  This
arrangement enables multiplexing work across multiple directories merely by
opening a different path.  Note: no need even to cd to a different
directory.  So now there are multiple copies of every file, some different,
most identical, in congruent directory trees.  Going back to my earlier
example using git-new-workspace I would now have two workspaces, B1 and B2,
corresponding to my two branches, each containing files F1 and F2 and each
containing an up to date tags database.  I perform some work in B1 with the
net effect that I have a loaded tags table describing B1.  I then visit
B2::F2, do some work involving tag T causing me to want to visit T's
definition.  At this point C-. takes me to T's definition in B1::F1, not in
B2::F1 as I would want.  The problem is that both files exist on my disk
and are similar if not identical.  Thus, unless I am especially vigilant, I
edit and save the wrong file.

To reiterate my effort to switch branch contexts was nothing more than C-x
C-f or C-x b.  Given the current tags implementation getting emacs to use
the correct tags database involves explicit manipulation of session state
via visit-tags-table.  Hence my gripe about statefulness.

I hope that helps,

/john

[-- Attachment #2: Type: text/html, Size: 3351 bytes --]

Re: Project support and completions (Was: Re: bug#19466: 25.0.50; xref-find-def doesn't find C functions)

[Eli Zaretskii]

> Date: Fri, 30 Jan 2015 08:39:17 -0500
> From: John Yates <john@yates-sheets.org>
> Cc: Brief Busters <dgutov@yandex.ru>, Emacs developers <emacs-devel@gnu.org>
> 
> The git-new-workstation script automates setting up light weight per branch
> workspaces. (They are light weight primarily in the sense that they share a
> single history database stored in a common parent workspace.) This arrangement
> enables multiplexing work across multiple directories merely by opening a
> different path. Note: no need even to cd to a different directory. So now there
> are multiple copies of every file, some different, most identical, in congruent
> directory trees. Going back to my earlier example using git-new-workspace I
> would now have two workspaces, B1 and B2, corresponding to my two branches,
> each containing files F1 and F2 and each containing an up to date tags
> database. I perform some work in B1 with the net effect that I have a loaded
> tags table describing B1. I then visit B2::F2, do some work involving tag T
> causing me to want to visit T's definition. At this point C-. takes me to T's
> definition in B1::F1, not in B2::F1 as I would want. The problem is that both
> files exist on my disk and are similar if not identical. Thus, unless I am
> especially vigilant, I edit and save the wrong file.

You could do one of 2: (a) have both TAGS tables for both workspaces
loaded at all times.  Then M-. will consider both, although this is
less efficient (and the efficiency goes down with more workspaces)
because there are more candidates for each tag.

The other possibility is to switch TAGS tables when you switch to the
other workspace.  This is slightly more complex, but scales much
better.  You could do this via find-file-hook, for example, or some
other device.  Emacs has enough features that can be used for this.

All in all, I see no problems either way, and I don't understand your
gripe.

> To reiterate my effort to switch branch contexts was nothing more than C-x C-f
> or C-x b. Given the current tags implementation getting emacs to use the
> correct tags database involves explicit manipulation of session state via
> visit-tags-table. Hence my gripe about statefulness.

I see no problem here.  Your own state changes as well.

Re: Project support and completions

[Stefan Monnier]

> better.  You could do this via find-file-hook, for example, or some
> other device.  Emacs has enough features that can be used for this.

The files of the various branches may (and often are) all be opened at
the same time.  I "switch project" simply by moving from one buffer
to another (which often amounts to moving the mouse from one window to
another).

Often I switch back-and-forth repeatedly, e.g. while comparing the code
in the different branches, and I'd want M-. to always jump to the
corresponding code.  So "switch project" needs to be fully implicit.
Determining the current project dynamically based on default-directory
is the most obvious and straightforward way to make it work.


        Stefan

Re: Project support and completions

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Fri, 30 Jan 2015 13:14:42 -0500
> Cc: emacs-devel@gnu.org, dgutov@yandex.ru, John Yates <john@yates-sheets.org>
> 
> > better.  You could do this via find-file-hook, for example, or some
> > other device.  Emacs has enough features that can be used for this.
> 
> The files of the various branches may (and often are) all be opened at
> the same time.  I "switch project" simply by moving from one buffer
> to another (which often amounts to moving the mouse from one window to
> another).
> 
> Often I switch back-and-forth repeatedly, e.g. while comparing the code
> in the different branches, and I'd want M-. to always jump to the
> corresponding code.  So "switch project" needs to be fully implicit.

Fine, then this use case should be added to requirements.  It doesn't
invalidate others, though.

> Determining the current project dynamically based on default-directory
> is the most obvious and straightforward way to make it work.

That's one implementation, but it's not the only one.

Re: Project support and completions

[Stefan Monnier]

>> Determining the current project dynamically based on default-directory
>> is the most obvious and straightforward way to make it work.
> That's one implementation, but it's not the only one.

I think we should aim to make "transparently switch to the right
project" work in *all* cases (including yours).


        Stefan

Re: Project support and completions

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Cc: emacs-devel@gnu.org, dgutov@yandex.ru, john@yates-sheets.org
> Date: Sat, 31 Jan 2015 01:37:36 -0500
> 
> >> Determining the current project dynamically based on default-directory
> >> is the most obvious and straightforward way to make it work.
> > That's one implementation, but it's not the only one.
> 
> I think we should aim to make "transparently switch to the right
> project" work in *all* cases (including yours).

I'm okay with that, but I don't yet see how that would be possible.
All the ideas in that direction offered this far basically tried to
convince me that my use case is uninteresting/invalid/whatever.  In
contrast, I think it's much more frequent and natural than the
git-new-workspace thing.

Re: Project support and completions

[Stefan Monnier]

> I'm okay with that, but I don't yet see how that would be possible.
> All the ideas in that direction offered this far basically tried to
> convince me that my use case is uninteresting/invalid/whatever.

I don't think your use case is uninteresting.  And I think we should
support it.  But I think we should support via some way to explain to
Emacs the relationship between the various sub-projects so that it does
what you want without you having to repeat that explanation every time.


        Stefan

Re: Project support and completions

[Stephen Leake]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> better.  You could do this via find-file-hook, for example, or some
>> other device.  Emacs has enough features that can be used for this.
>
> The files of the various branches may (and often are) all be opened at
> the same time.  I "switch project" simply by moving from one buffer
> to another (which often amounts to moving the mouse from one window to
> another).
>
> Often I switch back-and-forth repeatedly, e.g. while comparing the code
> in the different branches, and I'd want M-. to always jump to the
> corresponding code.  So "switch project" needs to be fully implicit.
> Determining the current project dynamically based on default-directory
> is the most obvious and straightforward way to make it work.

I agree this is one mode that is useful.

Another useful mode is "only switch on user command.

Is there any reason we can't have both?

-- 
-- Stephe

Re: Project support and completions

[Stefan Monnier]

> I never used git-new-workspace, so I don't really understand what you
> describe here.

He describes a situation just like mine: even if I spend all day hacking
on nothing else than Emacs, I end up frequently switching between at
least 2, often 3 different "projects", which are simply 3 different
branches ("lightweight checkouts", actually, in Bazaar speech) of Emacs
(typically, "master", "local-work", and "emacs-24").

So for him (and for me) "multiple projects" is the norm, not
the exception.  And indeed, I rarely used the etags facility for that
very reason: it all too often brings me to the file in the wrong branch.


        Stefan

Re: Project support and completions

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Fri, 30 Jan 2015 11:17:50 -0500
> Cc: emacs-devel@gnu.org, dgutov@yandex.ru, John Yates <john@yates-sheets.org>
> 
> He describes a situation just like mine: even if I spend all day hacking
> on nothing else than Emacs, I end up frequently switching between at
> least 2, often 3 different "projects", which are simply 3 different
> branches ("lightweight checkouts", actually, in Bazaar speech) of Emacs
> (typically, "master", "local-work", and "emacs-24").
> 
> So for him (and for me) "multiple projects" is the norm, not
> the exception.  And indeed, I rarely used the etags facility for that
> very reason: it all too often brings me to the file in the wrong branch.

Maybe I'm missing something, but I see no problem catering to this use
case.  See my other mail.

Re: Project support and completions

[Scott Frazer]

On 1/30/15 11:17 AM, Stefan Monnier wrote:
>> I never used git-new-workspace, so I don't really understand what you
>> describe here.
>
> He describes a situation just like mine: even if I spend all day hacking
> on nothing else than Emacs, I end up frequently switching between at
> least 2, often 3 different "projects", which are simply 3 different
> branches ("lightweight checkouts", actually, in Bazaar speech) of Emacs
> (typically, "master", "local-work", and "emacs-24").
>
> So for him (and for me) "multiple projects" is the norm, not
> the exception.  And indeed, I rarely used the etags facility for that
> very reason: it all too often brings me to the file in the wrong branch.

Might I suggest something like what XEmacs does, and set the tags table
by looking for a matching path substring in a list?  And maybe also by
searching upwards?  I wrote this a while ago as a possible starting place:

http://www.emacswiki.org/emacs/etags-table.el

Scott

Re: Project support and completions

[Eli Zaretskii]

> Date: Sun, 25 Jan 2015 01:25:12 +0200
> From: Dmitry Gutov <dgutov@yandex.ru>
> CC: emacs-devel@gnu.org
> 
>     For example, it sounds to me that by having an "add project" and
>     "remove project" commands, we can give the user the ability to tell
> 
> A specific project solution could provide those commands.

I'd be surprised if they didn't already.

>     which projects' databases of symbols are relevant to what she is doing
>     now.  Then Emacs can select completion candidates from the list of
>     projects currently in the active set.
> 
> This introduces a new constraint on the potential "project" API. While we want to be able to list files or buffers (and maybe do something with their contents) from all current projects, there are commands in Projectile that necessarily work just on one directory, such as:
> 
> - Visit project in Dired.
> - Run a build tool, ag, grep, or any plain shell command in root.
> - Run vc-dir in root.
> - (Might or might not also fit the description) Regenerate tags.
> 
> So the project API either has to be able to designate one project as the "main one" while implementing "list all files" differently, or dynamically look up the real current project root, to use in those commands.

I agree.

However, for the purposes of this discussion, all that is needed is
for xref to expose some interface that would allow a project API to
tell xref which databases to use.  Then this interface could be used
even if no project system is in use, from user-level commands or other
features.

The important aspect of this is that at least in some situations the
set of databases to use for symbol completion should be dictated by
some code outside xref that uses an interface provided by xref, and
not necessarily inferred automatically.

>     Such a paradigm will not
>     overwhelm users, since knowing which packages one is working on at any
>     given time is something we all do anyway.
>
> I believe some would still prefer just having it inferred.

Then this mode should also be provided.  But since it has inherent
limitations, it cannot be the only one.

>     Ideally, the popular extensions should already be there.  Working on
>     several packages at the same time is not an uncommon use case.
> 
> If the projects are unrelated, personally I'd rather keep them separate.

What do you mean by "unrelated" and by "separate" here?

Suppose there's some bug when Emacs calls a GnuTLS function.  Does
working on both of them trying to solve the bug makes them "related"?
Do you agree that being able to browse and complete on symbols from
both of them might be very useful in this use case?

> Or, in the example you've given before (application and a library it uses), at least with certain languages/environments the project solution could automatically infer all the libraries used by the current application, as well as their current sources, from the project file. That would make add-project and remove-project commands unnecessary, leaving only select-project. 

Such automatic inference is not easy.  Library dependencies come in
several forms: linking in a Makefile, dynamic loading in the sources,
dependencies in pkg-config style *.pc files, etc.  Also, some
libraries depend on other libraries.  Discovering all those
dependencies sounds like a non-trivial project to me.

More importantly, most of the time when I work on the application, I
don't care about symbols in the libraries used by an application.  So
doing what you suggest would fill my "namespace" with many symbols I
don't care about and don't want to see as completion candidates.  As
an extreme example, consider libc, the Standard C library, which any C
program uses, or the Standard C++ library.  Is it really a good idea
for xref to offer all those when the user types 'M-.'?

OTOH, when I work on an interface between the application and a
library, then I do want the symbols of that single library to appear
as completion candidates.

Once again, I don't think this can be resolved automatically in a
satisfactory manner, at least not in all important use cases.  So a
provision for manual instruction by the user is still required, IMO.

Re: Project support and completions

[Dmitry Gutov]

On 01/25/2015 06:08 PM, Eli Zaretskii wrote:

> However, for the purposes of this discussion, all that is needed is
> for xref to expose some interface that would allow a project API to
> tell xref which databases to use.

It already does. Please keep xref out of this discussion, I've split it 
out intentionally.

> The important aspect of this is that at least in some situations the
> set of databases to use for symbol completion should be dictated by
> some code outside xref that uses an interface provided by xref, and
> not necessarily inferred automatically.

Like I mentioned, xref itself doesn't keep any databases.

> Suppose there's some bug when Emacs calls a GnuTLS function.  Does
> working on both of them trying to solve the bug makes them "related"?

Sure. Emacs uses GnuTLS, so they're related.

> Do you agree that being able to browse and complete on symbols from
> both of them might be very useful in this use case?

Of course. Being able to complete on symbols from _all_ libraries that 
the current project uses is useful.

> Such automatic inference is not easy.  Library dependencies come in
> several forms: linking in a Makefile, dynamic loading in the sources,
> dependencies in pkg-config style *.pc files, etc.  Also, some
> libraries depend on other libraries.  Discovering all those
> dependencies sounds like a non-trivial project to me.

Different packages can provide support for different kind of projects. 
There's no reasons for users of languages and build systems where this 
is easier to suffer the same hurdles as you.

> More importantly, most of the time when I work on the application, I
> don't care about symbols in the libraries used by an application.  So
> doing what you suggest would fill my "namespace" with many symbols I
> don't care about and don't want to see as completion candidates.

If you had access to context-sensitive completion, you might change your 
mind. Or maybe not. But it would improve the namespace pollution 
considerably.

> As
> an extreme example, consider libc, the Standard C library, which any C
> program uses, or the Standard C++ library.  Is it really a good idea
> for xref to offer all those when the user types 'M-.'?

Yeah, why not? Want to see the definition of printf? There you go!

> Once again, I don't think this can be resolved automatically in a
> satisfactory manner, at least not in all important use cases.  So a
> provision for manual instruction by the user is still required, IMO.

It will be solved by different users using different code completion 
solutions.

Re: Project support and completions

[Eli Zaretskii]

> Date: Sun, 25 Jan 2015 20:18:59 +0200
> From: Dmitry Gutov <dgutov@yandex.ru>
> CC: emacs-devel@gnu.org
> 
> > As
> > an extreme example, consider libc, the Standard C library, which any C
> > program uses, or the Standard C++ library.  Is it really a good idea
> > for xref to offer all those when the user types 'M-.'?
> 
> Yeah, why not? Want to see the definition of printf?

No, I want to see prin1, princ, prin1-to-string, etc.  printf is
really not what I want to see.

> > Once again, I don't think this can be resolved automatically in a
> > satisfactory manner, at least not in all important use cases.  So a
> > provision for manual instruction by the user is still required, IMO.
> 
> It will be solved by different users using different code completion 
> solutions.

This discussion is about specifying the solutions.

Re: Project support and completions

[Dmitry Gutov]

On 01/25/2015 09:22 PM, Eli Zaretskii wrote:

> No, I want to see prin1, princ, prin1-to-string, etc.  printf is
> really not what I want to see.

Then don't press `M-.' when point is on `printf', and don't type 
`printf' into the `C-u M-.' prompt. Problem solved?

> This discussion is about specifying the solutions.

I wouldn't say so. For instance, I consider tag files a very 
unsatisfactory source for completions (imprecise and often incomplete), 
but you and a certain slice of users are fine with them. So I wouldn't 
spend a lot of effort on completion backend that uses them, but still 
I'd like to allow it to exist.

For me, this discussion is mostly about clarifying some new requirements 
for a project API.

Re: Project support and completions

[Eli Zaretskii]

> Date: Mon, 26 Jan 2015 01:03:18 +0200
> From: Dmitry Gutov <dgutov@yandex.ru>
> CC: emacs-devel@gnu.org
> 
> On 01/25/2015 09:22 PM, Eli Zaretskii wrote:
> 
> > No, I want to see prin1, princ, prin1-to-string, etc.  printf is
> > really not what I want to see.
> 
> Then don't press `M-.' when point is on `printf', and don't type 
> `printf' into the `C-u M-.' prompt. Problem solved?

No, because I don't only want to see the identifiers which are already
near point.  Sometimes they are, but not always.  That's why M-. still
allows me to type a symbol name, even though it offers what's at point
as the default guess.  E.g., imagine that I need to use some printing
function for a Lisp object, but don't quite remember which ones we
have and what they do, exactly.

> > This discussion is about specifying the solutions.
> 
> I wouldn't say so. For instance, I consider tag files a very 
> unsatisfactory source for completions (imprecise and often incomplete), 
> but you and a certain slice of users are fine with them.

I will be even more fine with a better solution.  So far, it's you who
insist that I stay with the etags backend.

> For me, this discussion is mostly about clarifying some new requirements 
> for a project API.

That's very much related.  You need to have the possible solutions in
your sights when you design the APIs.

Re: Project support and completions

[Stephen Leake]

Eli Zaretskii <eliz@gnu.org> writes:

> However, for the purposes of this discussion, all that is needed is
> for xref to expose some interface that would allow a project API to
> tell xref which databases to use.  Then this interface could be used
> even if no project system is in use, from user-level commands or other
> features.
>
> The important aspect of this is that at least in some situations the
> set of databases to use for symbol completion should be dictated by
> some code outside xref that uses an interface provided by xref, and
> not necessarily inferred automatically.

+1

-- 
-- Stephe

Re: Project support and completions

[Stefan Monnier]

> For example, it sounds to me that by having an "add project" and
> "remove project" commands, we can give the user the ability to tell
> which projects' databases of symbols are relevant to what she is doing
> now.

I'd rather have something more declarative.  E.g. let the user specify
that some projects are linked, such that whenever you're in a buffer
that belongs to one of those linked projects, Emacs knows that it should
take the other linked projects into account.


        Stefan

Re: Project support and completions

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Cc: Dmitry Gutov <dgutov@yandex.ru>, emacs-devel@gnu.org
> Date: Sun, 25 Jan 2015 10:01:03 -0500
> 
> > For example, it sounds to me that by having an "add project" and
> > "remove project" commands, we can give the user the ability to tell
> > which projects' databases of symbols are relevant to what she is doing
> > now.
> 
> I'd rather have something more declarative.  E.g. let the user specify
> that some projects are linked

Works for me.

> such that whenever you're in a buffer that belongs to one of those
> linked projects, Emacs knows that it should take the other linked
> projects into account.

This will not always DWIM.  For example, it could be that a buffer
belongs to none of the projects known to Emacs, in which case Emacs
should somehow fall back to "the last used project", whatever that
means.  A case in point is when I compose mail about some problem I
have working on a project.

Second, sometimes I go to another buffer temporarily, for whatever
reasons, like to see how a certain problem was solved there.  But I
still am working on the same project I was before I switched buffers,
not on that other buffer's project.

So there should be a way to tell Emacs about these use cases,
otherwise the feature will still annoy.  I don't see how automatic
inference based on the current buffer can cater to those use cases.

IOW, I think there should be support for "I only work on one project
at a time" kind of users, which means "don't switch projects on me
unless I approve".

Re: Project support and completions

[Stefan Monnier]

> This will not always DWIM.  For example, it could be that a buffer
> belongs to none of the projects known to Emacs, in which case Emacs
> should somehow fall back to "the last used project", whatever that
> means.  A case in point is when I compose mail about some problem I
> have working on a project.

If we want to support "project-local" behavior (and I think we very
much do), then inevitably there will be cases where the user will have
to do extra work to tell Emacs which project to use.

> IOW, I think there should be support for "I only work on one project
> at a time" kind of users, which means "don't switch projects on me
> unless I approve".

If you've only used a single project in this Emacs, then we should be
able to avoid having you select a project when you're in a buffer which
is not explicitly linked to "the only project".


        Stefan

Re: Project support and completions

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: dgutov@yandex.ru,  emacs-devel@gnu.org
> Date: Sun, 25 Jan 2015 13:33:30 -0500
> 
> > IOW, I think there should be support for "I only work on one project
> > at a time" kind of users, which means "don't switch projects on me
> > unless I approve".
> 
> If you've only used a single project in this Emacs, then we should be
> able to avoid having you select a project when you're in a buffer which
> is not explicitly linked to "the only project".

Sure, but then I also need a means to say "now I'm switching to
another".  Because amazingly, sometimes I do.

Re: Project support and completions

[Stefan Monnier]

>> > IOW, I think there should be support for "I only work on one project
>> > at a time" kind of users, which means "don't switch projects on me
>> > unless I approve".
>> If you've only used a single project in this Emacs, then we should be
>> able to avoid having you select a project when you're in a buffer which
>> is not explicitly linked to "the only project".
> Sure, but then I also need a means to say "now I'm switching to
> another".  Because amazingly, sometimes I do.

Normally, this should magically happen when you visit a file in that
other project, at which point, your Emacs session has 2 projects active
and the user will then, when using project-dependent operations from
a buffer not explicitly linked to any project, choose which project to use.


        Stefan

Re: Project support and completions

[Dmitry Gutov]

On 01/26/2015 12:43 AM, Stefan Monnier wrote:
> Normally, this should magically happen when you visit a file in that
> other project, at which point, your Emacs session has 2 projects active

I'd prefer not to have Emacs core make this decision for me. Quit often, 
when I visit files from different projects, they are just it: different 
projects.

Re: Project support and completions

[Stefan Monnier]

>> Normally, this should magically happen when you visit a file in that
>> other project, at which point, your Emacs session has 2 projects active
> I'd prefer not to have Emacs core make this decision for me.

Which decision?

> Quite often, when I visit files from different projects, they are just
> it: different projects.

Right, that's exactly what I expect.  Are we in violent agreement, or am
I missing something?


        Stefan

Re: Project support and completions

[Dmitry Gutov]

On 01/26/2015 05:46 AM, Stefan Monnier wrote:

> Which decision?

Whether to make the second project active as soon as I visit a file in 
it. Assuming you use Eli's meaning of "active" (used for completion and 
navigation in all (?) buffers).

Or whether to keep the previous project active, or entirely "leave" it.

>> Quite often, when I visit files from different projects, they are just
>> it: different projects.
>
> Right, that's exactly what I expect.  Are we in violent agreement, or am
> I missing something?

We probably are, but I'm also entertaining the idea of supporting that 
alternative approach mentioned above.

If we want to accomodate it, it will need separate notions of "active" 
projects (ones we're interested in in the current session) and the 
"current" project (the one the current file belongs to).

In short, you're expressing ideas about how things should work in an 
implementation, and I worry that someone might consider them good 
constraints on the project API.

Re: Project support and completions

[Stefan Monnier]

>> Which decision?
> Whether to make the second project active as soon as I visit a file in
> it.  Assuming you use Eli's meaning of "active" (used for completion and
> navigation in all (?) buffers).

No, a project would be only ever active in the buffers related to it
(plus in the buffers related to no project at all, but in that case the
user has to choose which project to use, unless there's only one).


        Stefan

Re: Project support and completions

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org
> Date: Mon, 26 Jan 2015 10:07:48 -0500
> 
> >> Which decision?
> > Whether to make the second project active as soon as I visit a file in
> > it.  Assuming you use Eli's meaning of "active" (used for completion and
> > navigation in all (?) buffers).
> 
> No, a project would be only ever active in the buffers related to it

In some use cases, this would be an annoying limitation.  So we need a
fire escape for those.

> (plus in the buffers related to no project at all, but in that case the
> user has to choose which project to use, unless there's only one).

For each such buffer?  If so, that's again an annoyance.

I don't really understand the rationale behind this design.  How
frequently (in time units) do you switch from one project to another?
Unless you do that several times a minute, I cannot see why would we
want this automation.

Me, I change project maybe once or twice a day, sometimes once a
week.  With such a low frequency, I don't need the automation too
much, but I do need the freedom to do project-related stuff from any
buffer I happen to be in.  Maybe I'm the odd one out.

Re: Project support and completions

[Stephen Leake]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
>> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org
>> Date: Mon, 26 Jan 2015 10:07:48 -0500
>> 
>> >> Which decision?
>> > Whether to make the second project active as soon as I visit a file in
>> > it.  Assuming you use Eli's meaning of "active" (used for completion and
>> > navigation in all (?) buffers).
>> 
>> No, a project would be only ever active in the buffers related to it
>
> In some use cases, this would be an annoying limitation.  So we need a
> fire escape for those.
>
>> (plus in the buffers related to no project at all, but in that case the
>> user has to choose which project to use, unless there's only one).
>
> For each such buffer?  If so, that's again an annoyance.
>
> I don't really understand the rationale behind this design.  How
> frequently (in time units) do you switch from one project to another?
> Unless you do that several times a minute, I cannot see why would we
> want this automation.
>
> Me, I change project maybe once or twice a day, sometimes once a
> week.  With such a low frequency, I don't need the automation too
> much, but I do need the freedom to do project-related stuff from any
> buffer I happen to be in.  Maybe I'm the odd one out.

In ada-mode, I use the compiler defined project definition files to
define Emacs projects.

One file can belong to several projects, due to project nesting. For
example:

common
    base
        file_1.ads

MAP
    models
        file_2.ads

MMS
    models
        file_3.ads

file_2.ads and file_3.ada both depend on file_1.ads, but MAP and MMS are
different (high-level) projects

The file common/base/file_1.ads belongs to the "common", "MAP", and "MMS"
projects.

I have no automation to select the project; the user must always choose;
I usually put a file local expression in the Makefile for each project
to set that project initially. But after I have several Makefiles open,
I switch between projects manually. And I like it that way.

The list of projects that Emacs knows about (ie, whose project
definition files have been parsed) is available on an ada-mode menu, so
it's easy to switch between projects.

ada-mode projects are actually multi-language; they include C and C++,
and anything else the user has told the compiler project file about.


On the other hand, I do edit elisp code (to fix ada-mode, for example)
while I am editing Ada code. So it might be nice to have Emacs switch
between "the current elisp project" and "the current real project"
automatically. Maybe based on major mode? ie "the current real project"
includes all buffers with ada, C, or C++ mode. In my case, those
languages are listed in the compiler project file, so that is at least
possible to implement without further user work. But I suspect this is a
_very_ special case :).


If there is project choosing automation, it must be possible to turn it
off or override it in some way.


I have not been following this thread closely; is there a proposed API
somewhere?

-- 
-- Stephe

Re: Project support and completions

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Cc: dgutov@yandex.ru, emacs-devel@gnu.org
> Date: Sun, 25 Jan 2015 17:43:52 -0500
> 
> >> > IOW, I think there should be support for "I only work on one project
> >> > at a time" kind of users, which means "don't switch projects on me
> >> > unless I approve".
> >> If you've only used a single project in this Emacs, then we should be
> >> able to avoid having you select a project when you're in a buffer which
> >> is not explicitly linked to "the only project".
> > Sure, but then I also need a means to say "now I'm switching to
> > another".  Because amazingly, sometimes I do.
> 
> Normally, this should magically happen when you visit a file in that
> other project, at which point, your Emacs session has 2 projects active
> and the user will then, when using project-dependent operations from
> a buffer not explicitly linked to any project, choose which project to use.

I have just described a few messages ago why this logic will sometimes
misfire.

Re: Project support and completions

[Stefan Monnier]

>> Normally, this should magically happen when you visit a file in that
>> other project, at which point, your Emacs session has 2 projects active
>> and the user will then, when using project-dependent operations from
>> a buffer not explicitly linked to any project, choose which project to use.
> I have just described a few messages ago why this logic will sometimes
> misfire.

I'm not sure which case you're referring to, but I have no doubt it won't
always do the right thing.  As mentioned, there is no single solution
that will always do the right thing without reading the user's mind.
But this solution should provide predictable behavior and when it
doesn't automatically do what the user wants, it's easy for the user to
instruct tell Emacs what she wants (i.e. by switching buffer).


        Stefan

Re: Project support and completions

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Cc: dgutov@yandex.ru, emacs-devel@gnu.org
> Date: Mon, 26 Jan 2015 10:11:09 -0500
> 
> >> Normally, this should magically happen when you visit a file in that
> >> other project, at which point, your Emacs session has 2 projects active
> >> and the user will then, when using project-dependent operations from
> >> a buffer not explicitly linked to any project, choose which project to use.
> > I have just described a few messages ago why this logic will sometimes
> > misfire.
> 
> I'm not sure which case you're referring to, but I have no doubt it won't
> always do the right thing.  As mentioned, there is no single solution
> that will always do the right thing without reading the user's mind.
> But this solution should provide predictable behavior and when it
> doesn't automatically do what the user wants, it's easy for the user to
> instruct tell Emacs what she wants (i.e. by switching buffer).

Why not instruct Emacs without switching buffers?  That seems easy
enough, and eliminates an annoyance of the buffer switch.