Re: master 1e3b0f2: Improve doc strings of project.el

Re: master 1e3b0f2: Improve doc strings of project.el

[Dmitry Gutov]

On 19.06.2020 10:54, Eli Zaretskii wrote:
> -  "Switch to a buffer in the current project."
> +  "Switch to another buffer that visits some file in the current project."

That's not exactly true: non-file-visiting buffers are also suggested.

Re: master 1e3b0f2: Improve doc strings of project.el

[Eli Zaretskii]

> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Fri, 19 Jun 2020 14:01:22 +0300
> 
> On 19.06.2020 10:54, Eli Zaretskii wrote:
> > -  "Switch to a buffer in the current project."
> > +  "Switch to another buffer that visits some file in the current project."
> 
> That's not exactly true: non-file-visiting buffers are also suggested.

Feel free to suggest a correction.  "Buffer in a current project"
sounds too vague at best to me: a project doesn't include any buffers.

Re: master 1e3b0f2: Improve doc strings of project.el

[Basil L. Contovounesios]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Dmitry Gutov <dgutov@yandex.ru>
>> Date: Fri, 19 Jun 2020 14:01:22 +0300
>> 
>> On 19.06.2020 10:54, Eli Zaretskii wrote:
>> > -  "Switch to a buffer in the current project."
>> > +  "Switch to another buffer that visits some file in the current project."
>> 
>> That's not exactly true: non-file-visiting buffers are also suggested.
>
> Feel free to suggest a correction.  "Buffer in a current project"
> sounds too vague at best to me: a project doesn't include any buffers.

Is "buffer under a project" any better, or just as vague?

-- 
Basil

Re: master 1e3b0f2: Improve doc strings of project.el

[Eli Zaretskii]

> From: "Basil L. Contovounesios" <contovob@tcd.ie>
> Cc: Dmitry Gutov <dgutov@yandex.ru>,  emacs-devel@gnu.org
> Date: Fri, 19 Jun 2020 12:37:39 +0100
> > Feel free to suggest a correction.  "Buffer in a current project"
> > sounds too vague at best to me: a project doesn't include any buffers.
> 
> Is "buffer under a project" any better, or just as vague?

Just as vague, IMO.  We need to explain what that means in practical
terms.

Re: master 1e3b0f2: Improve doc strings of project.el

[Dmitry Gutov]

On 19.06.2020 14:24, Eli Zaretskii wrote:

 > Feel free to suggest a correction.

My suggested correction would be the previous wording. It might have 
been more vague, but it covered both cases.

> "Buffer in a current project"
> sounds too vague at best to me: a project doesn't include any buffers.

It includes their default-directory-es. Any idea how to modify the 
docstring that the first sentence still fits on one line?

Re: master 1e3b0f2: Improve doc strings of project.el

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Fri, 19 Jun 2020 15:30:33 +0300
> 
> On 19.06.2020 14:24, Eli Zaretskii wrote:
> 
>  > Feel free to suggest a correction.
> 
> My suggested correction would be the previous wording. It might have 
> been more vague, but it covered both cases.

That'd be a step backwards, IMO.  I changed the doc string because in
its original form it didn't tell me enough to understand which buffers
could and couldn't be selected by that command.

In general, it is IMO wrong to make a doc string too vague because
there's some difficulty in saying something accurately and yet
succinctly enough.  We should look for solutions that don't leave our
doc strings too vague.

> > "Buffer in a current project"
> > sounds too vague at best to me: a project doesn't include any buffers.
> 
> It includes their default-directory-es. Any idea how to modify the 
> docstring that the first sentence still fits on one line?

What kind of buffers are those, in normal use?  IOW, what buffers are
relevant to the project, but don't visit any file/directory?

Armed with that knowledge, I could suggest a more accurate doc string
that still tells enough to understand what the command does.

Thanks.

Re: master 1e3b0f2: Improve doc strings of project.el

[Dmitry Gutov]

On 19.06.2020 15:44, Eli Zaretskii wrote:

>> My suggested correction would be the previous wording. It might have
>> been more vague, but it covered both cases.
> 
> That'd be a step backwards, IMO.  I changed the doc string because in
> its original form it didn't tell me enough to understand which buffers
> could and couldn't be selected by that command.

In general, those would be the buffers where (project-current) would 
return the current project. I think that's pretty unambiguous.

>>> "Buffer in a current project"
>>> sounds too vague at best to me: a project doesn't include any buffers.
>>
>> It includes their default-directory-es. Any idea how to modify the
>> docstring that the first sentence still fits on one line?
> 
> What kind of buffers are those, in normal use?  IOW, what buffers are
> relevant to the project, but don't visit any file/directory?

If you look at bug#41868, it mentions VC-Dir and Dired buffers. Perhaps 
Philip also has other examples.

It also cleans up some "invisible" buffers created to service the files 
in the project, such as ones created by Flymake, etc.

Re: master 1e3b0f2: Improve doc strings of project.el

[Philip K.]

Dmitry Gutov <dgutov@yandex.ru> writes:

>>>> "Buffer in a current project"
>>>> sounds too vague at best to me: a project doesn't include any buffers.
>>>
>>> It includes their default-directory-es. Any idea how to modify the
>>> docstring that the first sentence still fits on one line?
>> 
>> What kind of buffers are those, in normal use?  IOW, what buffers are
>> relevant to the project, but don't visit any file/directory?
>
> If you look at bug#41868, it mentions VC-Dir and Dired buffers. Perhaps 
> Philip also has other examples.

Magit, compile, xref, grep, occur, eshell would come to mind.

> It also cleans up some "invisible" buffers created to service the files 
> in the project, such as ones created by Flymake, etc.

On that topic, I noticed that it might confusing, since they are also
counted as "buffers to be killed", while project-switch-to-buffer
ignores these, perhaps the prompt should point that out? Just wanted to
mention that before I forget it again.

-- 
	Philip K.

Re: master 1e3b0f2: Improve doc strings of project.el

[Dmitry Gutov]

On 19.06.2020 16:27, Philip K. wrote:
>> It also cleans up some "invisible" buffers created to service the files
>> in the project, such as ones created by Flymake, etc.
> On that topic, I noticed that it might confusing, since they are also
> counted as "buffers to be killed", while project-switch-to-buffer
> ignores these, perhaps the prompt should point that out? Just wanted to
> mention that before I forget it again.

Indeed, I also noticed that, and it is somewhat confusing, though 
accurate. Those buffers can also be created in the background, without 
explicit user interaction.

Not sure what's the best approach. Some options:

- Kill xx buffers ([and/of them] xy invisible).
- Don't count the invisible ones.
- Keep the prompt as-is, just mention that issue in the docstring.

Re: master 1e3b0f2: Improve doc strings of project.el

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org, "Philip K." <philip@warpmail.net>
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Fri, 19 Jun 2020 15:54:56 +0300
> 
> > That'd be a step backwards, IMO.  I changed the doc string because in
> > its original form it didn't tell me enough to understand which buffers
> > could and couldn't be selected by that command.
> 
> In general, those would be the buffers where (project-current) would 
> return the current project. I think that's pretty unambiguous.

That's the implementation, but it cannot be used to make the doc
string useful, at least not directly.  It tells you what is the test
for whether a particular buffer is eligible for the switch, but the
doc string should instead tell up front what are the general traits of
all the eligible buffers.  IOW, the user should understand which of
the buffers he or she can switch to with this command.

> >>> "Buffer in a current project"
> >>> sounds too vague at best to me: a project doesn't include any buffers.
> >>
> >> It includes their default-directory-es. Any idea how to modify the
> >> docstring that the first sentence still fits on one line?
> > 
> > What kind of buffers are those, in normal use?  IOW, what buffers are
> > relevant to the project, but don't visit any file/directory?
> 
> If you look at bug#41868, it mentions VC-Dir and Dired buffers. Perhaps 
> Philip also has other examples.

So will something like the below do the job?

    Switch to another buffer related to the current project.
  A buffer is related to a project if its `default-directory'
  is inside the directory hierarchy of the project's root.

Btw, AFAIU this command can prompt for a project's root, but the doc
string doesn't mention that important fact.

> It also cleans up some "invisible" buffers created to service the files 
> in the project, such as ones created by Flymake, etc.

By "it" what do you mean here? you don't mean
project-switch-to-buffer, do you?  And if so, how is that related to
the issue at hand, which is the doc string of that function?

Re: master 1e3b0f2: Improve doc strings of project.el

[Eli Zaretskii]

> From: "Philip K." <philip@warpmail.net>
> Cc: eliz@gnu.org, emacs-devel@gnu.org
> Date: Fri, 19 Jun 2020 15:27:09 +0200
> 
> > If you look at bug#41868, it mentions VC-Dir and Dired buffers. Perhaps 
> > Philip also has other examples.
> 
> Magit, compile, xref, grep, occur, eshell would come to mind.

It sounds like _any_ buffer is considered "in a project" as long as
its default-directory is under the project's root, is that right?  So
if I start Emacs from somewhere inside the directory hierarchy rooted
at the project's root, then *scratch*, *Messages*, and temporary
buffers I create with "C-x b" will be considered to be part of the
project?  Does this really make sense?

Re: master 1e3b0f2: Improve doc strings of project.el

[Dmitry Gutov]

On 19.06.2020 17:11, Eli Zaretskii wrote:
> So
> if I start Emacs from somewhere inside the directory hierarchy rooted
> at the project's root, then*scratch*,*Messages*, and temporary
> buffers I create with "C-x b" will be considered to be part of the
> project?  Does this really make sense?

We can add scratch and Messages to the skip-conditions by default.

Re: master 1e3b0f2: Improve doc strings of project.el

[Dmitry Gutov]

On 19.06.2020 17:07, Eli Zaretskii wrote:
>> If you look at bug#41868, it mentions VC-Dir and Dired buffers. Perhaps
>> Philip also has other examples.
> So will something like the below do the job?
> 
>      Switch to another buffer related to the current project.
>    A buffer is related to a project if its `default-directory'
>    is inside the directory hierarchy of the project's root.

Sounds good. Except some projects can be inside one another, too.

> Btw, AFAIU this command can prompt for a project's root, but the doc
> string doesn't mention that important fact.

That is the case for all of the commands in there.

>> It also cleans up some "invisible" buffers created to service the files
>> in the project, such as ones created by Flymake, etc.
> By "it" what do you mean here? you don't mean
> project-switch-to-buffer, do you?  And if so, how is that related to
> the issue at hand, which is the doc string of that function?

Sorry, got my wires crossed: this was about project-kill-buffers.

But the rest of the discussion should apply to project-switch-buffer 
just as well.

BTW, I imagine that the current project-kill-buffers docstring should 
seem ambiguous to you as well.

Re: master 1e3b0f2: Improve doc strings of project.el

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Fri, 19 Jun 2020 17:18:17 +0300
> 
> On 19.06.2020 17:11, Eli Zaretskii wrote:
> > So
> > if I start Emacs from somewhere inside the directory hierarchy rooted
> > at the project's root, then*scratch*,*Messages*, and temporary
> > buffers I create with "C-x b" will be considered to be part of the
> > project?  Does this really make sense?
> 
> We can add scratch and Messages to the skip-conditions by default.

What about the others, though?  I wouldn't even expect *XREF* to be
eligible, as that is a kind-of "service" buffer to which I don't
really switch, I let the relevant commands put me there when needed.

Likewise for Eshell and Occur -- why are those candidates?  Basically,
any buffer whose name is *SOMETHING* should be by default excluded, I
think those that should be included are the exceptions, not the rule.

More generally, I question the wisdom of judging by the
default-directory in the first place.  Can't we come up with a better,
more specific criterion?

Re: master 1e3b0f2: Improve doc strings of project.el

[Theodor Thornhill]

[-- Attachment #1: Type: text/plain, Size: 358 bytes --]

> buffers I create with "C-x b" will be considered to be part of the
> project? Does this really make sense?

This is the reason I only included file bound buffers initially in switch-to-buffer. But as Andrii noted, there are other interesting buffers as well, such as xref etc.

Maybe we can add some logic to filter out more of the unrelated buffers?

Theo

[-- Attachment #2: Type: text/html, Size: 538 bytes --]

Re: master 1e3b0f2: Improve doc strings of project.el

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org, philip@warpmail.net
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Fri, 19 Jun 2020 17:23:36 +0300
> 
> >      Switch to another buffer related to the current project.
> >    A buffer is related to a project if its `default-directory'
> >    is inside the directory hierarchy of the project's root.
> 
> Sounds good. Except some projects can be inside one another, too.

Does that invalidate what the text above says?

> > Btw, AFAIU this command can prompt for a project's root, but the doc
> > string doesn't mention that important fact.
> 
> That is the case for all of the commands in there.

That just means they all should mention this.

> BTW, I imagine that the current project-kill-buffers docstring should 
> seem ambiguous to you as well.

Not really, no.  It's pretty self-explanatory, especially since the
usual situation where the command will be invoked is when I cease
working on a project.

Re: master 1e3b0f2: Improve doc strings of project.el

[Eli Zaretskii]

> Date: Fri, 19 Jun 2020 14:25:46 +0000
> From: Theodor Thornhill <theo@thornhill.no>
> Cc: emacs-devel@gnu.org, dgutov@yandex.ru
> 
>  buffers I create with "C-x b" will be considered to be part of the
>  project? Does this really make sense?
> 
> This is the reason I only included file bound buffers initially in switch-to-buffer. But as Andrii noted, there are
> other interesting buffers as well, such as xref etc. 

But the fact that a buffer visits a file doesn't yet make it a good
candidate, does it?

> Maybe we can add some logic to filter out more of the unrelated buffers?

Maybe we should step back and talk about the intended uses of this
command?  When would you want to switch to a buffer whose name you
don't know in advance?

As another provocative thought, should we perhaps consider the
major-mode of the buffer?

Re: master 1e3b0f2: Improve doc strings of project.el

[Philip K.]

Eli Zaretskii <eliz@gnu.org> writes:

>> Cc: emacs-devel@gnu.org
>> From: Dmitry Gutov <dgutov@yandex.ru>
>> Date: Fri, 19 Jun 2020 17:18:17 +0300
>> 
>> On 19.06.2020 17:11, Eli Zaretskii wrote:
>> > So
>> > if I start Emacs from somewhere inside the directory hierarchy rooted
>> > at the project's root, then*scratch*,*Messages*, and temporary
>> > buffers I create with "C-x b" will be considered to be part of the
>> > project?  Does this really make sense?
>> 
>> We can add scratch and Messages to the skip-conditions by default.
>
> What about the others, though?  I wouldn't even expect *XREF* to be
> eligible, as that is a kind-of "service" buffer to which I don't
> really switch, I let the relevant commands put me there when needed.
>
> Likewise for Eshell and Occur -- why are those candidates?  Basically,
> any buffer whose name is *SOMETHING* should be by default excluded, I
> think those that should be included are the exceptions, not the rule.
>
> More generally, I question the wisdom of judging by the
> default-directory in the first place.  Can't we come up with a better,
> more specific criterion?

My rule of thumb would be was a buffer generated "by" a file/action on
file in the project. The goal would be that if I open a project, do some
work and then call project-kill-buffers, and repeat this over 10, 100 or
however many (different) projects, that the list of live buffers doesn't
increase dramatically, or even proportionally to how many projects I
have used in general.

-- 
	Philip K.

Re: master 1e3b0f2: Improve doc strings of project.el

[Dmitry Gutov]

On 19.06.2020 17:37, Eli Zaretskii wrote:

 >>>      Switch to another buffer related to the current project.
 >>>    A buffer is related to a project if its `default-directory'
 >>>    is inside the directory hierarchy of the project's root.
 >>
 >> Sounds good. Except some projects can be inside one another, too.

 > Does that invalidate what the text above says?

If the current project contains a few other projects inside of its 
subdirectories, project-switch-to-buffer probably should offer any of 
"their" buffers for switching.

It current does, but that's arguably a bug. :-(

> Maybe we should step back and talk about the intended uses of this
> command?  When would you want to switch to a buffer whose name you
> don't know in advance?

When don't I? Do you use icomplete, or something similar?

As soon as you start relying on completion hints, you don't need to 
remember the names in advance. And it's also a good overview of the 
current open buffers in the current project.

Also, different projects can have files with conflicting names.

And yes, uniquify solves that somewhat by renaming one of the buffers, 
but the other keeps the base name, so it's not immediately apparent to 
which project it belongs.

Re: master 1e3b0f2: Improve doc strings of project.el

[Dmitry Gutov]

On 19.06.2020 17:37, Eli Zaretskii wrote:
> As another provocative thought, should we perhaps consider the
> major-mode of the buffer?

It's not the worst idea, but the set of "buffers belonging to the 
current project" should probably be the same between 
project-switch-to-buffer and project-kill-buffers.

And as Philip explained, it's useful for the latter to consider many 
different major modes.

Re: master 1e3b0f2: Improve doc strings of project.el

[Theodor Thornhill via Emacs development discussions.]

[-- Attachment #1: Type: text/plain, Size: 789 bytes --]

> But the fact that a buffer visits a file doesn't yet make it a good
> candidate, does it?

Well, no not generally.

> Maybe we should step back and talk about the intended uses of this
> command?

I wanted a way to filter out buffers unrelated to the project I’m in, without having to clean up the whole buffer list

> When would you want to switch to a buffer whose name you
> don't know in advance?

For me, almost always, though not because I don’t know its name. I’d like emacs to help me remember and offer auto completion.

> As another provocative thought, should we perhaps consider the
> major-mode of the buffer?

Maybe, but I don’t see how just yet. I thought about this, but a project isn’t generally bound to one major mode, is it? I’d love some tips here.

Theo

[-- Attachment #2: Type: text/html, Size: 1218 bytes --]

Re: master 1e3b0f2: Improve doc strings of project.el

[Eli Zaretskii]

> Cc: philip@warpmail.net, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Fri, 19 Jun 2020 17:49:00 +0300
> 
> On 19.06.2020 17:37, Eli Zaretskii wrote:
> 
>  >>>      Switch to another buffer related to the current project.
>  >>>    A buffer is related to a project if its `default-directory'
>  >>>    is inside the directory hierarchy of the project's root.
>  >>
>  >> Sounds good. Except some projects can be inside one another, too.
> 
>  > Does that invalidate what the text above says?
> 
> If the current project contains a few other projects inside of its 
> subdirectories, project-switch-to-buffer probably should offer any of 
> "their" buffers for switching.

"Should" or "should not"?  If the former, then the text I proposed is
fine as it is.

> > Maybe we should step back and talk about the intended uses of this
> > command?  When would you want to switch to a buffer whose name you
> > don't know in advance?
> 
> When don't I? Do you use icomplete, or something similar?
> 
> As soon as you start relying on completion hints, you don't need to 
> remember the names in advance. And it's also a good overview of the 
> current open buffers in the current project.

If this is just for completion's sake, then I could understand, but
still it sounds very strange to include stuff like *XREF*, Eshell,
Occur, *Compilation*, *Grep*, etc.  Those are basically singletons, at
least by default, and a simple "C-x b" should be enough to switch to
them.  project-switch-to-buffer should be more helpful by offering
only "useful" buffers.

> Also, different projects can have files with conflicting names.

That cannot be solved by completion, where only the basename is
usually shown.

Re: master 1e3b0f2: Improve doc strings of project.el

[Eli Zaretskii]

> Cc: philip@warpmail.net, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Fri, 19 Jun 2020 18:02:07 +0300
> 
> On 19.06.2020 17:37, Eli Zaretskii wrote:
> > As another provocative thought, should we perhaps consider the
> > major-mode of the buffer?
> 
> It's not the worst idea, but the set of "buffers belonging to the 
> current project" should probably be the same between 
> project-switch-to-buffer and project-kill-buffers.

No, not necessarily.  The purpose of the commands is different, so
they could start with the same basic list, but apply different filters
to it.

> And as Philip explained, it's useful for the latter to consider many 
> different major modes.

I'm talking about the former, not about the latter.

Re: master 1e3b0f2: Improve doc strings of project.el

[Eli Zaretskii]

> Date: Fri, 19 Jun 2020 15:02:45 +0000
> From: Theodor Thornhill <theothornhill@pm.me>
> Cc: philip@warpmail.net, dgutov@yandex.ru, emacs-devel@gnu.org
> 
>  As another provocative thought, should we perhaps consider the
>  major-mode of the buffer?
> 
> Maybe, but I don’t see how just yet. I thought about this, but a project isn’t generally bound to one major
> mode, is it? I’d love some tips here. 

A project could specify a list of major-modes, couldn't it?  That
would generally derive from the project's "purpose" or "goal", for
lack of a better word.  For example, a project where you build a
program would have prog-mode or some of its descendants in the list,
and perhaps also Texinfo and/or markdown for docs.  A project whose
purpose is to write an essay or a blog or might use descendants of
text-mode.  Etc., etc.

Re: master 1e3b0f2: Improve doc strings of project.el

[Theodor Thornhill]

> A project could specify a list of major-modes, couldn't it?  That
> would generally derive from the project's "purpose" or "goal", for
> lack of a better word.  For example, a project where you build a
> program would have prog-mode or some of its descendants in the list,
> and perhaps also Texinfo and/or markdown for docs.  A project whose
> purpose is to write an essay or a blog or might use descendants of
> text-mode.  Etc., etc.

Yes, absolutely. But isn't this sort of what is intended with the
cl-defmethods used to assign a project?

I believe project.el to be a bit ambiguous, since it is both a generic
"wrapper" for a generic interface "project", but also an implementation of
the "vc" version of this interface. So the behaviour you suggest here is
easily done in say:

(cl-defmethod project-root ((project (head some-major-mode)))
  ...)

The version used now is merely saying: "Show me everything but the
things ignored in .gitignore."

Or am I misunderstanding you?

Theo

Re: master 1e3b0f2: Improve doc strings of project.el

[Eli Zaretskii]

> Date: Fri, 19 Jun 2020 15:39:53 +0000
> From: Theodor Thornhill <theothornhill@pm.me>
> Cc: philip@warpmail.net, dgutov@yandex.ru, emacs-devel@gnu.org
> 
> > A project could specify a list of major-modes, couldn't it?  That
> > would generally derive from the project's "purpose" or "goal", for
> > lack of a better word.  For example, a project where you build a
> > program would have prog-mode or some of its descendants in the list,
> > and perhaps also Texinfo and/or markdown for docs.  A project whose
> > purpose is to write an essay or a blog or might use descendants of
> > text-mode.  Etc., etc.
> 
> Yes, absolutely. But isn't this sort of what is intended with the
> cl-defmethods used to assign a project?
> 
> I believe project.el to be a bit ambiguous, since it is both a generic
> "wrapper" for a generic interface "project", but also an implementation of
> the "vc" version of this interface. So the behaviour you suggest here is
> easily done in say:
> 
> (cl-defmethod project-root ((project (head some-major-mode)))
>   ...)
> 
> The version used now is merely saying: "Show me everything but the
> things ignored in .gitignore."
> 
> Or am I misunderstanding you?

Maybe.  I don't think I understand what you are suggesting in enough
detail to tell.

Re: master 1e3b0f2: Improve doc strings of project.el

[Theodor Thornhill]

>>
>> Or am I misunderstanding you?
>
> Maybe.  I don't think I understand what you are suggesting in enough
> detail to tell.

For a project of any kind (vc, major-mode-based, or any concept you can
think of) to decide what buffers are useful we need some set of
information about the project. If as you say, a text-mode based project
only should show buffers of say, org-mode, markdown etc, and exclude
buffers generated by org-babel or something, we need to place that
knowledge somewhere.

This knowledge is in part stored in 'project-ignores', where we can
ignore patterns such as "assets" "node_modules" etc. without it needing
to be put in a .gitignore [1]. However, then we lose the optimized search
functionalities offered by the "(head vc)" implementation in project.el.

If for instance the project-switch-to-buffer should have to know
everything about every imaginable combination of interesting derivates
of a project, then I believe that would be a quite a bit larger
function (or at least a huge curated map of sorts).

What I am thinking, surely without knowing all the intricacies here, is
that maybe the "switch-to-buffer" and "kill-buffers" also should be
defgenerics? Then the implementation of what to show can be put off for
later, and the "(head vc)" version can be decided now?

Otherwise, I guess the user has to specify a set of patterns, or at
least decide on major-modes to include, and that would seem like a
hassle to me.

I may be overthinking this, but it would be nice if "switch-to-buffer"
only would be a "brainless" buffer-picker with only useful choices.

Theo


[1]: If we choose to implement our own notion of a project.

Re: master 1e3b0f2: Improve doc strings of project.el

[Eli Zaretskii]

> Date: Fri, 19 Jun 2020 17:46:36 +0000
> From: Theodor Thornhill <theothornhill@pm.me>
> Cc: dgutov@yandex.ru, emacs-devel@gnu.org
> 
> If for instance the project-switch-to-buffer should have to know
> everything about every imaginable combination of interesting derivates
> of a project, then I believe that would be a quite a bit larger
> function (or at least a huge curated map of sorts).

What I had in mind was a simple filtering by major mode, not unlike
the current filtering by default-directory.  How complicate could that
be?

> What I am thinking, surely without knowing all the intricacies here, is
> that maybe the "switch-to-buffer" and "kill-buffers" also should be
> defgenerics? Then the implementation of what to show can be put off for
> later, and the "(head vc)" version can be decided now?

I don't see how this is relevant.  Whether the function we are talking
about is switch-to-buffer or project-switch-to-buffer, the problems it
will need to solve are the same, and the questions we need to answer
are the same.

My point was and remains that using default-directory as the single
criterion for deciding whether a buffer should be offered as
completion candidate to switch while working on a project, sounds like
not the best idea, since it will pick up many buffers to which users
are unlikely to want to switch as part of working on the project.

> Otherwise, I guess the user has to specify a set of patterns, or at
> least decide on major-modes to include, and that would seem like a
> hassle to me.

That is a one-time decision, when you start a project and decide what
kind of a project will it be.

Re: master 1e3b0f2: Improve doc strings of project.el

[Dmitry Gutov]

On 19.06.2020 21:03, Eli Zaretskii wrote:
>> If for instance the project-switch-to-buffer should have to know
>> everything about every imaginable combination of interesting derivates
>> of a project, then I believe that would be a quite a bit larger
>> function (or at least a huge curated map of sorts).
> What I had in mind was a simple filtering by major mode, not unlike
> the current filtering by default-directory.  How complicate could that
> be?

That's not how it works. A project is defined by the generic functions 
the API already has.

Even if some projects would prefer to work with files with certain 
extensions only (doesn't sound like a good idea to me), that would be an 
extra, new feature. As such, it is irrelevant to this discussion.

Re: master 1e3b0f2: Improve doc strings of project.el

[Dmitry Gutov]

On 19.06.2020 18:39, Theodor Thornhill wrote:
> I believe project.el to be a bit ambiguous, since it is both a generic
> "wrapper" for a generic interface "project", but also an implementation of
> the "vc" version of this interface. So the behaviour you suggest here is
> easily done in say:
> 
> (cl-defmethod project-root ((project (head some-major-mode)))
>    ...)
> 
> The version used now is merely saying: "Show me everything but the
> things ignored in .gitignore."

Let's not do that.

In any case, this doesn't help us to define what 
project-switch-to-buffer should do now. It would only complicate things.

Re: master 1e3b0f2: Improve doc strings of project.el

[Theodor Thornhill]

> My point was and remains that using default-directory as the single
> criterion for deciding whether a buffer should be offered as
> completion candidate to switch while working on a project, sounds like
> not the best idea, since it will pick up many buffers to which users
> are unlikely to want to switch as part of working on the project.
Yeah, I agree.  I can try to come up with something more appropriate
than the default-directory. Though I for one, want the *xref* to be part
of that, since "project-find-regexp" uses it. I believe I read you
wouldn't want it. I guess many others will have many different opinions
in this matter.

>> Otherwise, I guess the user has to specify a set of patterns, or at
>> least decide on major-modes to include, and that would seem like a
>> hassle to me.
>
> That is a one-time decision, when you start a project and decide what
> kind of a project will it be.
I see your point, though that seems to imply a little foresight from the
users. Also, I am not sure every emacs users are aware of what modes
inherits from what modes etc. 

In addition, when do we make that first desicion, really? Right now it
just sort of happens after you go "git init" in terminal. To me it
sounds like that new workflow adds some steps to what is right now a
quite seamless experience.

Theo

Re: master 1e3b0f2: Improve doc strings of project.el

[Dmitry Gutov]

On 19.06.2020 18:13, Eli Zaretskii wrote:

>> It's not the worst idea, but the set of "buffers belonging to the
>> current project" should probably be the same between
>> project-switch-to-buffer and project-kill-buffers.
> 
> No, not necessarily.  The purpose of the commands is different, so
> they could start with the same basic list, but apply different filters
> to it.

There can be subtle differences (e.g. ones stemming from 
project-kill-buffers-skip-conditions), but the general notion of what it 
mean to have a buffer belong to a project should be generally the same.

>> And as Philip explained, it's useful for the latter to consider many
>> different major modes.
> 
> I'm talking about the former, not about the latter.

And I am talking about both.

Re: master 1e3b0f2: Improve doc strings of project.el

[Theodor Thornhill]

"Dmitry Gutov" <dgutov@yandex.ru> writes:

> On 19.06.2020 18:39, Theodor Thornhill wrote:
>> I believe project.el to be a bit ambiguous, since it is both a generic
>> "wrapper" for a generic interface "project", but also an implementation of
>> the "vc" version of this interface. So the behaviour you suggest here is
>> easily done in say:
>>
>> (cl-defmethod project-root ((project (head some-major-mode)))
>>    ...)
>>
>> The version used now is merely saying: "Show me everything but the
>> things ignored in .gitignore."
>
> Let's not do that.

FWIW, I agree

Re: master 1e3b0f2: Improve doc strings of project.el

[Dmitry Gutov]

On 19.06.2020 18:11, Eli Zaretskii wrote:

>>   >>>      Switch to another buffer related to the current project.
>>   >>>    A buffer is related to a project if its `default-directory'
>>   >>>    is inside the directory hierarchy of the project's root.
>>   >>
>>   >> Sounds good. Except some projects can be inside one another, too.
>>
>>   > Does that invalidate what the text above says?
>>
>> If the current project contains a few other projects inside of its
>> subdirectories, project-switch-to-buffer probably should offer any of
>> "their" buffers for switching.
> 
> "Should" or "should not"?  If the former, then the text I proposed is
> fine as it is.

Sorry, should not.

But it's an edge case. So we could take your description now. But we'd 
need to change it when/if that edge case is fixed. The fix isn't 
difficult, I'm just unsure about its performance characteristics.

>> As soon as you start relying on completion hints, you don't need to
>> remember the names in advance. And it's also a good overview of the
>> current open buffers in the current project.
> 
> If this is just for completion's sake, then I could understand, but
> still it sounds very strange to include stuff like *XREF*, Eshell,
> Occur, *Compilation*, *Grep*, etc.  Those are basically singletons, at
> least by default,

Not really. In most of these cases, such a buffer shows results 
pertaining to a specific project, so its contents are basically 
irrelevant when a user is working on a different one.

Occur can be an exception because multi-occur can work across buffer 
(and thus, projects). Help buffers contain history, so they are special 
too. But the rest usually fall in with that rule. And there are also 
derived compilation modes, which pertain to specific projects even more 
(e.g. they contain test runner outputs).

> and a simple "C-x b" should be enough to switch to
> them.

'C-x b' can be used in a lot of cases, but what I have seen requested 
here, it being able to do all work on a project (or as much as possible) 
while only using 'C-x p b'. It makes some sense to me, and I think 
Projectile has proven the demand for such command.

> project-switch-to-buffer should be more helpful by offering
> only "useful" buffers.

That is indeed an option, but Andrii requested a different behavior. 
Please see the original discussion.

>> Also, different projects can have files with conflicting names.
> 
> That cannot be solved by completion, where only the basename is
> usually shown.

It can, if we ask Emacs to only show the buffers from the current 
project. Then you don't need to worry about conflicting with other projects.

Re: master 1e3b0f2: Improve doc strings of project.el

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Fri, 19 Jun 2020 21:19:37 +0300
> 
> > What I had in mind was a simple filtering by major mode, not unlike
> > the current filtering by default-directory.  How complicate could that
> > be?
> 
> That's not how it works. A project is defined by the generic functions 
> the API already has.
> 
> Even if some projects would prefer to work with files with certain 
> extensions only (doesn't sound like a good idea to me), that would be an 
> extra, new feature. As such, it is irrelevant to this discussion.

You have completely missed my point.  But since you say this is
irrelevant, I won't say more.

Re: master 1e3b0f2: Improve doc strings of project.el

[Eli Zaretskii]

> Date: Fri, 19 Jun 2020 18:22:48 +0000
> From: Theodor Thornhill <theothornhill@pm.me>
> Cc: dgutov@yandex.ru, emacs-devel@gnu.org
> 
> > My point was and remains that using default-directory as the single
> > criterion for deciding whether a buffer should be offered as
> > completion candidate to switch while working on a project, sounds like
> > not the best idea, since it will pick up many buffers to which users
> > are unlikely to want to switch as part of working on the project.
> Yeah, I agree.  I can try to come up with something more appropriate
> than the default-directory. Though I for one, want the *xref* to be part
> of that, since "project-find-regexp" uses it.

Why does it matter that project-find-regexp uses *XREF*?  the question
is why would you need to switch to it when it is not displayed in any
window?

> I believe I read you wouldn't want it. I guess many others will have
> many different opinions in this matter.

When I work with xref-related features, I keep the *XREF* buffer on
display as long as I use it.  When I'm done using it, I don't care
about it anymore.

In any case, there's just one such buffer, so switching to it is easy.

> In addition, when do we make that first desicion, really? Right now it
> just sort of happens after you go "git init" in terminal. To me it
> sounds like that new workflow adds some steps to what is right now a
> quite seamless experience.

That we mainly support projects that piggy-back VCS repositories
doesn't mean that's the only kind of project that could be useful.

And even if the backend is Git, no one said that I want all of the
files in the repository to be part of my project.  I may be working on
a project that includes just a handful of files from a repository that
has thousands of them.

Re: master 1e3b0f2: Improve doc strings of project.el

[Eli Zaretskii]

> Cc: theo@thornhill.no, philip@warpmail.net, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Fri, 19 Jun 2020 21:23:42 +0300
> 
> On 19.06.2020 18:13, Eli Zaretskii wrote:
> 
> >> It's not the worst idea, but the set of "buffers belonging to the
> >> current project" should probably be the same between
> >> project-switch-to-buffer and project-kill-buffers.
> > 
> > No, not necessarily.  The purpose of the commands is different, so
> > they could start with the same basic list, but apply different filters
> > to it.
> 
> There can be subtle differences (e.g. ones stemming from 
> project-kill-buffers-skip-conditions), but the general notion of what it 
> mean to have a buffer belong to a project should be generally the same.

"Generally", yes.  But your objection above was based on the premise
that they are identical.  If they are similar, then there's no
problem.

> >> And as Philip explained, it's useful for the latter to consider many
> >> different major modes.
> > 
> > I'm talking about the former, not about the latter.
> 
> And I am talking about both.

Are we still talking about the doc string of project-switch-to-buffer?
If we are, then the latter is a separate discussion.

Re: master 1e3b0f2: Improve doc strings of project.el

[Dmitry Gutov]

On 19.06.2020 21:36, Eli Zaretskii wrote:
>>> What I had in mind was a simple filtering by major mode, not unlike
>>> the current filtering by default-directory.  How complicate could that
>>> be?
>> That's not how it works. A project is defined by the generic functions
>> the API already has.
>>
>> Even if some projects would prefer to work with files with certain
>> extensions only (doesn't sound like a good idea to me), that would be an
>> extra, new feature. As such, it is irrelevant to this discussion.
> You have completely missed my point.  But since you say this is
> irrelevant, I won't say more.

Perhaps. So let me clarify a little.

What I originally though you meant, it adding a filter by major modes 
for non-file-visiting buffers. That kind of makes sense, but that kind 
of filter shouldn't depend on the project in question (it could, but 
that, again, would make things more complicated; so let's work with the 
current model first).

On the flip side, *some* projects could specify that they only work with 
certain major modes. But a lot of projects won't specify that. And we 
need these commands to work in both cases. So discussing the projects of 
the former kind is, again, a complication, and I'd rather we did that 
later in a separate discussion.

Re: master 1e3b0f2: Improve doc strings of project.el

[Dmitry Gutov]

On 19.06.2020 21:44, Eli Zaretskii wrote:
>>>> It's not the worst idea, but the set of "buffers belonging to the
>>>> current project" should probably be the same between
>>>> project-switch-to-buffer and project-kill-buffers.
>>> No, not necessarily.  The purpose of the commands is different, so
>>> they could start with the same basic list, but apply different filters
>>> to it.
>> There can be subtle differences (e.g. ones stemming from
>> project-kill-buffers-skip-conditions), but the general notion of what it
>> mean to have a buffer belong to a project should be generally the same.
> "Generally", yes.  But your objection above was based on the premise
> that they are identical.  If they are similar, then there's no
> problem.

If we agree on that, then we probably agree that the docstrings of those 
functions should also be similar to a certain degree.

Re: master 1e3b0f2: Improve doc strings of project.el

[Theodor Thornhill]

"Eli Zaretskii" <eliz@gnu.org> writes:

> Why does it matter that project-find-regexp uses *XREF*?  the question
> is why would you need to switch to it when it is not displayed in any
> window?
I don't always use a very big computer screen, so sometimes I don't
split the screen.
>
>> I believe I read you wouldn't want it. I guess many others will have
>> many different opinions in this matter.
>
> When I work with xref-related features, I keep the *XREF* buffer on
> display as long as I use it.  When I'm done using it, I don't care
> about it anymore.
Same here, though I have wanted to jump back to one. Not always, but
certainly not never.

> In any case, there's just one such buffer, so switching to it is easy.
Yeah, sure. I could use "switch-to-buffer" for that, but then I have to
context switch only a little. It isn't the end of the world at all, but
I'd like to have it there. (Though I didn't include it until Andrii
mentioned it)
>
>> In addition, when do we make that first desicion, really? Right now it
>> just sort of happens after you go "git init" in terminal. To me it
>> sounds like that new workflow adds some steps to what is right now a
>> quite seamless experience.
>
> That we mainly support projects that piggy-back VCS repositories
> doesn't mean that's the only kind of project that could be useful.
I definitively agree here.

> And even if the backend is Git, no one said that I want all of the
> files in the repository to be part of my project.  I may be working on
> a project that includes just a handful of files from a repository that
> has thousands of them.
True. I believe this is not yet supported fully by the api. For example,
a huge monorepo with .git only in root is a struggle.

Theo

Re: master 1e3b0f2: Improve doc strings of project.el

[Eli Zaretskii]

> Cc: theo@thornhill.no, philip@warpmail.net, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Fri, 19 Jun 2020 21:33:34 +0300
> 
> >> If the current project contains a few other projects inside of its
> >> subdirectories, project-switch-to-buffer probably should offer any of
> >> "their" buffers for switching.
> > 
> > "Should" or "should not"?  If the former, then the text I proposed is
> > fine as it is.
> 
> Sorry, should not.
> 
> But it's an edge case. So we could take your description now. But we'd 
> need to change it when/if that edge case is fixed. The fix isn't 
> difficult, I'm just unsure about its performance characteristics.

I'm not sure that the doc string will have to be changed (or how) when
you fix that, but that's a separate issue.

> >> As soon as you start relying on completion hints, you don't need to
> >> remember the names in advance. And it's also a good overview of the
> >> current open buffers in the current project.
> > 
> > If this is just for completion's sake, then I could understand, but
> > still it sounds very strange to include stuff like *XREF*, Eshell,
> > Occur, *Compilation*, *Grep*, etc.  Those are basically singletons, at
> > least by default,
> 
> Not really. In most of these cases, such a buffer shows results 
> pertaining to a specific project, so its contents are basically 
> irrelevant when a user is working on a different one.

But the default-directory of these buffers is a very weak evidence of
them being relevant to some project.  E.g., I could (and many times
do) grep some other project when I need to see how it solves some
problem which is relevant to what I'm working on now.  If we really
want to record in these buffers what project they are related to, we
need to have stronger evidence, like what was the current-buffer when
the command was invoked, or maybe something else (like name the
buffers in some special way).

> Occur can be an exception because multi-occur can work across buffer 
> (and thus, projects). Help buffers contain history, so they are special 
> too. But the rest usually fall in with that rule.

"The rest" are a legion.  I suspect that many/most of them are like
Occur and Help.  We should audit them before we make such far-reaching
conclusions (unless someone already did).

> > project-switch-to-buffer should be more helpful by offering
> > only "useful" buffers.
> 
> That is indeed an option, but Andrii requested a different behavior. 

That's fine, but Andrii's opinion is not the only one that counts, is
it?  We are talking about a general-purpose Emacs feature, not about a
command that will only be used by a single person.  It should make
sense to all of us.

Re: master 1e3b0f2: Improve doc strings of project.el

[Dmitry Gutov]

On 19.06.2020 21:41, Eli Zaretskii wrote:
> When I work with xref-related features, I keep the*XREF*  buffer on
> display as long as I use it.  When I'm done using it, I don't care
> about it anymore.
> 
> In any case, there's just one such buffer, so switching to it is easy.

For singleton buffers, we could conceivably go either way, but recall 
what I said about people wanting to be able to predominantly use 
project-switch-to-buffer while working on a project.

There are other kinds of buffers not tied to files that are not always 
singleton. E.g. rspec-mode creates compilation-based buffers that can be 
(depending on a user option) created per-project. And even when it's 
singleton, that particular buffer is only relevant to one project 
usually (the one that ran the tests most recently).

> That we mainly support projects that piggy-back VCS repositories
> doesn't mean that's the only kind of project that could be useful.

True.

> And even if the backend is Git, no one said that I want all of the
> files in the repository to be part of my project.  I may be working on
> a project that includes just a handful of files from a repository that
> has thousands of them.

That is, again, true. But what of the rest of files in that repository? 
Do they belong to a different project? Do you also visit them, or not? 
If both answers are "no", then the current approach should work well 
enough. Otherwise, you're welcome to file a new bug report with more 
details, and we'll see what can be improved to accommodate that 
situation too.

Re: master 1e3b0f2: Improve doc strings of project.el

[Eli Zaretskii]

> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Fri, 19 Jun 2020 21:49:59 +0300
> Cc: philip@warpmail.net, theo@thornhill.no, emacs-devel@gnu.org
> 
> If we agree on that, then we probably agree that the docstrings of those 
> functions should also be similar to a certain degree.

They are.  But if you think they should be more so, and would like to
add to the doc string of project-kill-buffers text similar to what I
wrote for project-switch-to-buffer, feel free, I won't object.

Re: master 1e3b0f2: Improve doc strings of project.el

[Dmitry Gutov]

On 19.06.2020 21:57, Theodor Thornhill wrote:
>> And even if the backend is Git, no one said that I want all of the
>> files in the repository to be part of my project.  I may be working on
>> a project that includes just a handful of files from a repository that
>> has thousands of them.
> True. I believe this is not yet supported fully by the api. For example,
> a huge monorepo with .git only in root is a struggle.

I suppose it could be fixed with whitelisting approach to 
project-ignores (feature planned).

But that depends on how you can/want to define such a project. We'll 
need examples. And a new bug report, probably.

Re: master 1e3b0f2: Improve doc strings of project.el

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Fri, 19 Jun 2020 22:04:49 +0300
> 
> > And even if the backend is Git, no one said that I want all of the
> > files in the repository to be part of my project.  I may be working on
> > a project that includes just a handful of files from a repository that
> > has thousands of them.
> 
> That is, again, true. But what of the rest of files in that repository? 
> Do they belong to a different project?

Yes.

> Do you also visit them, or not?

Not as part of working on the current project.  Though I might decide
to add some of them to the current project, as the need arises.

> If both answers are "no", then the current approach should work well 
> enough. Otherwise, you're welcome to file a new bug report with more 
> details, and we'll see what can be improved to accommodate that 
> situation too.

Which details are needed?

In any case, my point was that relying on the VCS to determine what is
and isn't in the project presents a restriction that a general-purpose
project-handling facility should lift early on to be useful, IMO.

Re: master 1e3b0f2: Improve doc strings of project.el

[Dmitry Gutov]

On 19.06.2020 22:01, Eli Zaretskii wrote:

>> But it's an edge case. So we could take your description now. But we'd
>> need to change it when/if that edge case is fixed. The fix isn't
>> difficult, I'm just unsure about its performance characteristics.
> 
> I'm not sure that the doc string will have to be changed (or how) when
> you fix that, but that's a separate issue.

The fix would look like this:

diff --git a/lisp/progmodes/project.el b/lisp/progmodes/project.el
index 89dcee97fa..d4eab9cd58 100644
--- a/lisp/progmodes/project.el
+++ b/lisp/progmodes/project.el
@@ -777,7 +777,7 @@ project-compile
  (defun project-switch-to-buffer ()
    "Switch to a buffer in the current project."
    (interactive)
-  (let* ((root (project-root (project-current t)))
+  (let* ((project (project-current t))
           (current-buffer (current-buffer))
           (other-buffer (other-buffer current-buffer))
           (other-name (buffer-name other-buffer))
@@ -785,9 +785,13 @@ project-switch-to-buffer
            (lambda (buffer)
              ;; BUFFER is an entry (BUF-NAME . BUF-OBJ) of Vbuffer_alist.
              (and (not (eq (cdr buffer) current-buffer))
-                 (when-let ((file (buffer-local-value 'default-directory
-                                                      (cdr buffer))))
-                   (file-in-directory-p file root))))))
+                 (cdr buffer)
+                 (when-let ((other-pr
+                             (project-current
+                              nil
+                              (buffer-local-value 'default-directory
+                                                  (cdr buffer)))))
+                   (equal project other-pr))))))
      (switch-to-buffer
       (read-buffer
        "Switch to buffer: "

It's reasonably fast here when there are not many open buffers, but I 
fear the accuracy improvement from it might not outweigh the performance 
problems that will come up in the case of a lot of buffers.

There can be a different approach, though. To define a new generic like 
project-contains-p.

>> Not really. In most of these cases, such a buffer shows results
>> pertaining to a specific project, so its contents are basically
>> irrelevant when a user is working on a different one.
> 
> But the default-directory of these buffers is a very weak evidence of
> them being relevant to some project.  E.g., I could (and many times
> do) grep some other project when I need to see how it solves some
> problem which is relevant to what I'm working on now.

That is a case of working on multiple projects at once. And whatever 
your purpose of grepping it, wouldn't you agree that that grep buffer is 
probably closer to the "other" project rather that to the current one?

To reuse your argument, 'M-x switch-to-buffer' is still available for 
borderline cases.

> If we really
> want to record in these buffers what project they are related to, we
> need to have stronger evidence, like what was the current-buffer when
> the command was invoked, or maybe something else (like name the
> buffers in some special way).

We would start with strong counter-examples.

Generally, though, new buffers inherit default-directory from the 
buffers that created them. So default-directory is not the worst indicator.

>> Occur can be an exception because multi-occur can work across buffer
>> (and thus, projects). Help buffers contain history, so they are special
>> too. But the rest usually fall in with that rule.
> 
> "The rest" are a legion.  I suspect that many/most of them are like
> Occur and Help.  We should audit them before we make such far-reaching
> conclusions (unless someone already did).

I think some people here already have some experience working with 
Projectile and its projectile-switch-to-buffer, which exposes the 
current behavior by default.

But we should also try the new command out and document our experiences.

>>> project-switch-to-buffer should be more helpful by offering
>>> only "useful" buffers.
>>
>> That is indeed an option, but Andrii requested a different behavior.
> 
> That's fine, but Andrii's opinion is not the only one that counts, is
> it?  We are talking about a general-purpose Emacs feature, not about a
> command that will only be used by a single person.  It should make
> sense to all of us.

So far Theodor agreed too. And myself as well. You alone have voiced 
disagreement. With this distribution of votes, it seems the default 
behavior should default to including non-file-visiting buffers (with 
some agreed-upon exceptions). And, of course, we can add a user option 
that would allow to tweak the choosing logic.

Re: master 1e3b0f2: Improve doc strings of project.el

[Dmitry Gutov]

On 19.06.2020 22:12, Eli Zaretskii wrote:

>> That is, again, true. But what of the rest of files in that repository?
>> Do they belong to a different project?
> 
> Yes.
> 
>> Do you also visit them, or not?
> 
> Not as part of working on the current project.  Though I might decide
> to add some of them to the current project, as the need arises.

I mean, do you visit them in the same Emacs session?

>> If both answers are "no", then the current approach should work well
>> enough. Otherwise, you're welcome to file a new bug report with more
>> details, and we'll see what can be improved to accommodate that
>> situation too.
> 
> Which details are needed?

How is the current project defined. How is the "other" project defined. 
Something like that.

> In any case, my point was that relying on the VCS to determine what is
> and isn't in the project presents a restriction that a general-purpose
> project-handling facility should lift early on to be useful, IMO.

I agree, but agreement alone is not enough to make it a reality.

OTOH, we already have a customization point that allows to exclude more 
files than .gitignore does (the project-vc-ignores variable).

*And* one can use the project API to introduce a project backend that 
does not rely on VC repositories. That's just not trivial, is all.

Re: master 1e3b0f2: Improve doc strings of project.el

[theo]

> But that depends on how you can/want to define such a project. We'll
> need examples. And a new bug report, probably.


I'll add a bug report for the troubles I had with that, and my ad-hoc
solution at the time.

Re: master 1e3b0f2: Improve doc strings of project.el

[Eli Zaretskii]

> Cc: theo@thornhill.no, philip@warpmail.net, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Fri, 19 Jun 2020 22:28:39 +0300
> 
> > I'm not sure that the doc string will have to be changed (or how) when
> > you fix that, but that's a separate issue.
> 
> The fix would look like this:

So all you want to add to the current doc string is "...and doesn't
belong to some other project"?

> > But the default-directory of these buffers is a very weak evidence of
> > them being relevant to some project.  E.g., I could (and many times
> > do) grep some other project when I need to see how it solves some
> > problem which is relevant to what I'm working on now.
> 
> That is a case of working on multiple projects at once.

No, it isn't.  I'm working on a single project, but need to look
outside of its directory to find some information.  A very natural
thing to do, and it doesn't mean I started working on another
project.  More importantly, I do want that Grep buffer be available to
me as part of the current project, because I'm likely to return there
more than once.

> And whatever your purpose of grepping it, wouldn't you agree that
> that grep buffer is probably closer to the "other" project rather
> that to the current one?

No.  And that "other" project doesn't even have to be a "project" in
the project.el sense of the word.

> To reuse your argument, 'M-x switch-to-buffer' is still available for 
> borderline cases.

An argument that you dismissed previously.

> > If we really
> > want to record in these buffers what project they are related to, we
> > need to have stronger evidence, like what was the current-buffer when
> > the command was invoked, or maybe something else (like name the
> > buffers in some special way).
> 
> We would start with strong counter-examples.

I just gave you one.  You want to dismiss it as "not strong", but I
think you are making a mistake.  Search-like activities are
ubiquitous, and not confined to the project's directory tree.

I think that non file-visiting buffers are rarely related to a
project, an exception rather than a rule.  My suggestion is therefore
to turn the table and come up with a list of such buffers that
_always_ are related to a specific project.  And instead of using the
default-directory as evidence for the buffer's relevance, we may need
a command that explicitly makes a buffer related to a project.

> So default-directory is not the worst indicator.

I'm saying it isn't the best, either.  We have just discovered at
least 2 problems with it.  We should try to find a better one.

> But we should also try the new command out and document our experiences.

IMNSHO, some thought is required even before we hope the experience
will teach us in due time.  Doing this only by trial and error runs
the risk of converging on a design that is found later to be
restrictive, or one that cannot be easily extended to support
behaviors not accounted for or envisioned originally.

We are just at the beginning of development here, and so this is the
right place for thinking about possible project-related activities,
and how they could be supported by different ways of determining which
buffers are related to a project and which aren't.  A simple solution
we decide on now could make future development and extension much
easier.

> > That's fine, but Andrii's opinion is not the only one that counts, is
> > it?
> 
> So far Theodor agreed too. And myself as well. You alone have voiced 
> disagreement. With this distribution of votes, it seems the default 
> behavior should default to including non-file-visiting buffers (with 
> some agreed-upon exceptions).

For an opinion to "count", it doesn't have to _replace_ the other
opinions.  It could be taken into consideration by augmenting the
design so that it supports both kinds of behaviors.  This is IME
better than flatly discarding the dissenting opinions.

> And, of course, we can add a user option that would allow to tweak
> the choosing logic.

Sub-optimal selection of the "belongs to a project" criteria will make
any such user options cumbersome and hard to use.  IOW, user options
shouldn't be considered as means to "fix" sub-optimal design.  We are
at a point where we can make the design better, if we consider a wider
variety of project-related activities.

Re: master 1e3b0f2: Improve doc strings of project.el

[Eli Zaretskii]

> Cc: theothornhill@pm.me, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Fri, 19 Jun 2020 22:33:07 +0300
> 
> On 19.06.2020 22:12, Eli Zaretskii wrote:
> 
> >> That is, again, true. But what of the rest of files in that repository?
> >> Do they belong to a different project?
> > 
> > Yes.
> > 
> >> Do you also visit them, or not?
> > 
> > Not as part of working on the current project.  Though I might decide
> > to add some of them to the current project, as the need arises.
> 
> I mean, do you visit them in the same Emacs session?

I just answered that above.  This is all in a single Emacs session.

> > In any case, my point was that relying on the VCS to determine what is
> > and isn't in the project presents a restriction that a general-purpose
> > project-handling facility should lift early on to be useful, IMO.
> 
> I agree, but agreement alone is not enough to make it a reality.
> 
> OTOH, we already have a customization point that allows to exclude more 
> files than .gitignore does (the project-vc-ignores variable).

I don't think exclusion alone is enough.  We need also a way of
_including_ files in a project.

> *And* one can use the project API to introduce a project backend that 
> does not rely on VC repositories.

I think we should have commands to do so in the core.  It's too basic
a capability for any IDE for us to leave it to add-ons.

Re: master 1e3b0f2: Improve doc strings of project.el

[Eli Zaretskii]

> From: "Philip K." <philip@warpmail.net>
> Cc: dgutov@yandex.ru, emacs-devel@gnu.org
> Date: Fri, 19 Jun 2020 16:41:03 +0200
> 
> > More generally, I question the wisdom of judging by the
> > default-directory in the first place.  Can't we come up with a better,
> > more specific criterion?
> 
> My rule of thumb would be was a buffer generated "by" a file/action on
> file in the project.

The question is: how to express this in Emacs terms that can be used
to code this?

Re: master 1e3b0f2: Improve doc strings of project.el

[Theodor Thornhill]

[-- Attachment #1: Type: text/plain, Size: 994 bytes --]

> No, it isn't. I'm working on a single project, but need to look
> outside of its directory to find some information. A very natural
> thing to do, and it doesn't mean I started working on another
> project. More importantly, I do want that Grep buffer be available to
> me as part of the current project, because I'm likely to return there
> more than once.

What if we offer this workflow:

- the git implementation of the backend offers only file bound alternatives. To me that seems kind of reasonable, since its task is to track actual files and ignores.

- in addition, we offer something similar to org-add-to-agenda-files. Then my *xref*, Andriis inferior python, shells and whatever you want to add can be added.

- obviously we then need an option to delete files from that new list

I imagine this being implemented just with a new defvar- project-additional-buffers, or something like that.

Then switch to buffer just adds them as well.

Could something like this be viable?

Theo

[-- Attachment #2: Type: text/html, Size: 1358 bytes --]

Re: master 1e3b0f2: Improve doc strings of project.el

[Eli Zaretskii]

> Date: Sat, 20 Jun 2020 07:43:57 +0000
> From: Theodor Thornhill <theothornhill@pm.me>
> Cc: philip@warpmail.net, emacs-devel@gnu.org
> 
> What if we offer this workflow:
> 
> - the git implementation of the backend offers only file bound  alternatives. To me that seems kind of
> reasonable, since its task is to track actual files and ignores. 
> 
> - in addition, we offer something similar to org-add-to-agenda-files. Then my *xref*, Andriis inferior python,
> shells and whatever you want to add can be added. 
> 
> - obviously we then need an option to delete files from that new list
> 
> I imagine this being implemented just with a new defvar- project-additional-buffers, or something like that. 
> 
> Then switch to buffer just adds them as well. 
> 
> Could something like this be viable?

I proposed to find a different way of telling which buffer is related
to a project, other than by looking at its default-directory.  We
could design a way of doing that which would then support adding
buffers to a project quite easily.  But Dmitry doesn't think we should
go that way.

More to the point, my question is how would adding a buffer to the
project be recorded and where, under your proposal?

Re: master 1e3b0f2: Improve doc strings of project.el

[Theodor Thornhill]

>> Could something like this be viable?
>
> I proposed to find a different way of telling which buffer is related
> to a project, other than by looking at its default-directory.  

My proposal here isn't using default-directory.  It would then check
(buffer-file-name (cdr buffer)) in project-root.  As such it would
exclude other 'junk' buffers.  After, the persisted other buffers are
included as well.

> We could design a way of doing that which would then support adding
> buffers to a project quite easily.  But Dmitry doesn't think we should
> go that way.

Maybe we should reconsider the API then?  It seems to me like you don't
want this generic interface, and rather would like a more general notion
of what a project is.  You want to have a more granular approach,
something akin to bookmark.el.  Am I reading you correctly here?

> More to the point, my question is how would adding a buffer to the
> project be recorded and where, under your proposal?

We already are persisting a project list under "user-emacs-directory" in
a readable lisp format.  I guess this could be extended to include the
added buffers?  Not sure how unwieldy this can get, though.

Re: master 1e3b0f2: Improve doc strings of project.el

[Eli Zaretskii]

> Date: Sat, 20 Jun 2020 09:29:45 +0000
> From: Theodor Thornhill <theo@thornhill.no>
> Cc: dgutov@yandex.ru, philip@warpmail.net, emacs-devel@gnu.org
> 
> > We could design a way of doing that which would then support adding
> > buffers to a project quite easily.  But Dmitry doesn't think we should
> > go that way.
> 
> Maybe we should reconsider the API then?  It seems to me like you don't
> want this generic interface, and rather would like a more general notion
> of what a project is.  You want to have a more granular approach,
> something akin to bookmark.el.  Am I reading you correctly here?

Which API did you have in mind?

In any case, my proposal was not about the API itself, it was more
about the implementation of the API.  For example, we could have an
implementation of the generic project-files API that consulted some
list instead of asking the VCS or the filesystem to provide the files.

> > More to the point, my question is how would adding a buffer to the
> > project be recorded and where, under your proposal?
> 
> We already are persisting a project list under "user-emacs-directory" in
> a readable lisp format.  I guess this could be extended to include the
> added buffers?  Not sure how unwieldy this can get, though.

That's one possibility, yes.

Re: master 1e3b0f2: Improve doc strings of project.el

[Theodor Thornhill]

> Which API did you have in mind?
>
> In any case, my proposal was not about the API itself, it was more
> about the implementation of the API.  For example, we could have an
> implementation of the generic project-files API that consulted some
> list instead of asking the VCS or the filesystem to provide the files.
>

Yeah, this is kind of what I was thinking with the
'project-additional-buffers' idea.  This could be implementation
agnostic as well, and be just a flat list connected to some root,
persisted in the project list.  As such, this will always be consulted
when invoking "project-find-files", "project-switch-to-buffer" etc.
Then we are free to implement and choose the "main" implementation of
the API, most likely at first the VC version, since it is the fastest
one.  Later, other notions of projects could be added.  However, I don't
want to step too far into Dmitrys domain without him chiming in.

Re: master 1e3b0f2: Improve doc strings of project.el

[Dmitry Gutov]

On 20.06.2020 09:43, Eli Zaretskii wrote:

> So all you want to add to the current doc string is "...and doesn't
> belong to some other project"?

I don't know about that. Interpreting that literally would mean that 
such buffers would never be suggested because they "belong to some other 
project" (according to the reasoning which gave rise to this wording) no 
matter which of the projects is the current one: "the outer", or "the 
inner".

So what we should say is just "belongs to the current project". The 
implication being that if it's inside a nested project, it doesn't 
belong to the outer one.

>>> But the default-directory of these buffers is a very weak evidence of
>>> them being relevant to some project.  E.g., I could (and many times
>>> do) grep some other project when I need to see how it solves some
>>> problem which is relevant to what I'm working on now.
>>
>> That is a case of working on multiple projects at once.
> 
> No, it isn't.  I'm working on a single project, but need to look
> outside of its directory to find some information.  A very natural
> thing to do, and it doesn't mean I started working on another
> project.  More importantly, I do want that Grep buffer be available to
> me as part of the current project, because I'm likely to return there
> more than once.

And bookmarks/switch-to-buffer/registers are not good options for this goal?

>> And whatever your purpose of grepping it, wouldn't you agree that
>> that grep buffer is probably closer to the "other" project rather
>> that to the current one?
> 
> No.  And that "other" project doesn't even have to be a "project" in
> the project.el sense of the word.

Here's one direction how it could be resolved:

Make sure project-vc-external-roots-function points to a function that 
returns a list which includes the "other" directory. By default, it will 
include the directories where tags tables reside.

Then create a new command, project-or-external-switch-to-buffer. Which 
would check for default-directory belonging not only to the project 
root, but for belonging the external roots as well.

>> To reuse your argument, 'M-x switch-to-buffer' is still available for
>> borderline cases.
> 
> An argument that you dismissed previously.

I dismissed it as in "people wanted to call this command less". But my 
specific wording also made provision for it being necessary at least 
sometimes.

But you made this argument. So perhaps it may be good enough for your 
purposes. Am I wrong here?

I don't want to dismiss your request outright, but we'll need to have 
some design which would fit the existing model and/or wouldn't be hard 
to support. Because exceptions and edge cases tend to pile up, if we 
just start adding exceptions here and there.

> I think that non file-visiting buffers are rarely related to a
> project, an exception rather than a rule.  My suggestion is therefore
> to turn the table and come up with a list of such buffers that
> _always_ are related to a specific project.  And instead of using the
> default-directory as evidence for the buffer's relevance, we may need
> a command that explicitly makes a buffer related to a project.

Well, Phil gave a list of examples. And in all of those cases, I think, 
default-directory worked well as an indicator.

In my work, having Grep buffer's default-directory belong to the current 
project is also a string indicator.

>> So default-directory is not the worst indicator.
> 
> I'm saying it isn't the best, either.  We have just discovered at
> least 2 problems with it.  We should try to find a better one.

One might interpret your request essentially as "I want any buffer, 
arbitrarily, based on my current intention, be associated with the 
current project". It's probably not as arbitrary, so you're welcome to 
suggest an alternative indicator.

>> But we should also try the new command out and document our experiences.
> 
> IMNSHO, some thought is required even before we hope the experience
> will teach us in due time.  Doing this only by trial and error runs
> the risk of converging on a design that is found later to be
> restrictive, or one that cannot be easily extended to support
> behaviors not accounted for or envisioned originally.

I'm not suggesting trying it for a few years. The command is here 
already, so the process has started.

>> So far Theodor agreed too. And myself as well. You alone have voiced
>> disagreement. With this distribution of votes, it seems the default
>> behavior should default to including non-file-visiting buffers (with
>> some agreed-upon exceptions).
> 
> For an opinion to "count", it doesn't have to _replace_ the other
> opinions.  It could be taken into consideration by augmenting the
> design so that it supports both kinds of behaviors.  This is IME
> better than flatly discarding the dissenting opinions.

Sure.

>> And, of course, we can add a user option that would allow to tweak
>> the choosing logic.
> 
> Sub-optimal selection of the "belongs to a project" criteria will make
> any such user options cumbersome and hard to use.  IOW, user options
> shouldn't be considered as means to "fix" sub-optimal design.  We are
> at a point where we can make the design better, if we consider a wider
> variety of project-related activities.

Also agree.

Re: master 1e3b0f2: Improve doc strings of project.el

[Dmitry Gutov]

On 20.06.2020 11:55, Eli Zaretskii wrote:
> I proposed to find a different way of telling which buffer is related
> to a project, other than by looking at its default-directory.  We
> could design a way of doing that which would then support adding
> buffers to a project quite easily.  But Dmitry doesn't think we should
> go that way.

If you can outline and propose that alternative way, I'm all for 
considering it.

Compared to an ad-hoc variable (or having a command that adds such a 
buffer manually), I'd much prefer an automatic approach. It's just that 
I'm not seeing anything concrete yet.

Re: master 1e3b0f2: Improve doc strings of project.el

[Dmitry Gutov]

On 20.06.2020 13:07, Eli Zaretskii wrote:
> In any case, my proposal was not about the API itself, it was more
> about the implementation of the API.  For example, we could have an
> implementation of the generic project-files API that consulted some
> list instead of asking the VCS or the filesystem to provide the files.

You can of course do this already, in 10 lines or so.

But imagine you did add such a backend where project-files returns an 
arbitrary list of files. And you use it.

How would project-switch-to-buffer consult it? Will it have to call 
project-files now, as opposed to just using project-root as the basis 
for comparison? And then compare buffer-file-name of all file-visiting 
buffers against that list?

That would work for smaller projects, but in large ones project-files is 
not instantaneous, so they will be penalized in performance by such an 
approach. And project-switch-to-buffer should work with all kinds of 
projects.

That also wouldn't solve the problem of non-file-visiting buffers like 
Grep from your example.

Re: master 1e3b0f2: Improve doc strings of project.el

[Dmitry Gutov]

On 20.06.2020 13:19, Theodor Thornhill wrote:
> Yeah, this is kind of what I was thinking with the
> 'project-additional-buffers' idea.  This could be implementation
> agnostic as well, and be just a flat list connected to some root,
> persisted in the project list.

Or it could be a new generic function on projects (probably not).

Or it could be a hook, where any user can add their desired 
buffer-finding logic. And that hook would be consulted by 
project-switch-to-buffer and project-kill-buffers (maybe).

> As such, this will always be consulted
> when invoking "project-find-files", "project-switch-to-buffer" etc.

Probably not project-find-files.

Re: master 1e3b0f2: Improve doc strings of project.el

[Dmitry Gutov]

On 20.06.2020 10:20, Eli Zaretskii wrote:

>>> Not as part of working on the current project.  Though I might decide
>>> to add some of them to the current project, as the need arises.
>>
>> I mean, do you visit them in the same Emacs session?
> 
> I just answered that above.  This is all in a single Emacs session.

What I meant is, is there harm in modeling that kind of project as VC 
project with just lots of exceptions (additional ignores). Or is that 
just difficult.

>> OTOH, we already have a customization point that allows to exclude more
>> files than .gitignore does (the project-vc-ignores variable).
> 
> I don't think exclusion alone is enough.  We need also a way of
> _including_ files in a project.

It's on my list. As long as we're talking about whitelisting files in 
the same directory tree.

But also see project-external-roots.

>> *And* one can use the project API to introduce a project backend that
>> does not rely on VC repositories.
> 
> I think we should have commands to do so in the core.  It's too basic
> a capability for any IDE for us to leave it to add-ons.

Commands? Project backends are applied automatically in the current model.

Re: master 1e3b0f2: Improve doc strings of project.el

[Kévin Le Gouguec]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 20.06.2020 11:55, Eli Zaretskii wrote:
>> I proposed to find a different way of telling which buffer is related
>> to a project, other than by looking at its default-directory.  We
>> could design a way of doing that which would then support adding
>> buffers to a project quite easily.  But Dmitry doesn't think we should
>> go that way.
>
> If you can outline and propose that alternative way, I'm all for
> considering it.

A few messages back, Eli said:

>                                                        If we really
> want to record in these buffers what project they are related to, we
> need to have stronger evidence, like what was the current-buffer when
> the command was invoked, or maybe something else (like name the
> buffers in some special way).

Where "these buffers" refers, IIUC, to "stuff like *XREF*, Eshell,
Occur, *Compilation*, *Grep*, etc.".

Would it make sense for these special modes to have a buffer-local
variable pointing to the buffer where the command was invoked?
project.el could then consult that variable first, then fallback to
default-directory?

Re: master 1e3b0f2: Improve doc strings of project.el

[Dmitry Gutov]

On 20.06.2020 14:46, Kévin Le Gouguec wrote:
> A few messages back, Eli said:
> 
>>                                                         If we really
>> want to record in these buffers what project they are related to, we
>> need to have stronger evidence, like what was the current-buffer when
>> the command was invoked, or maybe something else (like name the
>> buffers in some special way).
> Where "these buffers" refers, IIUC, to "stuff like*XREF*, Eshell,
> Occur,*Compilation*,*Grep*, etc.".
> 
> Would it make sense for these special modes to have a buffer-local
> variable pointing to the buffer where the command was invoked?
> project.el could then consult that variable first, then fallback to
> default-directory?

Perhaps. I don't know if that would be enough for Eli's purposes, however.

After all, in the Grep example, it could have been invoked from one of 
the buffers belonging to the current project, or just as well from an 
"outside" buffer (because, for example, that made it easier to select 
the intended directory).

Re: master 1e3b0f2: Improve doc strings of project.el

[Eli Zaretskii]

> Cc: theo@thornhill.no, philip@warpmail.net, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sat, 20 Jun 2020 14:25:50 +0300
> 
> On 20.06.2020 09:43, Eli Zaretskii wrote:
> 
> > So all you want to add to the current doc string is "...and doesn't
> > belong to some other project"?
> 
> I don't know about that. Interpreting that literally would mean that 
> such buffers would never be suggested because they "belong to some other 
> project" (according to the reasoning which gave rise to this wording) no 
> matter which of the projects is the current one: "the outer", or "the 
> inner".
> 
> So what we should say is just "belongs to the current project".

Not sure I understand.  The current doc string says

    "Switch to another buffer that is related to the current project.
  A buffer is related to a project if its `default-directory'
  is inside the directory hierarchy of the project's root."

You want to change the last sentence to say

  A buffer is related to a project if it belongs to the project.

?

> The implication being that if it's inside a nested project, it
> doesn't belong to the outer one.

How will this implication become apparent to the users, if the doc
string doesn't discuss the significance of being "nested"?

> > No, it isn't.  I'm working on a single project, but need to look
> > outside of its directory to find some information.  A very natural
> > thing to do, and it doesn't mean I started working on another
> > project.  More importantly, I do want that Grep buffer be available to
> > me as part of the current project, because I'm likely to return there
> > more than once.
> 
> And bookmarks/switch-to-buffer/registers are not good options for this goal?

The problem I raised is that such a Grep buffer will not be offered as
a completion candidate when it is definitely relevant to my work on a
project.  You suggest that I don't use the project.el commands, which
is exactly what bothers me: it means that project.el commands don't
support well a very simple project-related activity, which for me is
a very frequent one.

> Make sure project-vc-external-roots-function points to a function that 
> returns a list which includes the "other" directory. By default, it will 
> include the directories where tags tables reside.

But I don't want all the files in that directory to be part of the
project.  I want just the Grep buffer.

And anyway, coming up with a function that does some job is not a
user-level operation, at least not when the user needs to do it every
now and then, when the work on a project takes him/her outside of a
directory.  We need easier facilities, like commands and simple-valued
variables.

> >> To reuse your argument, 'M-x switch-to-buffer' is still available for
> >> borderline cases.
> > 
> > An argument that you dismissed previously.
> 
> I dismissed it as in "people wanted to call this command less". But my 
> specific wording also made provision for it being necessary at least 
> sometimes.
> 
> But you made this argument. So perhaps it may be good enough for your 
> purposes. Am I wrong here?

How can it be good enough for my purposes, but not good enough for
those of others?  What kind of logic is this?

> I don't want to dismiss your request outright, but we'll need to have 
> some design which would fit the existing model and/or wouldn't be hard 
> to support.

I presented a critique of the current design, whereby the buffer's
default-directory is used to decide whether the buffer belongs to a
project.  I presented several examples where this design doesn't do
what I as a user expect.  If you don't want to dismiss that critique
outright, I'd expect you to come up with some changes that would
support the use cases I described.  That is the expected response to a
critique such as the one I presented.  I don't understand why you
expect _me_ to have some design, I'm not working on developing
project.el, I'm just a (potential) user of it.

> Because exceptions and edge cases tend to pile up, if we just start
> adding exceptions here and there.

The use cases I presented aren't exceptions, they are something
developers do all the time.

> > I think that non file-visiting buffers are rarely related to a
> > project, an exception rather than a rule.  My suggestion is therefore
> > to turn the table and come up with a list of such buffers that
> > _always_ are related to a specific project.  And instead of using the
> > default-directory as evidence for the buffer's relevance, we may need
> > a command that explicitly makes a buffer related to a project.
> 
> Well, Phil gave a list of examples. And in all of those cases, I think, 
> default-directory worked well as an indicator.

The same examples could be supported by having a list of buffers that
should be considered.  Whether such an explicit list is a good or a
bad idea depends on how long it will have to be.  That is why I
suggested to see how many such buffers are there, because I suspect
there are only a few.

By contrast, making a far-reaching decision that default-directory is
all we need, based on such a small sample, might produce sub-optimal
results, as I described.

> In my work, having Grep buffer's default-directory belong to the current 
> project is also a string indicator.

My example was with a Grep buffer whose default-directory is NOT in
the current project.  IOW, my problem was with the false negative.

> >> So default-directory is not the worst indicator.
> > 
> > I'm saying it isn't the best, either.  We have just discovered at
> > least 2 problems with it.  We should try to find a better one.
> 
> One might interpret your request essentially as "I want any buffer, 
> arbitrarily, based on my current intention, be associated with the 
> current project". It's probably not as arbitrary, so you're welcome to 
> suggest an alternative indicator.

I already suggested one: let the user specify that with special
commands, and have other commands do that automatically where we know
the user will want that with very high probability.

Doing this means we need some indicator other than default-directory.
A buffer-local variable, perhaps, or membership in some data
structure, like a list of buffers associated with a project.  Or
anything else, really, that can support better, more accurate
decisions.  There are many alternatives; my point is that basing this
on default-directory is not the best strategy.

Re: master 1e3b0f2: Improve doc strings of project.el

[Eli Zaretskii]

> Cc: philip@warpmail.net, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sat, 20 Jun 2020 14:29:20 +0300
> 
> On 20.06.2020 11:55, Eli Zaretskii wrote:
> > I proposed to find a different way of telling which buffer is related
> > to a project, other than by looking at its default-directory.  We
> > could design a way of doing that which would then support adding
> > buffers to a project quite easily.  But Dmitry doesn't think we should
> > go that way.
> 
> If you can outline and propose that alternative way, I'm all for 
> considering it.

I think I did.

And I don't really understand what is being requested here.  Recording
something in a variable, be it a buffer-local one or a global one
indexed by a project, is easy, and doesn't require any complicated
design.  I'm talking as a user here, asking you to support what I
think is a very common usage pattern of working on some programming
problem that involves changes in a group of related files.

> Compared to an ad-hoc variable (or having a command that adds such a 
> buffer manually), I'd much prefer an automatic approach.

Not sure what does "automatic" mean here.  If that means using only
currently available buffer-local variables, then who's to say that
this is good enough?  Why limit ourselves to only such solutions?  the
few presented examples already uncovered some deficiencies in relying
on one such variable.

OTOH, we could introduce a new buffer-local variable to be
automatically set by, say, a find-file-hook, or by project-find-file,
or via some other feature we have.  Same with recording relevant
buffers in some project-specific data structure.

IOW, it sounds like we have any number of ways for doing something
like that, and the only argument is whether we should.

Re: master 1e3b0f2: Improve doc strings of project.el

[Basil L. Contovounesios]

Eli Zaretskii <eliz@gnu.org> writes:

>> One might interpret your request essentially as "I want any buffer, 
>> arbitrarily, based on my current intention, be associated with the 
>> current project". It's probably not as arbitrary, so you're welcome to 
>> suggest an alternative indicator.
>
> I already suggested one: let the user specify that with special
> commands, and have other commands do that automatically where we know
> the user will want that with very high probability.
>
> Doing this means we need some indicator other than default-directory.
> A buffer-local variable, perhaps, or membership in some data
> structure, like a list of buffers associated with a project.  Or
> anything else, really, that can support better, more accurate
> decisions.  There are many alternatives; my point is that basing this
> on default-directory is not the best strategy.

What if grep, xref, etc. "singleton" buffers get created per project,
either automatically by detecting that they are called from within a
project, or by providing project.el wrappers for their respective
commands?  Then they could be switched to based on default-directory
alone, right?  Would that fly at all?

-- 
Basil

Re: master 1e3b0f2: Improve doc strings of project.el

[Eli Zaretskii]

> Cc: philip@warpmail.net, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sat, 20 Jun 2020 14:35:24 +0300
> 
> On 20.06.2020 13:07, Eli Zaretskii wrote:
> > In any case, my proposal was not about the API itself, it was more
> > about the implementation of the API.  For example, we could have an
> > implementation of the generic project-files API that consulted some
> > list instead of asking the VCS or the filesystem to provide the files.
> 
> You can of course do this already, in 10 lines or so.

I'm a user.  I don't want to write code every time I want something
simple from the project-related commands.  I expect such simple use
cases to be supported OOTB.

> But imagine you did add such a backend where project-files returns an 
> arbitrary list of files. And you use it.
> 
> How would project-switch-to-buffer consult it? Will it have to call 
> project-files now, as opposed to just using project-root as the basis 
> for comparison? And then compare buffer-file-name of all file-visiting 
> buffers against that list?

That's one possibility, yes.

> That would work for smaller projects, but in large ones project-files is 
> not instantaneous, so they will be penalized in performance by such an 
> approach.

We have faster data structures than lists.  We could use them.

And in any case, we should identify the needs first, and then
implement the features that support those needs.  Rejecting some of
the needs because they don't fit the current implementation is the
wrong way for designing features such as this, which is supposed to be
part of our IDE infrastructure.

> And project-switch-to-buffer should work with all kinds of projects.

Yes.  And one such kind is an ad-hoc collection of files and buffers,
where only the user knows which ones he/she is interested in and which
ones he/she isn't.  Every IDE I saw supports something like that, so
we should do that as well, IMO.

> That also wouldn't solve the problem of non-file-visiting buffers like 
> Grep from your example.

You talked about project-files, so buffers that don't visit files are
excluded, of course.  But an indication that supports marking buffers
as belonging to a project could support both file-visiting and non
file-visiting buffers.

Re: master 1e3b0f2: Improve doc strings of project.el

[Eli Zaretskii]

> Cc: theothornhill@pm.me, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sat, 20 Jun 2020 14:41:22 +0300
> 
> On 20.06.2020 10:20, Eli Zaretskii wrote:
> 
> >>> Not as part of working on the current project.  Though I might decide
> >>> to add some of them to the current project, as the need arises.
> >>
> >> I mean, do you visit them in the same Emacs session?
> > 
> > I just answered that above.  This is all in a single Emacs session.
> 
> What I meant is, is there harm in modeling that kind of project as VC 
> project with just lots of exceptions (additional ignores). Or is that 
> just difficult.

I don't understand the question.  What is a "VC project", and how is
it relevant to the use case I described with grepping another
directory to see how others solved some problem?

> >> OTOH, we already have a customization point that allows to exclude more
> >> files than .gitignore does (the project-vc-ignores variable).
> > 
> > I don't think exclusion alone is enough.  We need also a way of
> > _including_ files in a project.
> 
> It's on my list. As long as we're talking about whitelisting files in 
> the same directory tree.

That's part of the issue, but it isn't all of it.

> But also see project-external-roots.

That supports only entire directory trees, so is not selective enough,
IMO.

> >> *And* one can use the project API to introduce a project backend that
> >> does not rely on VC repositories.
> > 
> > I think we should have commands to do so in the core.  It's too basic
> > a capability for any IDE for us to leave it to add-ons.
> 
> Commands? Project backends are applied automatically in the current model.

Don't understand what you are saying here, either.  Are you saying a
backend cannot support some notion of starting a project, or of adding
a file to a project?

Re: master 1e3b0f2: Improve doc strings of project.el

[Eli Zaretskii]

> Cc: Eli Zaretskii <eliz@gnu.org>, Theodor Thornhill <theothornhill@pm.me>,
>  philip@warpmail.net, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sat, 20 Jun 2020 14:57:07 +0300
> 
> > Would it make sense for these special modes to have a buffer-local
> > variable pointing to the buffer where the command was invoked?
> > project.el could then consult that variable first, then fallback to
> > default-directory?
> 
> Perhaps. I don't know if that would be enough for Eli's purposes, however.

It will solve some of them, to be sure.

> After all, in the Grep example, it could have been invoked from one of 
> the buffers belonging to the current project, or just as well from an 
> "outside" buffer (because, for example, that made it easier to select 
> the intended directory).

So you are saying that it's better to have no way of supporting that
use case than have some, even restricted, way?

Re: master 1e3b0f2: Improve doc strings of project.el

[Eli Zaretskii]

> From: "Basil L. Contovounesios" <contovob@tcd.ie>
> Cc: Dmitry Gutov <dgutov@yandex.ru>,  philip@warpmail.net,
>   theo@thornhill.no,  emacs-devel@gnu.org
> Date: Sat, 20 Jun 2020 13:30:57 +0100
> 
> > Doing this means we need some indicator other than default-directory.
> > A buffer-local variable, perhaps, or membership in some data
> > structure, like a list of buffers associated with a project.  Or
> > anything else, really, that can support better, more accurate
> > decisions.  There are many alternatives; my point is that basing this
> > on default-directory is not the best strategy.
> 
> What if grep, xref, etc. "singleton" buffers get created per project,
> either automatically by detecting that they are called from within a
> project, or by providing project.el wrappers for their respective
> commands?

That is something I suggested a few messages back, actually.

> Then they could be switched to based on default-directory alone,
> right?  Would that fly at all?

Why would we need the default-directory in this case?  If these
buffers are created to be project-specific, it would mean that they
are somehow different from one another, right?  We could then use
these differences directly to tell which ones are related to a
project.  For example, the names of these buffers could include the
project ID, or some buffer-local variable could do that.

Once again, default-directory is too random to be a strong evidence in
this case.  It will cause too many false positives and false
negatives.

Re: master 1e3b0f2: Improve doc strings of project.el

[Dmitry Gutov]

On 20.06.2020 15:37, Eli Zaretskii wrote:
>> Cc: Eli Zaretskii <eliz@gnu.org>, Theodor Thornhill <theothornhill@pm.me>,
>>   philip@warpmail.net, emacs-devel@gnu.org
>> From: Dmitry Gutov <dgutov@yandex.ru>
>> Date: Sat, 20 Jun 2020 14:57:07 +0300
>>
>>> Would it make sense for these special modes to have a buffer-local
>>> variable pointing to the buffer where the command was invoked?
>>> project.el could then consult that variable first, then fallback to
>>> default-directory?
>>
>> Perhaps. I don't know if that would be enough for Eli's purposes, however.
> 
> It will solve some of them, to be sure.

Solve them how? Do you always invoke that search command from a buffer 
belonging to the original project? 75% of the time? 30%?

>> After all, in the Grep example, it could have been invoked from one of
>> the buffers belonging to the current project, or just as well from an
>> "outside" buffer (because, for example, that made it easier to select
>> the intended directory).
> 
> So you are saying that it's better to have no way of supporting that
> use case than have some, even restricted, way?

No. I even suggested a few ways how this can be user-extensible.

And I'm saying we need a coherent design that's in line with what is 
already there. Or extends the behavior in a predictable way.

Going back to Kevin's suggestion, I'm fairly sure it will also generate 
both false positives and false negatives, going simply by my own usage 
habits: the originating buffer of where I'm calling a command doesn't 
necessarily correspond to which project I'm thinking of right now.

So I'm not sure if we can do this for the default behavior. Hence we'd 
need to use either one of the existing customization point, or add a new 
one.

Re: master 1e3b0f2: Improve doc strings of project.el

[Dmitry Gutov]

On 20.06.2020 15:36, Eli Zaretskii wrote:

>> What I meant is, is there harm in modeling that kind of project as VC
>> project with just lots of exceptions (additional ignores). Or is that
>> just difficult.
> 
> I don't understand the question.  What is a "VC project", and how is
> it relevant to the use case I described with grepping another
> directory to see how others solved some problem?

A VC project is a project recognized by the VC project backend.

It's tweakable by customizing project-vc-ignores (or, more preferably, 
setting that variable in dir-locals.el) and the 
project-vc-external-roots-function variable.

>>>> OTOH, we already have a customization point that allows to exclude more
>>>> files than .gitignore does (the project-vc-ignores variable).
>>>
>>> I don't think exclusion alone is enough.  We need also a way of
>>> _including_ files in a project.
>>
>> It's on my list. As long as we're talking about whitelisting files in
>> the same directory tree.
> 
> That's part of the issue, but it isn't all of it.
> 
>> But also see project-external-roots.
> 
> That supports only entire directory trees, so is not selective enough,
> IMO.

Perhaps. But does that affect your usage?

I'm still not sure about the entirety of your usage scenario.

You have this project, which is current. Do you switch to other projects 
in the same session? How do they generally look? Is is generally the 
same idea of having an arbitrary set of files and buffers?

>>>> *And* one can use the project API to introduce a project backend that
>>>> does not rely on VC repositories.
>>>
>>> I think we should have commands to do so in the core.  It's too basic
>>> a capability for any IDE for us to leave it to add-ons.
>>
>> Commands? Project backends are applied automatically in the current model.
> 
> Don't understand what you are saying here, either.  Are you saying a
> backend cannot support some notion of starting a project, or of adding
> a file to a project?

If we do decide on the separate backend approach, this can be done. But 
the commands would need to clearly specify that they work (and probably 
enable) a particular kind of project backend. So this is workable, but a 
bit clunky, so I'd like to see a bit more general support for this 
approach before adding it to project.el. But that shouldn't stop us from 
working on it.

BTW, you're saying it's a "basic IDE capability", and that other editors 
have that. Do you have a particular editor in mind as an example? So 
we're speaking the same language.

Re: master 1e3b0f2: Improve doc strings of project.el

[Dmitry Gutov]

On 20.06.2020 15:32, Eli Zaretskii wrote:

>> On 20.06.2020 13:07, Eli Zaretskii wrote:
>>> In any case, my proposal was not about the API itself, it was more
>>> about the implementation of the API.  For example, we could have an
>>> implementation of the generic project-files API that consulted some
>>> list instead of asking the VCS or the filesystem to provide the files.
>>
>> You can of course do this already, in 10 lines or so.
> 
> I'm a user.  I don't want to write code every time I want something
> simple from the project-related commands.  I expect such simple use
> cases to be supported OOTB.

I think we're generally comfortable recommending users customize 
something in their init script, if that's a one-off request, and nobody 
else comes with the same request.

Of course, when such customization turns out to be useful and elegant 
enough, we can always incorporate it later.

For less technical users, we sometimes offer a snippet to use.

Even if you just do this for yourself, it can be a valuable of the API 
is defined, whether it can support your usage scenario this way.

>> But imagine you did add such a backend where project-files returns an
>> arbitrary list of files. And you use it.
>>
>> How would project-switch-to-buffer consult it? Will it have to call
>> project-files now, as opposed to just using project-root as the basis
>> for comparison? And then compare buffer-file-name of all file-visiting
>> buffers against that list?
> 
> That's one possibility, yes.

I'm asking for a concrete suggestion, if you have it.

And I have just outlined a problem with the most obvious approach.

>> That would work for smaller projects, but in large ones project-files is
>> not instantaneous, so they will be penalized in performance by such an
>> approach.
> 
> We have faster data structures than lists.  We could use them.

List as a data structure is a secondary problem. Big projects can take a 
while to fetch that list.

> And in any case, we should identify the needs first, and then
> implement the features that support those needs.  Rejecting some of
> the needs because they don't fit the current implementation is the
> wrong way for designing features such as this, which is supposed to be
> part of our IDE infrastructure.

As outlined above, even if I recommend trying to implement a usage 
pattern in one's init script first, that doesn't necessarily mean I 
rejected it.

>> And project-switch-to-buffer should work with all kinds of projects.
> 
> Yes.  And one such kind is an ad-hoc collection of files and buffers,
> where only the user knows which ones he/she is interested in and which
> ones he/she isn't.  Every IDE I saw supports something like that, so
> we should do that as well, IMO.

I'm curious about those "every IDE". I don't recall such a feature in 
ones I tried. Perhaps I just didn't use it, of course.

>> That also wouldn't solve the problem of non-file-visiting buffers like
>> Grep from your example.
> 
> You talked about project-files, so buffers that don't visit files are
> excluded, of course.  But an indication that supports marking buffers
> as belonging to a project could support both file-visiting and non
> file-visiting buffers.

Buffers are essentially transient. An indication could be a list of 
files, but the project API has no provision for "list of buffers", and 
such a thing has never come up until now.

So my first instinct would be to make this customizable through some 
hook, rather than extend the API.

RE: master 1e3b0f2: Improve doc strings of project.el

[Drew Adams]

(Caveat: I haven't been following this thread, I'll
likely continue to not follow it, and I know almost
nothing about project.el.)

From one msg:

> > And bookmarks/switch-to-buffer/registers are not good
> > options for this goal?
> 
> The problem I raised is that such a Grep buffer will not be offered as
> a completion candidate when it is definitely relevant to my work on a
> project.  You suggest that I don't use the project.el commands, which
> is exactly what bothers me: it means that project.el commands don't
> support well a very simple project-related activity, which for me is
> a very frequent one.

From another msg:

> I proposed to find a different way of telling which buffer is related
> to a project, other than by looking at its default-directory.  We
> could design a way of doing that which would then support adding
> buffers to a project quite easily.  But Dmitry doesn't think we should
> go that way.
> 
> More to the point, my question is how would adding a buffer to the
> project be recorded and where, under your proposal?

I imagine that project.el defines a project as including
something like these things, and perhaps some other stuff:

1. A set of files.
2. Possibly a set of other Emacs/Lisp objects: buffers,
   vars, whatever.
3. Associations among such things, and among them and
   other things, including things (e.g. tools) outside
   Emacs.

I imagine that project.el can persist a project.

I'll just mention that bookmarks, bookmark files, and
bookmark-list displays are objects that can be persisted.

A bookmark can record nearly anything: a set of variables,
a location or set of locations,... anything.

Bookmark+ takes advantage of this generality in ways that
could be useful for project mgt.  I mention this because
vanilla Emacs bookmarks could too.  I'm not proposing
using Bookmark+, but just mentioning some of what you can
do using bookmarks that I think can be relevant to
working with or defining projects.

You can bookmark a Dired buffer, with its markings,
omit-set, switches, inserted subdirs, hidden subdirs, etc.
And Dired can list an arbitrary set of files and dirs; it
need not show only part of a single tree.

You can bookmark a set of Dired bookmarks that represent
a dir hierarchy and are opened together.

You can bookmark buffers, and locations in them.  Of
course, if restored from persistence the bookmark must
know how to regenerate the buffer.

One thing I haven't gotten around to implementing, but
is possible, is to bookmark the marked buffers in either
Ibuffer or Buffer Menu.  (Again, supposing knowledge of
how to regenerate or recuperate the buffer in a new
session.)

You can bookmark a function.  For example, a bookmark to
restore a given `*grep*' listing, by recomputing it,
records a function that performs the given `grep'
invocation.

From a `grep', `occur', compilation, etc. buffer you can
bookmark one or more of the individual hits, so that
jumping to such a bookmark takes you to the hit destination
(e.g. position in the grepped file).

You can bookmark all hits visible in the `grep', `occur',
etc. buffer, so after editing to remove some hits you can
get the effect of having "marked" some and then bookmarked
each of the marked ones.

In Dired, you can bookmark each of the marked files or
dirs.

In the bookmark-list display, you can dired the marked
bookmarks, i.e., create a Dired buffer with just the
files and dirs that are the targets of the marked
bookmarks.

You can bookmark a bookmark-list display itself (what
`C-x r l' shows).  This includes filtering, sorting, etc.

You can bookmark a bookmark file, so that jumping to
the bookmark switches to that set of bookmarks or loads
them additionally.

There are bookmark jump commands specific to different
types or sets of bookmarks.  E.g. jump to the bookmarks
for a specific project, choosing only among "project"
bookmarks.

There are many ways to define a project in this regard.
One is to mark the "project" bookmarks in a bookmark
list and use `C-c C-j' to define a command that jumps
to them (they are the only completion candidates).

You can bookmark a desktop.

Besides bookmark-file bookmarks, bookmark-list bookmarks,
Dired bookmarks, desktop bookmarks, etc., as ways of
organizing bookmarks into sets (~projects), you have
tags.

You can tag any set of bookmarks any way you like,
including with any Lisp object as tag value instead
of just a string.  You can list or complete against
bookmarks that have certain combinations of tags.

So your wanting to get to a `*grep*' buffer that
belongs to a given project is realized trivially:
just tag it for the project, then use a command that
jumps to a bookmark with that tag (i.e., only those
bookmarks are completion candidates).

Tags are the most flexible way to organize bookmarks.

You can even use bookmarks to tag files or other
objects in various ways just for purposes of
organizing them, whether or not you ever use the
bookmarks as a way to visit them.

Different projects can have different tags for the
same sets of files, directories,..., whatever.  No
need for copies - just different tagging.  Bookmarks
are persistent pointers.

If a "project" is mostly about grouping things to be
able to act on them together (search, visit, apply a
tool, etc.), and if a "project" should be able to be
persisted, then consider that you might do well to
leverage bookmarks - a flexible, open-ended way to
persist and restore or otherwise act on nearly
anything.

If "project" means code-development project for you,
or even if it means something else, what it does is
likely to be more or less a subset of what you can
do with bookmarks.  You might want to build features
for code-development project using, or on top of,
bookmarks.

And if you're thinking of adding very general
capabilities to project.el, you might want to think
instead about adding them at a more general level,
and then adapting them for the more specific use
for projects.

Just a thought.
____

For more info, search here for "project", "tag", or
any other terms used above:

https://www.emacswiki.org/emacs/BookmarkPlus

Re: master 1e3b0f2: Improve doc strings of project.el

[Dmitry Gutov]

On 20.06.2020 15:12, Eli Zaretskii wrote:

>> I don't know about that. Interpreting that literally would mean that
>> such buffers would never be suggested because they "belong to some other
>> project" (according to the reasoning which gave rise to this wording) no
>> matter which of the projects is the current one: "the outer", or "the
>> inner".
>>
>> So what we should say is just "belongs to the current project".
> 
> Not sure I understand.  The current doc string says
> 
>      "Switch to another buffer that is related to the current project.
>    A buffer is related to a project if its `default-directory'
>    is inside the directory hierarchy of the project's root."
> 
> You want to change the last sentence to say
> 
>    A buffer is related to a project if it belongs to the project.
> 
> ?

It could say "A buffer is related to project PR if 'project-current', 
when called in it, returns a value equal to PR". But that could be 
considered tautological.

So alternatively, the first line would just say

   Switch to another buffer that belongs to the current project.

But only if the implementation is made more precise in this respect.

>> The implication being that if it's inside a nested project, it
>> doesn't belong to the outer one.
> 
> How will this implication become apparent to the users, if the doc
> string doesn't discuss the significance of being "nested"?

The doc string of project-try-vc can describe that. Or the general 
description of VC projects, in the Commentary header and/or the manual.

>>> No, it isn't.  I'm working on a single project, but need to look
>>> outside of its directory to find some information.  A very natural
>>> thing to do, and it doesn't mean I started working on another
>>> project.  More importantly, I do want that Grep buffer be available to
>>> me as part of the current project, because I'm likely to return there
>>> more than once.
>>
>> And bookmarks/switch-to-buffer/registers are not good options for this goal?
> 
> The problem I raised is that such a Grep buffer will not be offered as
> a completion candidate when it is definitely relevant to my work on a
> project.  You suggest that I don't use the project.el commands, which
> is exactly what bothers me: it means that project.el commands don't
> support well a very simple project-related activity, which for me is
> a very frequent one.

It is a possibility.

But so far I don't understand very well whether your usage habits and 
the view of how a project should be defined are common enough, or fairly 
singular.

In any case, we should weigh the added complexity of implementing it 
(using any of the proposed ways) against the added utility.

For instance, you also said that you generally work on a single project 
at once (in an Emacs session). I could have misunderstood you there, but 
if that's true, the added benefit of using project-switch-to-buffer, as 
opposed to switch-to-buffer, would be marginal.

The reason the former function was added, as I see it, is for people who 
switch between different projects in the same session to be able to 
"narrow down" their view of the buffer list, to more easily find the one 
they are currently looking for.

>> Make sure project-vc-external-roots-function points to a function that
>> returns a list which includes the "other" directory. By default, it will
>> include the directories where tags tables reside.
> 
> But I don't want all the files in that directory to be part of the
> project.  I want just the Grep buffer.

Fair enough.

> And anyway, coming up with a function that does some job is not a
> user-level operation, at least not when the user needs to do it every
> now and then, when the work on a project takes him/her outside of a
> directory.  We need easier facilities, like commands and simple-valued
> variables.

Alternatively, it could be a customization variable you could set in 
some .dir-locals.el. Or a full-blown configuration file, specific to a 
new project backend, that would list which files belong to which 
projects. Not sure if such configurations would reside in some "project 
roots" (the meaning of the term becomes a bit blurry when project files 
are allowed to reside outside of its directory tree), or inside .emacs.d.

>>>> To reuse your argument, 'M-x switch-to-buffer' is still available for
>>>> borderline cases.
>>>
>>> An argument that you dismissed previously.
>>
>> I dismissed it as in "people wanted to call this command less". But my
>> specific wording also made provision for it being necessary at least
>> sometimes.
>>
>> But you made this argument. So perhaps it may be good enough for your
>> purposes. Am I wrong here?
> 
> How can it be good enough for my purposes, but not good enough for
> those of others?  What kind of logic is this?

Different usage patterns. I'm sure you're familiar with that notion.

>> I don't want to dismiss your request outright, but we'll need to have
>> some design which would fit the existing model and/or wouldn't be hard
>> to support.
> 
> I presented a critique of the current design, whereby the buffer's
> default-directory is used to decide whether the buffer belongs to a
> project.  I presented several examples where this design doesn't do
> what I as a user expect.  If you don't want to dismiss that critique
> outright, I'd expect you to come up with some changes that would
> support the use cases I described.  That is the expected response to a
> critique such as the one I presented.  I don't understand why you
> expect _me_ to have some design, I'm not working on developing
> project.el, I'm just a (potential) user of it.

I don't want to dismiss the critique outright. Believe it or not, I 
really do want you, personally, to be happier and more productive as a 
result of things I do here. I spent several hours today just thinking 
about this discussion. But our workflows are different, and expectations 
are different as well, and it seems that your requests tend to conflict 
with some design choices I have originally made.

One of them was to make the VC project a hands-off backend, one that 
immediately "just works" (or tries to), with a few possibilities for 
customization through local variables.

Consider also that your critique might be based on a different view of 
what a "project" is.

In my understanding, it is, roughly speaking, a directory with files. 
Some of them are "junk", build artefacts and the like. But the rest 
usually support a program, or a document, etc, and together they result 
in one whole. Some stuff working together. A project could have 
dependencies, they often reside outside of that directory (but not 
always), but they aren't as much a part of the said project.

Consequently, if there is a regexp search across these files, I 
immediately consider that search (the process, and the search results) a 
part of that project. Because it searches in that project's files. No 
matter the end goal of that search.

You seem to think (and this is only my guess, of course) that a project 
is a unit of work. And that whatever files, or activities, are pertinent 
to your current goal, are a part of that project. Hence, if you do a 
search anywhere, in any directory, but in pursuit of that goal, the 
search results are certainly a part of the current project. It is 
certainly a valid viewpoint, but one that I have never considered before.

I _think_ (though I can't be sure) that the former viewpoint is more 
prevalent among other editors, and among our users as well. And with 
that definition, your example doesn't show anomalous behavior, and the 
outcome is something that a user would expect. Even if someone might say 
that they would be able to reach that Grep buffer quickly by this or 
other means.

So I'm not sure where to go from here. If the latter viewpoint has more 
supporters, perhaps an new, alternative backend is the way to go. This 
would be a test of the API, how well it adapts to different goals.

But it we were able to adopt the same language, perhaps that would be 
more productive after all. And we might reach the additional goals by 
other means.

>>> I think that non file-visiting buffers are rarely related to a
>>> project, an exception rather than a rule.  My suggestion is therefore
>>> to turn the table and come up with a list of such buffers that
>>> _always_ are related to a specific project.  And instead of using the
>>> default-directory as evidence for the buffer's relevance, we may need
>>> a command that explicitly makes a buffer related to a project.
>>
>> Well, Phil gave a list of examples. And in all of those cases, I think,
>> default-directory worked well as an indicator.
> 
> The same examples could be supported by having a list of buffers that
> should be considered.  Whether such an explicit list is a good or a
> bad idea depends on how long it will have to be.  That is why I
> suggested to see how many such buffers are there, because I suspect
> there are only a few.

IMO (and I know at least some people who hold a similar view), 
constructing a list of buffers/files/etc that constitute the current 
project by hand, is not user-friendly.

So I'd rather the default behavior didn't require extra hand-holding by 
the user.

> By contrast, making a far-reaching decision that default-directory is
> all we need, based on such a small sample, might produce sub-optimal
> results, as I described.

So far it's not a done decision, and a lot can be improved as a result 
of this and nearby discussions.

A far-reaching decision would be to make an API change, or to add a new 
backend.

Re: master 1e3b0f2: Improve doc strings of project.el

[Dmitry Gutov]

On 20.06.2020 15:25, Eli Zaretskii wrote:

>> Compared to an ad-hoc variable (or having a command that adds such a
>> buffer manually), I'd much prefer an automatic approach.
> 
> Not sure what does "automatic" mean here.  If that means using only
> currently available buffer-local variables, then who's to say that
> this is good enough?

And/or adding new custom variables and hooks.

> Why limit ourselves to only such solutions?  the
> few presented examples already uncovered some deficiencies in relying
> on one such variable.

As explained in the other email, it's a matter of interpretation.

> OTOH, we could introduce a new buffer-local variable to be
> automatically set by, say, a find-file-hook, or by project-find-file,
> or via some other feature we have.  Same with recording relevant
> buffers in some project-specific data structure.

find-file-hook seems a bit late to use if we wanted to save which 
project the current command was called from.

project-find-file would only affect file-visiting buffers, and only ones 
that we already consider a part of the current project. So what new info 
would it save?

"Some other feature" sounds nice.

> IOW, it sounds like we have any number of ways for doing something
> like that, and the only argument is whether we should.

I'm still not certain of what is the optimal solution.

Re: master 1e3b0f2: Improve doc strings of project.el

[Eli Zaretskii]

> Cc: theo@thornhill.no, philip@warpmail.net, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sun, 21 Jun 2020 02:11:18 +0300
> 
> I don't want to dismiss the critique outright. Believe it or not, I really do want you, personally, to be happier and more productive as a result of things I do here. I spent several hours today just thinking about this discussion. But our workflows are different, and expectations are different as well, and it seems that your requests tend to conflict with some design choices I have originally made.
> 
> 
> One of them was to make the VC project a hands-off backend, one that immediately "just works" (or tries to), with a few possibilities for customization through local variables.

This must be supported, of course.  And the backend which treats an
entire directory with its subdirectories as a project must also be
supported.

I'm not arguing for dropping any of these two.  I'm arguing for adding
yet another possibility, whereby the files belonging to a project are
selected by the user, whether by marking in a Dired-like directory
display, or by explicitly naming each file to be added, or by
drag-n-drop.

Based on the discussion of non file-visiting buffers related to a
project, I think there should also be a command that would allow the
user to include/exclude such buffers from the project, because it
doesn't make sense to me to decide up front that any *shell* or *grep*
or whatever buffers are automatically considered to be part of the
project based on something as ephemeral as their default-directory.

> You seem to think (and this is only my guess, of course) that a project is a unit of work. And that whatever files, or activities, are pertinent to your current goal, are a part of that project. Hence, if you do a search anywhere, in any directory, but in pursuit of that goal, the search results are certainly a part of the current project. It is certainly a valid viewpoint, but one that I have never considered before.

I think it needs to be considered because it's a valid use case and
happens in practice.  It would be useful to support it OOTB.  Even if
all the files belonging to a project are in the same directory, the
MO where _all_ (or a vast majority) of the files in a directory belong
to the project is a serious limitation, and we shouldn't impose it on
our users.  Granted, one can produce a large enough exclude/ignore
list to leave only a handful of files, but if just 5% or 10% of files
in a directory should be part of a project, excluding the other 90% or
95% is a nuisance and an unnatural thing to do.

> So I'm not sure where to go from here. If the latter viewpoint has more supporters, perhaps an new, alternative backend is the way to go. This would be a test of the API, how well it adapts to different goals.

I'm not talking in terms of backends, I'm talking in terms of
user-facing features.  I think we should decide whether a feature such
as the above should be part of what project.el supports, and then
consider how to implement it.  I don't see why the implementation
should be very complicated, FWIW, so there's no need to bring the
implementation into the picture, not yet.

>         And project-switch-to-buffer should work with all kinds of projects.
> 
>     Yes.  And one such kind is an ad-hoc collection of files and buffers,
>     where only the user knows which ones he/she is interested in and which
>     ones he/she isn't.  Every IDE I saw supports something like that, so
>     we should do that as well, IMO.
> 
> I'm curious about those "every IDE". I don't recall such a feature in ones I tried. Perhaps I just didn't use it, of course. 

Few examples below:

  Code::Blocks:
  https://www.cs.odu.edu/~zeil/FAQs/Public/newIDEProject/index.html#creating-a-project-from-existing-code

  Visual Studio:
  https://docs.microsoft.com/en-us/visualstudio/ide/creating-solutions-and-projects?view=vs-2019
  Look under "Create a project from existing code files", "Add files
  to a solution", "Create empty solutions"

  QNX:
  https://www.qnx.com/developers/docs/6.4.1/ide_en/user_guide/tutorials.html

  Netbeans:
  https://netbeans.apache.org/kb/docs/cnd/quickstart.html#_adding_existing_files_to_your_project

  TI's Code Composer:
  http://software-dl.ti.com/ccs/esd/documents/users_guide/ccs_project-management.html#adding-or-linking-source-files-to-project

Re: master 1e3b0f2: Improve doc strings of project.el

[Juri Linkov]

>>     And one such kind is an ad-hoc collection of files and buffers,
>>     where only the user knows which ones he/she is interested in and which
>>     ones he/she isn't.  Every IDE I saw supports something like that, so
>>     we should do that as well, IMO.
>> 
>> I'm curious about those "every IDE".  I don't recall such a feature
>> in ones I tried.  Perhaps I just didn't use it, of course.
>
> Few examples below:
>
>   Code::Blocks:
>   https://www.cs.odu.edu/~zeil/FAQs/Public/newIDEProject/index.html#creating-a-project-from-existing-code
>
>   Visual Studio:
>   https://docs.microsoft.com/en-us/visualstudio/ide/creating-solutions-and-projects?view=vs-2019
>   Look under "Create a project from existing code files", "Add files
>   to a solution", "Create empty solutions"
>
>   QNX:
>   https://www.qnx.com/developers/docs/6.4.1/ide_en/user_guide/tutorials.html
>
>   Netbeans:
>   https://netbeans.apache.org/kb/docs/cnd/quickstart.html#_adding_existing_files_to_your_project
>
>   TI's Code Composer:
>   http://software-dl.ti.com/ccs/esd/documents/users_guide/ccs_project-management.html#adding-or-linking-source-files-to-project

Such point-and-click wizards are more of a hassle than a help.
If someone doesn't want to specify files to *exclude* in .gitignore,
then in Emacs a better thing to do would be to support a special file
.project with the same syntax as .gitignore but specifying what files
to *include* in the project.  Should such file also support specifying
buffer names, I'm not sure.

Re: master 1e3b0f2: Improve doc strings of project.el

[Eli Zaretskii]

> From: Juri Linkov <juri@linkov.net>
> Cc: Dmitry Gutov <dgutov@yandex.ru>,  philip@warpmail.net,
>   theo@thornhill.no,  emacs-devel@gnu.org
> Date: Mon, 22 Jun 2020 01:24:15 +0300
> 
> >   Code::Blocks:
> >   https://www.cs.odu.edu/~zeil/FAQs/Public/newIDEProject/index.html#creating-a-project-from-existing-code
> >
> >   Visual Studio:
> >   https://docs.microsoft.com/en-us/visualstudio/ide/creating-solutions-and-projects?view=vs-2019
> >   Look under "Create a project from existing code files", "Add files
> >   to a solution", "Create empty solutions"
> >
> >   QNX:
> >   https://www.qnx.com/developers/docs/6.4.1/ide_en/user_guide/tutorials.html
> >
> >   Netbeans:
> >   https://netbeans.apache.org/kb/docs/cnd/quickstart.html#_adding_existing_files_to_your_project
> >
> >   TI's Code Composer:
> >   http://software-dl.ti.com/ccs/esd/documents/users_guide/ccs_project-management.html#adding-or-linking-source-files-to-project
> 
> Such point-and-click wizards are more of a hassle than a help.

I didn't post that to ask for the wizard functionality, I posted that
at Dmitry's request to show IDE facilities that allow to add files to
a (possibly empty) project.

Re: master 1e3b0f2: Improve doc strings of project.el

[Dmitry Gutov]

Hi Eli,

Thanks for these examples. Commends below.

On 21.06.2020 18:08, Eli Zaretskii wrote:
>>     And project-switch-to-buffer should work with all kinds of projects.
>>
>>      Yes.  And one such kind is an ad-hoc collection of files and buffers,
>>      where only the user knows which ones he/she is interested in and which
>>      ones he/she isn't.  Every IDE I saw supports something like that, so
>>      we should do that as well, IMO.
>>
>> I'm curious about those "every IDE". I don't recall such a feature in ones I tried. Perhaps I just didn't use it, of course.
> Few examples below:
> 
>    Code::Blocks:
>    https://www.cs.odu.edu/~zeil/FAQs/Public/newIDEProject/index.html#creating-a-project-from-existing-code

Going by "Navigate to the directory where you already have your code.", 
it seems to expect that all project code resides in the directory the 
user specifies. Although, for some reason or other, it decides to 
default to not listing existing source code in there in the "sources" 
section. Perhaps that has to do with build configuration.

Note A: Being able to tell "source files" from the rest of the project 
files is a concern that we haven't looked into yet (nor into its 
applications). In the default backend, though, I would like to have it 
work automatically as well (probably by having a configurable list of 
globs/file extensions/etc).

Note B: If you only meant this as an illustration of including certain 
files in the directory, this setup should be supportable when we add 
support for whitelisting entries (as opposed to just blacklisting ones, 
through in project-ignores). That would still require all files to be 
contained inside the directory that is designated as "project root 
directory".

>    Visual Studio:
>    https://docs.microsoft.com/en-us/visualstudio/ide/creating-solutions-and-projects?view=vs-2019
>    Look under "Create a project from existing code files", "Add files
>    to a solution", "Create empty solutions"

"Create a project from existing code files" actually makes VS pick up 
all the source files in the chosen directory, from what I can tell. In 
that, it's closer to our project-vc backend than to Code::Blocks.

"Add files to a solution" talks about having a file that "applies to 
multiple projects". Which talks about a case of spreading the current 
work contents across multiple root directories. Which is what its 
"solutions" are.

As far as how to support a notion similar to solutions best, I'm not 
sure. Perhaps it indeed would be a separate thing (package/feature/etc), 
with a file-based configuration, that points at projects included in it. 
That way, project-vc backend (and others) could be reused for included 
projects. And we could have solution-level commands (e.g. 
solution-find-file, which scans across all included projects). I don't 
understand the level of demand for this among our users, and as such, 
the necessary features.

We're not at the level of complexity that Visual Studio has (WRT build 
configurations support, etc), and most other text editors I know don't 
have this feature, so perhaps it's a bit premature. The good news here, 
however, is that implementing solutions as a separate feature on top of 
project.el should be relatively simple. And people are welcome to 
experiment.

>    QNX:
>    https://www.qnx.com/developers/docs/6.4.1/ide_en/user_guide/tutorials.html

It seems to describe a situation similar to MS VS, except "solutions" 
are called "workspaces" here.

Otherwise, you don't select individual files inside a project to be added.

>    Netbeans:
>    https://netbeans.apache.org/kb/docs/cnd/quickstart.html#_adding_existing_files_to_your_project

"Creating a Project With Existing Sources" seems to import all source 
files within a project directory automatically.

I wonder what "Adding Existing Files to Your Project" does. Perhaps it 
copies files from elsewhere into the project directory, if it's not 
there yet.

>    TI's Code Composer:
>    http://software-dl.ti.com/ccs/esd/documents/users_guide/ccs_project-management.html#adding-or-linking-source-files-to-project

The "physically copied" case is obvious.

Regarding "When you link a file/folder to a project ...", does it create 
a symlink, maybe? That wouldn't require a configuration change.

To sum up, what I saw here is mostly what I'm already used to anyway: a 
project is basically a directory with some files in it (the set is 
generally based on the principle of exclusion, but some subviews can be 
based on inclusion/whitelist as well), and not an arbitrary set of files 
from random places on disk.

Not to discourage alternative workflows, but this is the concept we 
should work on supporting well first.

I should also note that these other editors have no concept of 
"buffers", and thus no way to configure their inclusion of exclusion. 
Thus, any entity that might correspond to our non-file-visiting buffers 
(such as a search window, or a compilation output window) is likely 
implicitly considered to just be part of the current project or 
solution. Please feel free to correct me here.

(To be continued, to address the rest of the email.)

Re: master 1e3b0f2: Improve doc strings of project.el

[Eli Zaretskii]

> Cc: theo@thornhill.no, philip@warpmail.net, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sun, 28 Jun 2020 03:56:29 +0300
> 
> To sum up, what I saw here is mostly what I'm already used to anyway: a 
> project is basically a directory with some files in it (the set is 
> generally based on the principle of exclusion, but some subviews can be 
> based on inclusion/whitelist as well), and not an arbitrary set of files 
> from random places on disk.
> 
> Not to discourage alternative workflows, but this is the concept we 
> should work on supporting well first.

What do you mean "first"? why cannot we support both the workflow you
are familiar with and the other kind?  It isn't like adding such
support is exceptionally hard, it just needs to find an alternative to
making decisions based on the default-directory alone.

And how is your summary above not a "discouragement"?

> I should also note that these other editors have no concept of 
> "buffers", and thus no way to configure their inclusion of exclusion. 
> Thus, any entity that might correspond to our non-file-visiting buffers 
> (such as a search window, or a compilation output window) is likely 
> implicitly considered to just be part of the current project or 
> solution. Please feel free to correct me here.

The problem I raised is twofold:

 . non file-visiting buffers could have a default-directory outside of
   the project's root, and still be relevant to the project (I
   provided a couple of examples)

 . file-visiting buffers can live in the same directory, but belong to
   different projects

Thus, using default-directory as the sole or the main indicator of
belonging to a project limits us in the use cases we can easily
support.

The examples I provided were in response to your request to show IDE
capabilities to add existing files to an empty project, thereby
creating a project from existing files and not from a directory.

But I have said all this several times already, and at this point it
sounds like whatever I say, whichever examples I show, you will still
discard them and maintain that using default-directory is a good
method, so it doesn't look like this discussion is going anywhere...

Re: master 1e3b0f2: Improve doc strings of project.el

[Dmitry Gutov]

(Continuing from the previous email.)

On 21.06.2020 18:08, Eli Zaretskii wrote:
>> I don't want to dismiss the critique outright. Believe it or not, I really do want you, personally, to be happier and more productive as a result of things I do here. I spent several hours today just thinking about this discussion. But our workflows are different, and expectations are different as well, and it seems that your requests tend to conflict with some design choices I have originally made.
>>
>>
>> One of them was to make the VC project a hands-off backend, one that immediately "just works" (or tries to), with a few possibilities for customization through local variables.
> 
> This must be supported, of course.  And the backend which treats an
> entire directory with its subdirectories as a project must also be
> supported.

At this point I'm somewhat inclined to merge these two into one.

> I'm not arguing for dropping any of these two.  I'm arguing for adding
> yet another possibility, whereby the files belonging to a project are
> selected by the user, whether by marking in a Dired-like directory
> display, or by explicitly naming each file to be added, or by
> drag-n-drop.

Acknowledged.

It's a pretty alien UI paradigm for me (micromanaging project 
configuration like that). Similarly, I never got into using tags tables 
because that requires to both generate a file, re-generate it manually, 
and visit it manually as well. I have seen similar sentiments expressed 
by other users. (Having said that, I hope to return to adding a feature 
that auto-generates and regenerates tags tables for the current project 
using etags, one of these days.)

As such, I'd rather let someone else implement the project backend with 
the features you outlined. And if someone decides to take it up, I have 
a number of thoughts on how it can be better integrated.

OTOH, along the lines of Juri's opinion, perhaps your requirements could 
be satisfied by making use of whitelisting entries in VC project config. 
But if your projects indeed span multiple directories, the config 
couldn't reside in some particular root, then that calls for a separate 
backend.

> Based on the discussion of non file-visiting buffers related to a
> project, I think there should also be a command that would allow the
> user to include/exclude such buffers from the project, because it
> doesn't make sense to me to decide up front that any *shell* or *grep*
> or whatever buffers are automatically considered to be part of the
> project based on something as ephemeral as their default-directory.

As I explained, with project-vc you don't "decide" anything like that up 
front, you just internalize the idea that any buffer with 
default-directory inside the project's root more or less "belongs" to 
that "project" (the meanings of both terms are apparently different from 
how you use that words). So, if you need to switch to said buffer, you 
make sure to use the appropriate command (either project-switch-buffer, 
or switch-buffer, or project-switch-project).

>> You seem to think (and this is only my guess, of course) that a project is a unit of work. And that whatever files, or activities, are pertinent to your current goal, are a part of that project. Hence, if you do a search anywhere, in any directory, but in pursuit of that goal, the search results are certainly a part of the current project. It is certainly a valid viewpoint, but one that I have never considered before.
> 
> I think it needs to be considered because it's a valid use case and
> happens in practice.  It would be useful to support it OOTB.

I'd like to make this clear: it's not a "use case", it's a point of 
view. If one considers a project to roughly be a "directory with files 
in it", and their work spans several such directories, they will think 
that the work spans several projects, and that's it.

> Even if
> all the files belonging to a project are in the same directory, the
> MO where _all_ (or a vast majority) of the files in a directory belong
> to the project is a serious limitation, and we shouldn't impose it on
> our users.  Granted, one can produce a large enough exclude/ignore
> list to leave only a handful of files, but if just 5% or 10% of files
> in a directory should be part of a project, excluding the other 90% or
> 95% is a nuisance and an unnatural thing to do.

This sounds like a use case for whitelist entries.

>> So I'm not sure where to go from here. If the latter viewpoint has more supporters, perhaps an new, alternative backend is the way to go. This would be a test of the API, how well it adapts to different goals.
> 
> I'm not talking in terms of backends, I'm talking in terms of
> user-facing features.  I think we should decide whether a feature such
> as the above should be part of what project.el supports, and then
> consider how to implement it.  I don't see why the implementation
> should be very complicated, FWIW, so there's no need to bring the
> implementation into the picture, not yet.

A naive implementation should be pretty easy. What is difficult is 
fitting this starkly different kind of interaction and configuration in 
the current design.

Having interactive commands affect project-vc's configuration is a murky 
idea, having two different ways to configure it. And it would have to 
add some new variables as well, that affect its behavior. Or some other 
configuration storage.

So ultimately, if you really want this kind of interaction, it would be 
better to have a separate backend. It could also have a different author 
than myself, thereby validating the idea that it is, indeed, something 
that users want.

Re: master 1e3b0f2: Improve doc strings of project.el

[Dmitry Gutov]

On 28.06.2020 17:28, Eli Zaretskii wrote:

>> To sum up, what I saw here is mostly what I'm already used to anyway: a
>> project is basically a directory with some files in it (the set is
>> generally based on the principle of exclusion, but some subviews can be
>> based on inclusion/whitelist as well), and not an arbitrary set of files
>> from random places on disk.
>>
>> Not to discourage alternative workflows, but this is the concept we
>> should work on supporting well first.
> 
> What do you mean "first"?

Meaning, I have to prioritize the tasks in my personal queue. And there 
are other features I should get to as well. I hope you'll like them 
either way.

> why cannot we support both the workflow you
> are familiar with and the other kind?  It isn't like adding such
> support is exceptionally hard, it just needs to find an alternative to
> making decisions based on the default-directory alone.
> 
> And how is your summary above not a "discouragement"?

I think it's an objective summary. Because you said that not having this 
capability keeps Emacs at a severe disadvantage to other editors. So it 
was clearly a subject that required analysis.

On the other hand, I'm open to having another backend that works in a 
different way, and willing to provide guidance to anyone who wanted to 
write it. And to try to keep the API sufficiently un-opinionated that 
both kinds of backends can be used through it successfully, with 
workarounds kept to a minimum.

>> I should also note that these other editors have no concept of
>> "buffers", and thus no way to configure their inclusion of exclusion.
>> Thus, any entity that might correspond to our non-file-visiting buffers
>> (such as a search window, or a compilation output window) is likely
>> implicitly considered to just be part of the current project or
>> solution. Please feel free to correct me here.
> 
> The problem I raised is twofold:
> 
>   . non file-visiting buffers could have a default-directory outside of
>     the project's root, and still be relevant to the project (I
>     provided a couple of examples)

Buffers are unique to Emacs, so it's hard to take example from other 
editors. I have to ask, though.

The model that other editors use, and the one I'm assuming you do as 
well (guessing mostly due to how tags' UI works), is that you work only 
on one "project" (in your sense of the word) at a time.

Then, would I be correct to assume that if there exists a Grep buffer in 
the current session, then it mostly likely belongs to the current 
"project"? If so, would there be any particular advantage to using 
project-switch-to-buffer instead of plain switch-to-buffer?

>   . file-visiting buffers can live in the same directory, but belong to
>     different projects
> 
> Thus, using default-directory as the sole or the main indicator of
> belonging to a project limits us in the use cases we can easily
> support.
> 
> The examples I provided were in response to your request to show IDE
> capabilities to add existing files to an empty project, thereby
> creating a project from existing files and not from a directory.

To the best of my understanding, in most of those examples, the files 
were either copied into said directory as a result of that action, or, 
if they had already been inside, their status had changed from "other 
files" to "sources files". The only example that I understood things to 
be different with any certainty, was Visual Studio's "solutions".

> But I have said all this several times already, and at this point it
> sounds like whatever I say, whichever examples I show, you will still
> discard them and maintain that using default-directory is a good
> method, so it doesn't look like this discussion is going anywhere...

I never said that it's an ideal method. And I'm sure there can be 
problematic examples out there. But I wasn't convinced by the example 
you gave because the reason for why it was bad was based on your 
understanding on the word "project", and not on mine. Thus, it's hard 
for me to choose a good solution for the project backend which is based 
on my understanding of the word "project".

Re: master 1e3b0f2: Improve doc strings of project.el

[Eli Zaretskii]

> Cc: theo@thornhill.no, philip@warpmail.net, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Mon, 29 Jun 2020 01:14:36 +0300
> 
> As such, I'd rather let someone else implement the project backend with 
> the features you outlined. And if someone decides to take it up, I have 
> a number of thoughts on how it can be better integrated.

Implementing this, and the priority assigned to doing so, is not the
main issue that bothered me, and caused me to start this thread.  This
is a tangent; see below.

> OTOH, along the lines of Juri's opinion, perhaps your requirements could 
> be satisfied by making use of whitelisting entries in VC project config. 

Whitelisting is a nuisance when you need to whitelist too many files;
blacklisting is a nuisance when you need to blacklist too many files.
Other than that, both are fine.  But again, that's not the issue here.

> > I'm not talking in terms of backends, I'm talking in terms of
> > user-facing features.  I think we should decide whether a feature such
> > as the above should be part of what project.el supports, and then
> > consider how to implement it.  I don't see why the implementation
> > should be very complicated, FWIW, so there's no need to bring the
> > implementation into the picture, not yet.
> 
> A naive implementation should be pretty easy. What is difficult is 
> fitting this starkly different kind of interaction and configuration in 
> the current design.

If this use case doesn't fit the current design, IMO we should rethink
the current design.  We are just beginning to implement the user-level
features of project.el, so we should make sure adding other reasonable
use cases will be as easy as we can envision today.  _That_ is the
problem that bothers me the most: that we already have restrictions on
which use cases can and cannot be supported, although we barely
started.

> So ultimately, if you really want this kind of interaction, it would be 
> better to have a separate backend. It could also have a different author 
> than myself, thereby validating the idea that it is, indeed, something 
> that users want.

We are constantly losing focus in this discussion, at least from my
POV.  The main reason I started this is that I think using
default-directory for the decision which files/buffers belong to a
project is not the best idea.  Everything else I said, every example I
brought up, was to show why we should rethink that design decision,
and do so soon, not in some uncertain future.  This is the only issue
that really bothers me.  I didn't ask you to implement the "project
from existing files" feature, let alone change your priorities, I
didn't request any new commands or features, I just wanted to show how
this single design decision limits our future options.  This issue,
which is the only one important for me in this discussion, gets lost
time and again, because you take each my example and each my argument,
and start arguing about each one of them separately, thus drowning the
main issue in an ocean of related, but secondary aspects of the
problem.

Please, try re-reading or reconsidering all I said in that light, and
let's please return to talk about that one issue.  Assuming, that is,
that you are open to discussing that issue, and could be convinced to
change your mind; otherwise, let's agree to disagree and finish this.

TIA

Re: master 1e3b0f2: Improve doc strings of project.el

[Eli Zaretskii]

> Cc: theo@thornhill.no, philip@warpmail.net, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Mon, 29 Jun 2020 01:35:19 +0300
> 
> >> Not to discourage alternative workflows, but this is the concept we
> >> should work on supporting well first.
> > 
> > What do you mean "first"?
> 
> Meaning, I have to prioritize the tasks in my personal queue. And there 
> are other features I should get to as well. I hope you'll like them 
> either way.

I didn't mean to ask you to change your priorities.  I was only
talking about the design of deciding what buffers belong to which
project, and the implications of that design.

> I think it's an objective summary. Because you said that not having this 
> capability keeps Emacs at a severe disadvantage to other editors. So it 
> was clearly a subject that required analysis.

Once again, my problem is not with the schedule of providing the
support for the use case I described, but with the current design
which AFAICT makes it hard to add such support in the future.  We
should modify the design to make such use cases easier to add.

> The model that other editors use, and the one I'm assuming you do as 
> well (guessing mostly due to how tags' UI works), is that you work only 
> on one "project" (in your sense of the word) at a time.
> 
> Then, would I be correct to assume that if there exists a Grep buffer in 
> the current session, then it mostly likely belongs to the current 
> "project"? If so, would there be any particular advantage to using 
> project-switch-to-buffer instead of plain switch-to-buffer?

No and no.  My Emacs sessions run for many days on end, and during
that time I work on several projects.  Sometimes I need to switch
between them every few minutes (e.g., when I read my email and need to
answer questions and request, or review code, related to several
projects, in the order in which I read the incoming email).  More
often, I would work for several hours on one project, then switch to
another, then to yet another or back to the first.

When I do switch, I don't want to lose the "payload" of the project I
switch from: its files, its Grep, XREF, and Compilation buffers, its
documentation buffers, etc. -- because I know I will come back there
in hours or days.  This means each project should stay readily
accessible, so that I could pick up where I left off.

It is true that the last Grep buffer I created most probably belongs
to the current project, but that doesn't mean I want to give up the
previous Grep buffer -- I might need it shortly.

> I never said that it's an ideal method. And I'm sure there can be 
> problematic examples out there. But I wasn't convinced by the example 
> you gave because the reason for why it was bad was based on your 
> understanding on the word "project", and not on mine. Thus, it's hard 
> for me to choose a good solution for the project backend which is based 
> on my understanding of the word "project".

I'm not asking you to come up with a backend suitable for the use case
I described.  I'm asking you to take that use case into consideration
when you design the means of deciding which buffer belongs to what
project.  If we design this decision in a way that cannot be easily
extended to reasonable use cases we can envision, we will limit
ourselves in the use cases we can usefully support, ever.  Let's try
to avoid producing such a restrictive design, if we can.

I can understand why some design might not be conducive to supporting
use cases we never thought could be relevant.  But this use case we
already envision, and unless you really think it's completely not
relevant, the design should IMO be able to support it easily enough.

Re: master 1e3b0f2: Improve doc strings of project.el

[Stephen Leake]

Eli Zaretskii <eliz@gnu.org> writes:

> No and no.  My Emacs sessions run for many days on end, and during
> that time I work on several projects.  Sometimes I need to switch
> between them every few minutes (e.g., when I read my email and need to
> answer questions and request, or review code, related to several
> projects, in the order in which I read the incoming email).  More
> often, I would work for several hours on one project, then switch to
> another, then to yet another or back to the first.

I have a similar work pattern. My solution to remembering project info
is implemented in the 'wisi' GNU ELPA package; there is one active
project; (project-current) returns that project, unless a buffer has
overridden project-find-functions locally, which I do for elisp mode.
There is a menu of known projects the user can choose from. In
addition, running 'compile' in a Makefile sets the project to the one
relevant to that makefile.

I have not tried to integrate this with the new project menu in
project.el yet.

-- 
-- Stephe

Re: master 1e3b0f2: Improve doc strings of project.el

[Juri Linkov]

> I have a similar work pattern. My solution to remembering project info
> is implemented in the 'wisi' GNU ELPA package; there is one active
> project; (project-current) returns that project, unless a buffer has
> overridden project-find-functions locally, which I do for elisp mode.
> There is a menu of known projects the user can choose from. In
> addition, running 'compile' in a Makefile sets the project to the one
> relevant to that makefile.

This description reminds how next-error works in a similar way.
By default, it uses the global value of the last navigation -
in project.el it could use default-directory by default.
But like a buffer can override next-error navigation,
project.el should allow a buffer to set a buffer-local value
to define its project like your project-find-functions.

Re: master 1e3b0f2: Improve doc strings of project.el

[Dmitry Gutov]

On 30.06.2020 01:35, Juri Linkov wrote:
>> I have a similar work pattern. My solution to remembering project info
>> is implemented in the 'wisi' GNU ELPA package; there is one active
>> project; (project-current) returns that project, unless a buffer has
>> overridden project-find-functions locally, which I do for elisp mode.
>> There is a menu of known projects the user can choose from. In
>> addition, running 'compile' in a Makefile sets the project to the one
>> relevant to that makefile.
> 
> This description reminds how next-error works in a similar way.
> By default, it uses the global value of the last navigation -
> in project.el it could use default-directory by default.
> But like a buffer can override next-error navigation,

Like a user customization can, you mean?

I would be very surprised to see a buffer-local value of 
next-error-found-function.

> project.el should allow a buffer to set a buffer-local value
> to define its project like your project-find-functions.

That would mean to have a buffer-local value of project-find-functions. 
Which is possible, but I would struggle to consider it to be a good 
idea. Unless the value is technically buffer-local, but in essence just 
applied by a globalized major mode.

It might still be better to create a straight global major mode in that 
case, though.

Re: master 1e3b0f2: Improve doc strings of project.el

[Dmitry Gutov]

On 29.06.2020 18:07, Stephen Leake wrote:
> I have a similar work pattern. My solution to remembering project info
> is implemented in the 'wisi' GNU ELPA package; there is one active
> project; (project-current) returns that project, unless a buffer has
> overridden project-find-functions locally, which I do for elisp mode.

That sounds weird to me, but I hope it's working out okay.

> There is a menu of known projects the user can choose from. In
> addition, running 'compile' in a Makefile sets the project to the one
> relevant to that makefile.

Global project switching. Why not.

> I have not tried to integrate this with the new project menu in
> project.el yet.

Let us know how it works out for you.

I fear it might be incompatible with having session-global projects 
(ones where it's impossible to guess based simply on directory), but I 
like to be proven wrong.

Re: master 1e3b0f2: Improve doc strings of project.el

[Dmitry Gutov]

Hi Eli,

On 29.06.2020 17:50, Eli Zaretskii wrote:

> I didn't mean to ask you to change your priorities.  I was only
> talking about the design of deciding what buffers belong to which
> project, and the implications of that design.

If you recall, I even posted a "more correct" patch a while back that 
wouldn't make the choice of which buffers belong to the project depend 
solely on the value of default-directory.

>> I think it's an objective summary. Because you said that not having this
>> capability keeps Emacs at a severe disadvantage to other editors. So it
>> was clearly a subject that required analysis.
> 
> Once again, my problem is not with the schedule of providing the
> support for the use case I described, but with the current design
> which AFAICT makes it hard to add such support in the future.  We
> should modify the design to make such use cases easier to add.

...and I asked for design suggestions that would make the behavior 
better from your standpoint.

Some ideas, old and new:

1. We can call 'project-current' in every buffer, and then compare the 
returned values (this is what that alternative patch did). But recalling 
bug#41029, it seems some users can have outstanding numbers of buffers 
open, and this approach might heavily limit the performance of 
project-switch-to-buffer, unless we employ some very heavy 
buffer->project caching. And I'd really like to stay away from manual 
cache invalidation.

2. We could call 'project-files' on the current project and compare the 
values of buffer-file-name of every buffer. This could become slow with 
larger projects. And the complexity will be N^2 (at least with naive 
implementation), so it can be worse than project-find-file for those 
projects. Also, this doesn't solve your problem with Grep buffers. But 
it would help in the situation of having a project contain arbitrary 
files from arbitrary directories.

3. Create a new generic function (or several) which would delegate the 
inclusion logic to individual project backends. This would require work 
on naming, semantics, what have you, and would likely still come out 
clunkier than either of the previous two options. Further, with this 
approach, I'm still not sure of a good "fast" solution for project-vc 
which leads to correct behavior in the presence of nested projects.

>> The model that other editors use, and the one I'm assuming you do as
>> well (guessing mostly due to how tags' UI works), is that you work only
>> on one "project" (in your sense of the word) at a time.
>>
>> Then, would I be correct to assume that if there exists a Grep buffer in
>> the current session, then it mostly likely belongs to the current
>> "project"? If so, would there be any particular advantage to using
>> project-switch-to-buffer instead of plain switch-to-buffer?
> 
> No and no.  My Emacs sessions run for many days on end, and during
> that time I work on several projects.  Sometimes I need to switch
> between them every few minutes (e.g., when I read my email and need to
> answer questions and request, or review code, related to several
> projects, in the order in which I read the incoming email).  More
> often, I would work for several hours on one project, then switch to
> another, then to yet another or back to the first.

I see, thank you.

> When I do switch, I don't want to lose the "payload" of the project I
> switch from: its files, its Grep, XREF, and Compilation buffers, its
> documentation buffers, etc. -- because I know I will come back there
> in hours or days.  This means each project should stay readily
> accessible, so that I could pick up where I left off.
> 
> It is true that the last Grep buffer I created most probably belongs
> to the current project, but that doesn't mean I want to give up the
> previous Grep buffer -- I might need it shortly.

What I meant, would there be a lot of downside to using switch-to-buffer 
specifically to switch to file-less buffers such as Grep when a need arises.

This way, they wouldn't be "lost", but you'd have to deal with filtering 
through many buffers. Even so, Grep buffers are usually named uniquely, 
so it's unlikely to be a total disaster. And you wouldn't be able to 
clean then up easily with project-kill-buffers.

Anyway, I'm talking about the backup plan here. One to consider if we 
don't manage to settle on a better approach.

>> I never said that it's an ideal method. And I'm sure there can be
>> problematic examples out there. But I wasn't convinced by the example
>> you gave because the reason for why it was bad was based on your
>> understanding on the word "project", and not on mine. Thus, it's hard
>> for me to choose a good solution for the project backend which is based
>> on my understanding of the word "project".
> 
> I'm not asking you to come up with a backend suitable for the use case
> I described.  I'm asking you to take that use case into consideration
> when you design the means of deciding which buffer belongs to what
> project.  If we design this decision in a way that cannot be easily
> extended to reasonable use cases we can envision, we will limit
> ourselves in the use cases we can usefully support, ever.  Let's try
> to avoid producing such a restrictive design, if we can.

Sure, agreed.

Re: master 1e3b0f2: Improve doc strings of project.el

[Eli Zaretskii]

> Cc: theo@thornhill.no, philip@warpmail.net, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Thu, 2 Jul 2020 01:57:37 +0300
> 
> > Once again, my problem is not with the schedule of providing the
> > support for the use case I described, but with the current design
> > which AFAICT makes it hard to add such support in the future.  We
> > should modify the design to make such use cases easier to add.
> 
> ...and I asked for design suggestions that would make the behavior 
> better from your standpoint.

I proposed one: have a special per-buffer variable which will give the
project (or a list of projects, if we want this to be more general) to
which the buffer is related.  I don't think we discussed that
possibility in detail, did we?

> 1. We can call 'project-current' in every buffer, and then compare the 
> returned values (this is what that alternative patch did). But recalling 
> bug#41029, it seems some users can have outstanding numbers of buffers 
> open, and this approach might heavily limit the performance of 
> project-switch-to-buffer, unless we employ some very heavy 
> buffer->project caching. And I'd really like to stay away from manual 
> cache invalidation.
> 
> 2. We could call 'project-files' on the current project and compare the 
> values of buffer-file-name of every buffer. This could become slow with 
> larger projects. And the complexity will be N^2 (at least with naive 
> implementation), so it can be worse than project-find-file for those 
> projects. Also, this doesn't solve your problem with Grep buffers. But 
> it would help in the situation of having a project contain arbitrary 
> files from arbitrary directories.
> 
> 3. Create a new generic function (or several) which would delegate the 
> inclusion logic to individual project backends. This would require work 
> on naming, semantics, what have you, and would likely still come out 
> clunkier than either of the previous two options.

#3 sounds like the best alternative to me, if a simple buffer-local
variable doesn't do the job for some reason.

> Further, with this approach, I'm still not sure of a good "fast"
> solution for project-vc which leads to correct behavior in the
> presence of nested projects.

project-vc could then store the list of files in the project to serve
the request, or a directory if all the files in the directory belong
to the project.

> > When I do switch, I don't want to lose the "payload" of the project I
> > switch from: its files, its Grep, XREF, and Compilation buffers, its
> > documentation buffers, etc. -- because I know I will come back there
> > in hours or days.  This means each project should stay readily
> > accessible, so that I could pick up where I left off.
> > 
> > It is true that the last Grep buffer I created most probably belongs
> > to the current project, but that doesn't mean I want to give up the
> > previous Grep buffer -- I might need it shortly.
> 
> What I meant, would there be a lot of downside to using switch-to-buffer 
> specifically to switch to file-less buffers such as Grep when a need arises.

This would mean we give up on supporting this use case by project
commands, wouldn't it?  Then I'd ask why this case is unsupported,
while the one described by Andrii is?

> Anyway, I'm talking about the backup plan here. One to consider if we 
> don't manage to settle on a better approach.

Understood.

Re: master 1e3b0f2: Improve doc strings of project.el

[Dmitry Gutov]

On 04.07.2020 10:15, Eli Zaretskii wrote:

>> ...and I asked for design suggestions that would make the behavior
>> better from your standpoint.
> 
> I proposed one: have a special per-buffer variable which will give the
> project (or a list of projects, if we want this to be more general) to
> which the buffer is related.  I don't think we discussed that
> possibility in detail, did we?

You mean like caching the return value of 'project-current'?

The problems I can see with that are:

- Since project is not a minor mode, we don't call it in major mode 
hooks. Which, by itself, is probably a good thing (causing no 
unnecessary work/slowdown). But that means that in the middle of any 
session we could have 100s of buffers that don't have any cached project 
value. Fetching the current project of all those at once could result in 
a long pause (some benchmarking on various machines/project 
configurations/sessioning habits would be welcome).

- A project configuration can change mid-session (due to a customization 
of some user option, or due to an edit in some .dir-locals.el). Or 
simply due to 'git init'. As a result, a given file's current project 
can change. It will be good to be able to support that without 
necessitating Emacs' restart or requiring manual cache invalidation.

>> 1. We can call 'project-current' in every buffer, and then compare the
>> returned values (this is what that alternative patch did). But recalling
>> bug#41029, it seems some users can have outstanding numbers of buffers
>> open, and this approach might heavily limit the performance of
>> project-switch-to-buffer, unless we employ some very heavy
>> buffer->project caching. And I'd really like to stay away from manual
>> cache invalidation.
>>
>> 2. We could call 'project-files' on the current project and compare the
>> values of buffer-file-name of every buffer. This could become slow with
>> larger projects. And the complexity will be N^2 (at least with naive
>> implementation), so it can be worse than project-find-file for those
>> projects. Also, this doesn't solve your problem with Grep buffers. But
>> it would help in the situation of having a project contain arbitrary
>> files from arbitrary directories.
>>
>> 3. Create a new generic function (or several) which would delegate the
>> inclusion logic to individual project backends. This would require work
>> on naming, semantics, what have you, and would likely still come out
>> clunkier than either of the previous two options.
> 
> #3 sounds like the best alternative to me, if a simple buffer-local
> variable doesn't do the job for some reason.

>> Further, with this approach, I'm still not sure of a good "fast"
>> solution for project-vc which leads to correct behavior in the
>> presence of nested projects.
> 
> project-vc could then store the list of files in the project to serve
> the request,

Store like cache the return value of 'project-files'? Still the same 
issue with cache invalidation, described above.

> or a directory if all the files in the directory belong
> to the project.

The problem is in the case of *nested* projects. That's the situation 
when all of the files belong to a directory _except_ those that belong 
to projects inside the current one. Like the monorepo example Theodor 
described.

>>> When I do switch, I don't want to lose the "payload" of the project I
>>> switch from: its files, its Grep, XREF, and Compilation buffers, its
>>> documentation buffers, etc. -- because I know I will come back there
>>> in hours or days.  This means each project should stay readily
>>> accessible, so that I could pick up where I left off.
>>>
>>> It is true that the last Grep buffer I created most probably belongs
>>> to the current project, but that doesn't mean I want to give up the
>>> previous Grep buffer -- I might need it shortly.
>>
>> What I meant, would there be a lot of downside to using switch-to-buffer
>> specifically to switch to file-less buffers such as Grep when a need arises.
> 
> This would mean we give up on supporting this use case by project
> commands, wouldn't it?  Then I'd ask why this case is unsupported,
> while the one described by Andrii is?

Andrii's (and others') workflow implies more frequent switching between 
"projects", rather than working on a single one for a long time.

I would imagine that with such approach, using commands scoped to the 
current project's buffer would be more necessary.

Having said all that, I've had a couple more ideas for transparent 
caching, such as caching by default-directory in the project-vc backend, 
and clearing the cache in post-command-hook.

So we can go ahead with idea #1 (which implies no hard change to the 
API) and see how far we can get push this idea without manual cache 
invalidation, while keeping decent performance for most of our users.

N.B.: project-vc currently caches the "current project" info more 
persistently than would be ideal; a part of this experiment will be 
scaling back on that caching, and wait for complaints about performance.

Re: master 1e3b0f2: Improve doc strings of project.el

[Eli Zaretskii]

> Cc: theo@thornhill.no, philip@warpmail.net, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sun, 5 Jul 2020 01:22:50 +0300
> 
> > I proposed one: have a special per-buffer variable which will give the
> > project (or a list of projects, if we want this to be more general) to
> > which the buffer is related.  I don't think we discussed that
> > possibility in detail, did we?
> 
> You mean like caching the return value of 'project-current'?

Something like that, yes.

> The problems I can see with that are:
> 
> - Since project is not a minor mode, we don't call it in major mode 
> hooks. Which, by itself, is probably a good thing (causing no 
> unnecessary work/slowdown). But that means that in the middle of any 
> session we could have 100s of buffers that don't have any cached project 
> value.

We have project-find-file and other similar project-* commands that
could cache that as side effect.

> - A project configuration can change mid-session (due to a customization 
> of some user option, or due to an edit in some .dir-locals.el). Or 
> simply due to 'git init'. As a result, a given file's current project 
> can change. It will be good to be able to support that without 
> necessitating Emacs' restart or requiring manual cache invalidation.

I don't see how you can avoid some user command which would mean a
project configuration change.  Alternatively, this could be solved
lazily, when the cached value is found to be outdated.

We don't need to solve every possible complication up front before we
decide that a method is workable, we just need to have a vague idea
how it could be solved when the time comes.

> > project-vc could then store the list of files in the project to serve
> > the request,
> 
> Store like cache the return value of 'project-files'? Still the same 
> issue with cache invalidation, described above.
> 
> > or a directory if all the files in the directory belong
> > to the project.
> 
> The problem is in the case of *nested* projects.

The first part above should take care of that.  The second part was to
suggest a shortcut: let a directory stand for all the files in it.

> >> What I meant, would there be a lot of downside to using switch-to-buffer
> >> specifically to switch to file-less buffers such as Grep when a need arises.
> > 
> > This would mean we give up on supporting this use case by project
> > commands, wouldn't it?  Then I'd ask why this case is unsupported,
> > while the one described by Andrii is?
> 
> Andrii's (and others') workflow implies more frequent switching between 
> "projects", rather than working on a single one for a long time.

I don't see how that matters.  We cannot/shouldn't prefer some valid
workflows to other valid workflows, we should try supporting all the
reasonable ones.

> So we can go ahead with idea #1 (which implies no hard change to the 
> API) and see how far we can get push this idea without manual cache 
> invalidation, while keeping decent performance for most of our users.

OK.

Re: master 1e3b0f2: Improve doc strings of project.el

[Dmitry Gutov]

On 05.07.2020 17:26, Eli Zaretskii wrote:

>> The problems I can see with that are:
>>
>> - Since project is not a minor mode, we don't call it in major mode
>> hooks. Which, by itself, is probably a good thing (causing no
>> unnecessary work/slowdown). But that means that in the middle of any
>> session we could have 100s of buffers that don't have any cached project
>> value.
> 
> We have project-find-file and other similar project-* commands that
> could cache that as side effect.

That would still potentially leave hundreds of buffers in which said 
commands have never been used.

>> - A project configuration can change mid-session (due to a customization
>> of some user option, or due to an edit in some .dir-locals.el). Or
>> simply due to 'git init'. As a result, a given file's current project
>> can change. It will be good to be able to support that without
>> necessitating Emacs' restart or requiring manual cache invalidation.
> 
> I don't see how you can avoid some user command which would mean a
> project configuration change.  Alternatively, this could be solved
> lazily, when the cached value is found to be outdated.

Speaking of standards set by other editors, have you ever seen an IDE 
with an "invalidate project cache" button? I haven't.

Luckily, in the common configuration we can rely on 'git ls-files' to be 
pretty fast, though that will only handle projects up to certain size.

I was thinking filenotify could help us with cache invalidation, but 
past discussions have left an impression that we can't fully rely on it 
either.

> We don't need to solve every possible complication up front before we
> decide that a method is workable, we just need to have a vague idea
> how it could be solved when the time comes.

As mentioned, I have that "vague idea" now. Though it will also only 
scale up to a certain number of buffers/directories.

>>> or a directory if all the files in the directory belong
>>> to the project.
>>
>> The problem is in the case of *nested* projects.
> 
> The first part above should take care of that.  The second part was to
> suggest a shortcut: let a directory stand for all the files in it.

For its direct contents, but not subdirectories? Yes, that's what I was 
thinking of as well.

Anyway, the current version of code should support your and Stephen's 
kinds of projects. Someone can try experimenting with writing the 
backend now.

Re: master 1e3b0f2: Improve doc strings of project.el

[Eli Zaretskii]

> Cc: theo@thornhill.no, philip@warpmail.net, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sun, 5 Jul 2020 22:45:19 +0300
> 
> >> - Since project is not a minor mode, we don't call it in major mode
> >> hooks. Which, by itself, is probably a good thing (causing no
> >> unnecessary work/slowdown). But that means that in the middle of any
> >> session we could have 100s of buffers that don't have any cached project
> >> value.
> > 
> > We have project-find-file and other similar project-* commands that
> > could cache that as side effect.
> 
> That would still potentially leave hundreds of buffers in which said 
> commands have never been used.

I don't see how this would happen.  But we can cross that bridge if we
ever get to it.

> >> - A project configuration can change mid-session (due to a customization
> >> of some user option, or due to an edit in some .dir-locals.el). Or
> >> simply due to 'git init'. As a result, a given file's current project
> >> can change. It will be good to be able to support that without
> >> necessitating Emacs' restart or requiring manual cache invalidation.
> > 
> > I don't see how you can avoid some user command which would mean a
> > project configuration change.  Alternatively, this could be solved
> > lazily, when the cached value is found to be outdated.
> 
> Speaking of standards set by other editors, have you ever seen an IDE 
> with an "invalidate project cache" button? I haven't.

I didn't say anything about a need for such a knob.  You are probably
misinterpreting what I wrote above.

> > We don't need to solve every possible complication up front before we
> > decide that a method is workable, we just need to have a vague idea
> > how it could be solved when the time comes.
> 
> As mentioned, I have that "vague idea" now.

Then why are we still arguing?

> Though it will also only scale up to a certain number of
> buffers/directories.

I think you are wrong when you fear this scalability problem; I don't
think it exists.

Re: master 1e3b0f2: Improve doc strings of project.el

[Dmitry Gutov]

On 07.07.2020 18:04, Eli Zaretskii wrote:

>> That would still potentially leave hundreds of buffers in which said
>> commands have never been used.
> 
> I don't see how this would happen.  But we can cross that bridge if we
> ever get to it.

In the same fashion that the author of bug#41029 arrived at the number 
of ~6000 buffers (we could ask). Maybe some long search-and-replace? Or 
something more automated.

I also worry about Tramp buffers, where both process invocations and 
traversals up the directory tree are unavoidably slower.

>>>> - A project configuration can change mid-session (due to a customization
>>>> of some user option, or due to an edit in some .dir-locals.el). Or
>>>> simply due to 'git init'. As a result, a given file's current project
>>>> can change. It will be good to be able to support that without
>>>> necessitating Emacs' restart or requiring manual cache invalidation.
>>>
>>> I don't see how you can avoid some user command which would mean a
>>> project configuration change.  Alternatively, this could be solved
>>> lazily, when the cached value is found to be outdated.
>>
>> Speaking of standards set by other editors, have you ever seen an IDE
>> with an "invalidate project cache" button? I haven't.
> 
> I didn't say anything about a need for such a knob.  You are probably
> misinterpreting what I wrote above.

All right.

If you have suggestions of how we would reliably find outdated cached 
values, please don't hesitate to tell.

Examples of caches we could use:

   * file or directory -> project
   * project -> list of files

>>> We don't need to solve every possible complication up front before we
>>> decide that a method is workable, we just need to have a vague idea
>>> how it could be solved when the time comes.
>>
>> As mentioned, I have that "vague idea" now.
> 
> Then why are we still arguing?

I thought we were having a design discussion.

Does commit 4ca13d9 look good to you? It's been in master shortly after 
my last reply here.

Re: master 1e3b0f2: Improve doc strings of project.el

[Eli Zaretskii]

> Cc: theo@thornhill.no, philip@warpmail.net, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Tue, 7 Jul 2020 18:23:17 +0300
> 
> On 07.07.2020 18:04, Eli Zaretskii wrote:
> 
> >> That would still potentially leave hundreds of buffers in which said
> >> commands have never been used.
> > 
> > I don't see how this would happen.  But we can cross that bridge if we
> > ever get to it.
> 
> In the same fashion that the author of bug#41029 arrived at the number 
> of ~6000 buffers (we could ask).

Please do ask, because I don't see how could hundreds of buffers
belong to a project without being created by commands that know about
the project.

> I also worry about Tramp buffers, where both process invocations and 
> traversals up the directory tree are unavoidably slower.

Once again, as long as a buffer's project is determined when the
buffer is created, I don't think we need to worry about this.

> >> Speaking of standards set by other editors, have you ever seen an IDE
> >> with an "invalidate project cache" button? I haven't.
> > 
> > I didn't say anything about a need for such a knob.  You are probably
> > misinterpreting what I wrote above.
> 
> All right.
> 
> If you have suggestions of how we would reliably find outdated cached 
> values, please don't hesitate to tell.
> 
> Examples of caches we could use:
> 
>    * file or directory -> project
>    * project -> list of files

These should be updated when the file is added to a project or removed
from it.  Or maybe I don't see the difficulties you have in mind.

> Does commit 4ca13d9 look good to you?

Not really:

  . it makes the doc string of project-switch-to-buffer intentionally
    obfuscated by "explaining" stuff in terms of the implementation,
    which makes it not very useful (as I already tried to explain in
    the past)

  . the new doc string is confusing: "if 'project-current' returns the
    same (equal) value" is incomplete, because it doesn't say the same
    as what

So that commit looks like a step backwards to me.

(Also, the log message is in error (mentions the same function twice),
but that's water under the bridge.)

Re: master 1e3b0f2: Improve doc strings of project.el

[Dmitry Gutov]

On 10.07.2020 17:27, Eli Zaretskii wrote:

>> In the same fashion that the author of bug#41029 arrived at the number
>> of ~6000 buffers (we could ask).
> 
> Please do ask, because I don't see how could hundreds of buffers
> belong to a project without being created by commands that know about
> the project.

They can open a bunch of files that reside in a project's directory 
(roughly speaking; if the project backend is directory-based). You don't 
need a command named project-* to do that.

>> I also worry about Tramp buffers, where both process invocations and
>> traversals up the directory tree are unavoidably slower.
> 
> Once again, as long as a buffer's project is determined when the
> buffer is created

I have already explained that we don't do that. project is not a minor 
mode, and we don't waste CPU cycles determining each buffer's project 
eagerly.

Which is most likely a plus for Tramp, we don't need to make buffer 
opening even slower there.

>>>> Speaking of standards set by other editors, have you ever seen an IDE
>>>> with an "invalidate project cache" button? I haven't.
>>>
>>> I didn't say anything about a need for such a knob.  You are probably
>>> misinterpreting what I wrote above.
>>
>> All right.
>>
>> If you have suggestions of how we would reliably find outdated cached
>> values, please don't hesitate to tell.
>>
>> Examples of caches we could use:
>>
>>     * file or directory -> project
>>     * project -> list of files
> 
> These should be updated when the file is added to a project or removed
> from it.  Or maybe I don't see the difficulties you have in mind.

File can be added to a project in a multitude of different ways. By 
using Vim in a terminal, for instance. Or the 'touch' command.

Removal also happens through different venues (removal in Dired, on 'rm' 
in the terminal, or changing the project configuration to ignore the 
said file).

>> Does commit 4ca13d9 look good to you?
> 
> Not really:
> 
>    . it makes the doc string of project-switch-to-buffer intentionally
>      obfuscated by "explaining" stuff in terms of the implementation,
>      which makes it not very useful (as I already tried to explain in
>      the past)

What are you suggesting?

>    . the new doc string is confusing: "if 'project-current' returns the
>      same (equal) value" is incomplete, because it doesn't say the same
>      as what

Same as the current project value, if any (otherwise, same as the value 
returned by the project selection prompt).

Implementation-wise: same as the value returned by (project-current t) 
in the current buffer.

> So that commit looks like a step backwards to me.

Just because you don't like the new docstring?

That's harsh.

> (Also, the log message is in error (mentions the same function twice),
> but that's water under the bridge.)

Yes. Sorry. :-(

Re: master 1e3b0f2: Improve doc strings of project.el

[Eli Zaretskii]

> Cc: theo@thornhill.no, philip@warpmail.net, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Fri, 10 Jul 2020 17:57:12 +0300
> 
> On 10.07.2020 17:27, Eli Zaretskii wrote:
> 
> >> In the same fashion that the author of bug#41029 arrived at the number
> >> of ~6000 buffers (we could ask).
> > 
> > Please do ask, because I don't see how could hundreds of buffers
> > belong to a project without being created by commands that know about
> > the project.
> 
> They can open a bunch of files that reside in a project's directory 
> (roughly speaking; if the project backend is directory-based). You don't 
> need a command named project-* to do that.

We could add something to find-file, like we do for VC-related files.

> >> I also worry about Tramp buffers, where both process invocations and
> >> traversals up the directory tree are unavoidably slower.
> > 
> > Once again, as long as a buffer's project is determined when the
> > buffer is created
> 
> I have already explained that we don't do that. project is not a minor 
> mode, and we don't waste CPU cycles determining each buffer's project 
> eagerly.

Maybe we should waste those CPU cycles, so that they don't add up.

> Which is most likely a plus for Tramp, we don't need to make buffer 
> opening even slower there.

We could have a defcustom to disable that for those who will seldom
if ever use project.el.  And again, we do that for VC.

> >>     * file or directory -> project
> >>     * project -> list of files
> > 
> > These should be updated when the file is added to a project or removed
> > from it.  Or maybe I don't see the difficulties you have in mind.
> 
> File can be added to a project in a multitude of different ways. By 
> using Vim in a terminal, for instance. Or the 'touch' command.
> 
> Removal also happens through different venues (removal in Dired, on 'rm' 
> in the terminal, or changing the project configuration to ignore the 
> said file).

Why should we care about adding or removing a file that are not
visited in Emacs?

> >    . it makes the doc string of project-switch-to-buffer intentionally
> >      obfuscated by "explaining" stuff in terms of the implementation,
> >      which makes it not very useful (as I already tried to explain in
> >      the past)
> 
> What are you suggesting?

Explain what project-current does, most probably, right in that doc
string.

> >    . the new doc string is confusing: "if 'project-current' returns the
> >      same (equal) value" is incomplete, because it doesn't say the same
> >      as what
> 
> Same as the current project value, if any (otherwise, same as the value 
> returned by the project selection prompt).

Please fix the sentence, then.

> > So that commit looks like a step backwards to me.
> 
> Just because you don't like the new docstring?
> 
> That's harsh.

I didn't see any other significant changes, sorry.

Re: master 1e3b0f2: Improve doc strings of project.el

[Dmitry Gutov]

On 10.07.2020 21:14, Eli Zaretskii wrote:

>>> Please do ask, because I don't see how could hundreds of buffers
>>> belong to a project without being created by commands that know about
>>> the project.
>>
>> They can open a bunch of files that reside in a project's directory
>> (roughly speaking; if the project backend is directory-based). You don't
>> need a command named project-* to do that.
> 
> We could add something to find-file, like we do for VC-related files.

I mean, yeah. We could. That would be one way to cache (eagerly). Then 
whatever operation that created those 6000 buffers would pay the price.

> Maybe we should waste those CPU cycles, so that they don't add up.

I'm inclined to say "no" (Tramp is slow enough). But that's a secondary 
decision.

Whether we do cache eagerly (by wasting said CPU cycles in advance), or 
save per-buffer "current project" value on demand, either way that would 
mean caching it.

And either way, that brings up the issue of cache invalidation.

>> Which is most likely a plus for Tramp, we don't need to make buffer
>> opening even slower there.
> 
> We could have a defcustom to disable that for those who will seldom
> if ever use project.el.  And again, we do that for VC.

Tramp has extra-special handling for VC. It probably wouldn't be able to 
do that for all possible project backends.

>>>>      * file or directory -> project
>>>>      * project -> list of files
>>>
>>> These should be updated when the file is added to a project or removed
>>> from it.  Or maybe I don't see the difficulties you have in mind.
>>
>> File can be added to a project in a multitude of different ways. By
>> using Vim in a terminal, for instance. Or the 'touch' command.
>>
>> Removal also happens through different venues (removal in Dired, on 'rm'
>> in the terminal, or changing the project configuration to ignore the
>> said file).
> 
> Why should we care about adding or removing a file that are not
> visited in Emacs?

A file already opened in Emacs might be added to some project this way 
(by editing project config, or doing 'git init'). Then, ideally, two 
things need to happen:

- The "current project" cache of said buffer (if it is already visited) 
needs to be invalidated.
- The "project-files" cache (currently not cached) of said project needs 
to be invalidated as well.

When a file is removed from project, the latter needs to happen.

>>>     . it makes the doc string of project-switch-to-buffer intentionally
>>>       obfuscated by "explaining" stuff in terms of the implementation,
>>>       which makes it not very useful (as I already tried to explain in
>>>       the past)
>>
>> What are you suggesting?
> 
> Explain what project-current does, most probably, right in that doc
> string.

project-current finds the current project of the buffer. Or the project 
to which the current buffer is related. Would that explanation make 
things better?

The exact logic of project-current depends on the configuration of 
project-find-functions.

>>>     . the new doc string is confusing: "if 'project-current' returns the
>>>       same (equal) value" is incomplete, because it doesn't say the same
>>>       as what
>>
>> Same as the current project value, if any (otherwise, same as the value
>> returned by the project selection prompt).
> 
> Please fix the sentence, then.

I'll do my best.

>>> So that commit looks like a step backwards to me.
>>
>> Just because you don't like the new docstring?
>>
>> That's harsh.
> 
> I didn't see any other significant changes, sorry.

You asked for the buffer-matching logic not to depend on 
default-directory anymore. It doesn't.

Now project-switch-to-buffer and project-kill-buffers should work with 
Stephen's backend, as well as any "manually managed project" backend you 
might choose to write.

Re: master 1e3b0f2: Improve doc strings of project.el

[Eli Zaretskii]

> Cc: theo@thornhill.no, philip@warpmail.net, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Fri, 10 Jul 2020 22:36:48 +0300
> 
> >> Which is most likely a plus for Tramp, we don't need to make buffer
> >> opening even slower there.
> > 
> > We could have a defcustom to disable that for those who will seldom
> > if ever use project.el.  And again, we do that for VC.
> 
> Tramp has extra-special handling for VC. It probably wouldn't be able to 
> do that for all possible project backends.

Why assume the worse?  The situation is no different from that with
VC, so I see no reason to assume Tramp will do something similar for
supporting project.el.

> >> Removal also happens through different venues (removal in Dired, on 'rm'
> >> in the terminal, or changing the project configuration to ignore the
> >> said file).
> > 
> > Why should we care about adding or removing a file that are not
> > visited in Emacs?
> 
> A file already opened in Emacs might be added to some project this way 
> (by editing project config, or doing 'git init'). Then, ideally, two 
> things need to happen:
> 
> - The "current project" cache of said buffer (if it is already visited) 
> needs to be invalidated.
> - The "project-files" cache (currently not cached) of said project needs 
> to be invalidated as well.
> 
> When a file is removed from project, the latter needs to happen.

Having commands to add or remove files from the project would allow us
to advertise those commands as the supported means of doing so.  We
could also have a command to "refresh" the project, for those who for
some reasons will prefer doing that externally.  But even without such
a command, we could say that users will have to go the project.el way
to have their project always up to date with the files it consists of,
and discourage doing that by external means.

> >>>     . it makes the doc string of project-switch-to-buffer intentionally
> >>>       obfuscated by "explaining" stuff in terms of the implementation,
> >>>       which makes it not very useful (as I already tried to explain in
> >>>       the past)
> >>
> >> What are you suggesting?
> > 
> > Explain what project-current does, most probably, right in that doc
> > string.
> 
> project-current finds the current project of the buffer. Or the project 
> to which the current buffer is related. Would that explanation make 
> things better?

No, because that's not enough detail.

> The exact logic of project-current depends on the configuration of 
> project-find-functions.

Exactly.  Then mentioning that as well would also help.

> >>>     . the new doc string is confusing: "if 'project-current' returns the
> >>>       same (equal) value" is incomplete, because it doesn't say the same
> >>>       as what
> >>
> >> Same as the current project value, if any (otherwise, same as the value
> >> returned by the project selection prompt).
> > 
> > Please fix the sentence, then.
> 
> I'll do my best.

I see you simply reverted to the original wording, more or less, which
is what prompted me to improve it.  If you are still working on
improving that, I will wait; but if this is what you think the doc
string should say, then I will fix it again to be better.

> >>> So that commit looks like a step backwards to me.
> >>
> >> Just because you don't like the new docstring?
> >>
> >> That's harsh.
> > 
> > I didn't see any other significant changes, sorry.
> 
> You asked for the buffer-matching logic not to depend on 
> default-directory anymore. It doesn't.
> 
> Now project-switch-to-buffer and project-kill-buffers should work with 
> Stephen's backend, as well as any "manually managed project" backend you 
> might choose to write.

Yes, and that's good.  Thanks.

Re: master 1e3b0f2: Improve doc strings of project.el

[Dmitry Gutov]

On 11.07.2020 09:58, Eli Zaretskii wrote:

>> Tramp has extra-special handling for VC. It probably wouldn't be able to
>> do that for all possible project backends.
> 
> Why assume the worse?  The situation is no different from that with
> VC, so I see no reason to assume Tramp will do something similar for
> supporting project.el.

We shall see.

>> - The "current project" cache of said buffer (if it is already visited)
>> needs to be invalidated.
>> - The "project-files" cache (currently not cached) of said project needs
>> to be invalidated as well.
>>
>> When a file is removed from project, the latter needs to happen.
> 
> Having commands to add or remove files from the project would allow us
> to advertise those commands as the supported means of doing so.  We
> could also have a command to "refresh" the project, for those who for
> some reasons will prefer doing that externally.

These are options, of course. But this will make Emacs project support 
more high-maintenance than in "other editors".

Take a random modern editor, open an existing project in it (it will 
show up in the side panel, usually). Then create a new file in its root 
directory using some external means. If it's not ignored by existing 
configuration, it will automatically show in the project drawer as well.

We should support that.

> But even without such
> a command, we could say that users will have to go the project.el way
> to have their project always up to date with the files it consists of,
> and discourage doing that by external means.

What's the project.el way? I suppose we could add hooks to find-file, 
dired-delete-file and after-save-hook in .dir-locals.el, if we give up 
on this kind of feature parity.

>> The exact logic of project-current depends on the configuration of
>> project-find-functions.
> 
> Exactly.  Then mentioning that as well would also help.

Sounds good.

>>>>>      . the new doc string is confusing: "if 'project-current' returns the
>>>>>        same (equal) value" is incomplete, because it doesn't say the same
>>>>>        as what
>>>>
>>>> Same as the current project value, if any (otherwise, same as the value
>>>> returned by the project selection prompt).
>>>
>>> Please fix the sentence, then.
>>
>> I'll do my best.
> 
> I see you simply reverted to the original wording, more or less, which
> is what prompted me to improve it.  If you are still working on
> improving that, I will wait;

I "fixed" it to match the docstring of another command which you also 
improved in the past. Guessing that it could be good enough.

> but if this is what you think the doc
> string should say, then I will fix it again to be better.

That sounds threatening.

Like I said, you're welcome to suggest a better phrasing, for this and 
the other command (as long as the descriptions remain accurate). Go 
ahead and commit the new versions, if you like.

Re: master 1e3b0f2: Improve doc strings of project.el

[Eli Zaretskii]

> Cc: theo@thornhill.no, philip@warpmail.net, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sat, 11 Jul 2020 14:29:04 +0300
> 
> > Having commands to add or remove files from the project would allow us
> > to advertise those commands as the supported means of doing so.  We
> > could also have a command to "refresh" the project, for those who for
> > some reasons will prefer doing that externally.
> 
> These are options, of course. But this will make Emacs project support 
> more high-maintenance than in "other editors".
> 
> Take a random modern editor, open an existing project in it (it will 
> show up in the side panel, usually). Then create a new file in its root 
> directory using some external means. If it's not ignored by existing 
> configuration, it will automatically show in the project drawer as well.
> 
> We should support that.

As long as it makes sense, yes.  But when it starts requiring us to
jump through hoops, we don't need to be afraid of doing things
differently.

> > but if this is what you think the doc
> > string should say, then I will fix it again to be better.
> 
> That sounds threatening.

You are always interpreting what I say in the worst possible way, for
some reason.

> Like I said, you're welcome to suggest a better phrasing, for this and 
> the other command (as long as the descriptions remain accurate). Go 
> ahead and commit the new versions, if you like.

Done.

Re: master 1e3b0f2: Improve doc strings of project.el

[Michael Albinus]

Dmitry Gutov <dgutov@yandex.ru> writes:

>>> Tramp has extra-special handling for VC. It probably wouldn't be able to
>>> do that for all possible project backends.
>> Why assume the worse?  The situation is no different from that with
>> VC, so I see no reason to assume Tramp will do something similar for
>> supporting project.el.
>
> We shall see.

There is `dired-uncache'. Calling it shall be sufficient to remove
respective caches in Tramp, whatever they are.

Best regards, Michael.

Re: master 1e3b0f2: Improve doc strings of project.el

[Dmitry Gutov]

On 11.07.2020 15:35, Eli Zaretskii wrote:

>> Take a random modern editor, open an existing project in it (it will
>> show up in the side panel, usually). Then create a new file in its root
>> directory using some external means. If it's not ignored by existing
>> configuration, it will automatically show in the project drawer as well.
>>
>> We should support that.
> 
> As long as it makes sense, yes.  But when it starts requiring us to
> jump through hoops, we don't need to be afraid of doing things
> differently.

It's a quality of life feature. Of course, we could forgo it, or ask the 
users to jump through hoops themselves on occasion.

Anyway, sounds like there are no better "universal" options at hand than 
filenotify. I'll try to whip up a prototype with it one of these days.

>> Like I said, you're welcome to suggest a better phrasing, for this and
>> the other command (as long as the descriptions remain accurate). Go
>> ahead and commit the new versions, if you like.
> 
> Done.

Thank you.

I've scaled back the explicitness a little: we don't need people to rely 
on (or try to replicate) the exact format of what project-try-vc 
returns. We have enough trouble with user-defined functions returning 
(cons 'transient some-root-dir) already.

Re: master 1e3b0f2: Improve doc strings of project.el

[Eli Zaretskii]

> Cc: theo@thornhill.no, philip@warpmail.net, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sun, 12 Jul 2020 03:48:52 +0300
> 
> I've scaled back the explicitness a little:

I've added back the information you removed.  I think the removal was
a mistake, as explained below.

> we don't need people to rely on (or try to replicate) the exact format of what project-try-vc returns.

This is a reason to have _more_ documentation, not _less_.  E.g., if
you want the users not to assume or do something, say that explicitly
in the documentation.  Or add some other information which will make
it abundantly clear that relying on some detail is a bad idea.

But other than that, if someone needs to write code for the VC project
backend, how can they NOT rely on the form of the object that
project-try-vc returns?  Why should that form, which is important for
handling any VC project, and is thus part of a public API, be left
undocumented?

You anyway cannot conceal from users information about the innards of
a Lisp implementation: since the source code is there for everyone to
see and study, any information you try to conceal will be eventually
found.  All you can do is cause users aggravation by forcing them to
hunt for that information through many rabbit holes.  There are no
advantages to doing this, only disadvantages.

More generally, please put yourself in the shoes of someone who wants
to extend an existing backend, or add a new backend, and try to read
the documentation through their eyes.  They will need to know
something about the methods they should implement/extend, about which
ones are and which ones aren't mandatory.  They will need to know
something about what the project object is or can be, and what minimum
capabilities should it have.  The documentation in project.el
currently has very little to say about this.  Various crucial details,
like how project instances are compared, are never really explained,
and only become evident from reading the code.  And on top of that you
insist on removing details from the documentation that _is_ available.
This makes documentation worse, not better.

> We have enough trouble with user-defined functions returning (cons 'transient some-root-dir) already.

Which trouble is that?  Is it possible that this trouble was caused by
not documenting enough about project.el interfaces and objects?

And as long as we are talking about this: you mentioned the
"transient" project without defining what it is.  That is not useful.
I tried to add some minimal explanation of that, and also fixed
another related omission.

Thanks.

Re: master 1e3b0f2: Improve doc strings of project.el

[Dmitry Gutov]

On 12.07.2020 17:00, Eli Zaretskii wrote:
>> Cc: theo@thornhill.no, philip@warpmail.net, emacs-devel@gnu.org
>> From: Dmitry Gutov <dgutov@yandex.ru>
>> Date: Sun, 12 Jul 2020 03:48:52 +0300
>>
>> I've scaled back the explicitness a little:
> 
> I've added back the information you removed.  I think the removal was
> a mistake, as explained below.

And you were asking why I "interpret what you are saying in the worst 
possible way"? You're behaving like an autocrat.

If I said something is a bad idea, don't go ahead and ignore me. This is 
extremely rude.

>> we don't need people to rely on (or try to replicate) the exact format of what project-try-vc returns.
> 
> This is a reason to have _more_ documentation, not _less_.  E.g., if
> you want the users not to assume or do something, say that explicitly
> in the documentation.  Or add some other information which will make
> it abundantly clear that relying on some detail is a bad idea.

Not documenting internal information is a valid choice.

> But other than that, if someone needs to write code for the VC project
> backend, how can they NOT rely on the form of the object that
> project-try-vc returns?

Nobody should "write code for the VC project backend" except its 
author(s). And this backend is maintained strictly within project.el.

Whatever fuctionality users need to implement, should target the open 
API. Which does not depend on the shape of the return value of 
project-try-vc.

> Why should that form, which is important for
> handling any VC project, and is thus part of a public API, be left
> undocumented?

No, they can't. That information can freely change between versions, so 
any reliance on it will create compatibility problems when upgrading.

> You anyway cannot conceal from users information about the innards of
> a Lisp implementation: since the source code is there for everyone to
> see and study, any information you try to conceal will be eventually
> found.  All you can do is cause users aggravation by forcing them to
> hunt for that information through many rabbit holes.  There are no
> advantages to doing this, only disadvantages.

If my method of saying "don't rely on this format" is insufficient, 
please go ahead and add some more clarifications on that part. But 
without documenting this said value format.

> More generally, please put yourself in the shoes of someone who wants
> to extend an existing backend, or add a new backend, and try to read
> the documentation through their eyes.

If they create a new backend, they will read the code of the existing 
one. They're free to copy, but they won't be bitten in the ass when the 
format changes, because their copy will not.

> They will need to know
> something about the methods they should implement/extend, about which
> ones are and which ones aren't mandatory.

That is orthogonal to this issue.

> They will need to know
> something about what the project object is or can be, and what minimum
> capabilities should it have.

Hence the description in my latest commit.

> The documentation in project.el
> currently has very little to say about this.  Various crucial details,
> like how project instances are compared, are never really explained,
> and only become evident from reading the code.
Some extra guidance can be helpful. That doesn't mean your particular 
choice is the best one.

>> We have enough trouble with user-defined functions returning (cons 'transient some-root-dir) already.
> 
> Which trouble is that?

People try to customize the root-finding logic this way, and as a result 
get a project backend that behaves more slowly than the VC one. And that 
affects all of project-* commands that they might like to use.

> Is it possible that this trouble was caused by
> not documenting enough about project.el interfaces and objects?

Perhaps. It's undoubtedly (as everywhere) both a documentation and user 
comprehension problem. I've tried to improve the docs.

> And as long as we are talking about this: you mentioned the
> "transient" project without defining what it is.  That is not useful.
> I tried to add some minimal explanation of that, and also fixed
> another related omission.

I would say thank you, but you again documented the return value there.

The value is *irrelevant*, and it is, again, internal to the backend.

Re: master 1e3b0f2: Improve doc strings of project.el

[Eli Zaretskii]

> Cc: theo@thornhill.no, philip@warpmail.net, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sun, 12 Jul 2020 17:16:19 +0300
> 
> If I said something is a bad idea, don't go ahead and ignore me. This is 
> extremely rude.

This goes both ways.

> Not documenting internal information is a valid choice.

It isn't internal.  Anyone who wants to use the return value will need
to understand its possible form(s).

> > But other than that, if someone needs to write code for the VC project
> > backend, how can they NOT rely on the form of the object that
> > project-try-vc returns?
> 
> Nobody should "write code for the VC project backend" except its 
> author(s).

This is a community project, and everybody is welcome to contribute
code to any part of Emacs.  Restricting some parts of code to a single
person is not a good way of ensuring Emacs's future.

> Whatever fuctionality users need to implement, should target the open 
> API. Which does not depend on the shape of the return value of 
> project-try-vc.

With respect, I disagree, having read the code and having considered
how I would implement something related to it.  That is the single
most important reason why I'm trying to improve the documentation of
this package: to make it easier for others to expand and extend it.

> > Why should that form, which is important for
> > handling any VC project, and is thus part of a public API, be left
> > undocumented?
> 
> No, they can't. That information can freely change between versions, so 
> any reliance on it will create compatibility problems when upgrading.

Then let's say that.  But concealing the information will not solve
the problem, because you cannot conceal it.  By not mentioning it and
the caveats to go with it we will be punished twice: we will make it
harder for people to understand the current code and contribute to it,
and we will not make sure they realize that relying on these details
is not a good idea.

> > You anyway cannot conceal from users information about the innards of
> > a Lisp implementation: since the source code is there for everyone to
> > see and study, any information you try to conceal will be eventually
> > found.  All you can do is cause users aggravation by forcing them to
> > hunt for that information through many rabbit holes.  There are no
> > advantages to doing this, only disadvantages.
> 
> If my method of saying "don't rely on this format" is insufficient, 
> please go ahead and add some more clarifications on that part. But 
> without documenting this said value format.

You cannot usefully tell people to not rely on something without
describing that something.  And there's the other disadvantage to
concealing information: obfuscation of the code.

> > More generally, please put yourself in the shoes of someone who wants
> > to extend an existing backend, or add a new backend, and try to read
> > the documentation through their eyes.
> 
> If they create a new backend, they will read the code of the existing 
> one.

That's not how we encourage extending Emacs.  Some minimal
documentation is needed before we can in good faith tell users to read
the code, and project.el currently falls short of that minimum, IMO.

> > They will need to know
> > something about the methods they should implement/extend, about which
> > ones are and which ones aren't mandatory.
> 
> That is orthogonal to this issue.

I don't think it's orthogonal, I think it's all part of the same
issue.

> > They will need to know
> > something about what the project object is or can be, and what minimum
> > capabilities should it have.
> 
> Hence the description in my latest commit.

It was insufficient.  And what we have now, after my changes, is still
insufficient, IMO.  More work is needed.

> >> We have enough trouble with user-defined functions returning (cons 'transient some-root-dir) already.
> > 
> > Which trouble is that?
> 
> People try to customize the root-finding logic this way, and as a result 
> get a project backend that behaves more slowly than the VC one. And that 
> affects all of project-* commands that they might like to use.

People learn by doing and by making mistakes.  It is not a catastrophe
if their first attempt is suboptimal.  Making the documentation better
might be one way of helping them understand how to DTRT.

> > And as long as we are talking about this: you mentioned the
> > "transient" project without defining what it is.  That is not useful.
> > I tried to add some minimal explanation of that, and also fixed
> > another related omission.
> 
> I would say thank you, but you again documented the return value there.

Of course.  It's a value that others need to be aware of.

> The value is *irrelevant*, and it is, again, internal to the backend.

Which backend is that?  AFAICT, project-current is backend-agnostic.

Re: master 1e3b0f2: Improve doc strings of project.el

[Dmitry Gutov]

On 11.07.2020 15:59, Michael Albinus wrote:
> There is `dired-uncache'. Calling it shall be sufficient to remove
> respective caches in Tramp, whatever they are.

Cache invalidation aside, when talking about Tramp I was more curious 
whether its caching would be enough to speed up repeated invocations of 
project-try-vc. Or even said invocations from files belonging to the 
same directory, for instance.

Right now it relies on vc-root being cached, which allows it to skip 
some operations most of the time. But it also performs remote IO to read 
local variable values, and to fetch the list of git submodules.

And if we allow customizing root markers, that would stop make 
project-vc call locate-dominating-file directly. I'm guess that would 
circumvent any existing caches in Tramp. Please correct me if I'm wrong.

Re: master 1e3b0f2: Improve doc strings of project.el

[Dmitry Gutov]

On 12.07.2020 17:47, Eli Zaretskii wrote:

>> If I said something is a bad idea, don't go ahead and ignore me. This is
>> extremely rude.
> 
> This goes both ways.

It never does with you.

If I work on parts of Emacs I'm not responsible for, I make doubly sure 
to check with authors/maintainers.

How about you extend me this courtesy? I'm the author and the maintainer 
of this package.

>> Not documenting internal information is a valid choice.
> 
> It isn't internal.  Anyone who wants to use the return value will need
> to understand its possible form(s).

Yes, it is. Apparently you don't understand how cl-generic works.

>>> But other than that, if someone needs to write code for the VC project
>>> backend, how can they NOT rely on the form of the object that
>>> project-try-vc returns?
>>
>> Nobody should "write code for the VC project backend" except its
>> author(s).
> 
> This is a community project, and everybody is welcome to contribute
> code to any part of Emacs.  Restricting some parts of code to a single
> person is not a good way of ensuring Emacs's future.

Not the person. The place. project-vc lives in project.el.

Internal information can reside there without being documented, if all 
of its producers and consumers also reside there.

>> Whatever fuctionality users need to implement, should target the open
>> API. Which does not depend on the shape of the return value of
>> project-try-vc.
> 
> With respect, I disagree, having read the code and having considered
> how I would implement something related to it.  That is the single
> most important reason why I'm trying to improve the documentation of
> this package: to make it easier for others to expand and extend it.

If you had asked me questions like "how do I implement this or that", 
none of my answers would have included the format of the return value of 
project-try-vc.

So the docstring shouldn't either.

You can still ask those questions.

>>> Why should that form, which is important for
>>> handling any VC project, and is thus part of a public API, be left
>>> undocumented?
>>
>> No, they can't. That information can freely change between versions, so
>> any reliance on it will create compatibility problems when upgrading.
> 
> Then let's say that.  But concealing the information will not solve
> the problem, because you cannot conceal it.  By not mentioning it and
> the caveats to go with it we will be punished twice: we will make it
> harder for people to understand the current code and contribute to it,

There's nothing hard in understanding what the current code returns: you 
navigate to the function's definition and look at the end.

> and we will not make sure they realize that relying on these details
> is not a good idea.

Like I said, if you're not satisfied with how that recommendation is 
conveyed, feel free to add clarifications. *Without* giving the example 
of said values in the docstrings.

>> If my method of saying "don't rely on this format" is insufficient,
>> please go ahead and add some more clarifications on that part. But
>> without documenting this said value format.
> 
> You cannot usefully tell people to not rely on something without
> describing that something.

And yet, somehow, parents around the world manage to tell their children 
not to use swear words without swearing profusely themselves.

> And there's the other disadvantage to
> concealing information: obfuscation of the code.

The code is not the issue.

>>> More generally, please put yourself in the shoes of someone who wants
>>> to extend an existing backend, or add a new backend, and try to read
>>> the documentation through their eyes.
>>
>> If they create a new backend, they will read the code of the existing
>> one.
> 
> That's not how we encourage extending Emacs.  Some minimal
> documentation is needed before we can in good faith tell users to read
> the code, and project.el currently falls short of that minimum, IMO.

You seem to be criticizing something else than what I said.

And I never said we don't need documentation.

>>> They will need to know
>>> something about the methods they should implement/extend, about which
>>> ones are and which ones aren't mandatory.
>>
>> That is orthogonal to this issue.
> 
> I don't think it's orthogonal, I think it's all part of the same
> issue.

Mentioning (vc . "path/to/root") somehow helps the users know which 
methods they need to implement?

You must be kidding.

>>> They will need to know
>>> something about what the project object is or can be, and what minimum
>>> capabilities should it have.
>>
>> Hence the description in my latest commit.
> 
> It was insufficient.  And what we have now, after my changes, is still
> insufficient, IMO.  More work is needed.

Of course. On *different* aspects.

>>>> We have enough trouble with user-defined functions returning (cons 'transient some-root-dir) already.
>>>
>>> Which trouble is that?
>>
>> People try to customize the root-finding logic this way, and as a result
>> get a project backend that behaves more slowly than the VC one. And that
>> affects all of project-* commands that they might like to use.
> 
> People learn by doing and by making mistakes.  It is not a catastrophe
> if their first attempt is suboptimal.  Making the documentation better
> might be one way of helping them understand how to DTRT.

Right.

>>> And as long as we are talking about this: you mentioned the
>>> "transient" project without defining what it is.  That is not useful.
>>> I tried to add some minimal explanation of that, and also fixed
>>> another related omission.
>>
>> I would say thank you, but you again documented the return value there.
> 
> Of course.  It's a value that others need to be aware of.
> 
>> The value is *irrelevant*, and it is, again, internal to the backend.
> 
> Which backend is that?

The 'transient' backend.

Okay, internal to project.el, if we want to be more precise.

 > AFAICT, project-current is backend-agnostic.

Except when we want to return some project instance where none were 
detected.

Re: master 1e3b0f2: Improve doc strings of project.el

[Eli Zaretskii]

> Cc: theo@thornhill.no, philip@warpmail.net, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sun, 12 Jul 2020 18:08:50 +0300
> 
> If I work on parts of Emacs I'm not responsible for, I make doubly sure 
> to check with authors/maintainers.
> 
> How about you extend me this courtesy? I'm the author and the maintainer 
> of this package.

I do.  This longish and frustrating discussion is the best evidence to
that.

> >> Not documenting internal information is a valid choice.
> > 
> > It isn't internal.  Anyone who wants to use the return value will need
> > to understand its possible form(s).
> 
> Yes, it is. Apparently you don't understand how cl-generic works.

Please drop the attitude.

> >>> But other than that, if someone needs to write code for the VC project
> >>> backend, how can they NOT rely on the form of the object that
> >>> project-try-vc returns?
> >>
> >> Nobody should "write code for the VC project backend" except its
> >> author(s).
> > 
> > This is a community project, and everybody is welcome to contribute
> > code to any part of Emacs.  Restricting some parts of code to a single
> > person is not a good way of ensuring Emacs's future.
> 
> Not the person. The place. project-vc lives in project.el.

Irrelevant.  I meant people other than the author of the code, the
person.  Those others will need more detailed documentation than what
you are prepared to provide.

> Internal information can reside there without being documented, if all 
> of its producers and consumers also reside there.

This is a new definition of what is "internal", and I don't agree with
it.  Our accepted definition is different.

> > With respect, I disagree, having read the code and having considered
> > how I would implement something related to it.  That is the single
> > most important reason why I'm trying to improve the documentation of
> > this package: to make it easier for others to expand and extend it.
> 
> If you had asked me questions like "how do I implement this or that", 
> none of my answers would have included the format of the return value of 
> project-try-vc.

I know.  Which is why I don't think your method of answering such
questions is useful enough to be used as basis for good documentation.

> > Then let's say that.  But concealing the information will not solve
> > the problem, because you cannot conceal it.  By not mentioning it and
> > the caveats to go with it we will be punished twice: we will make it
> > harder for people to understand the current code and contribute to it,
> 
> There's nothing hard in understanding what the current code returns: you 
> navigate to the function's definition and look at the end.

Documentation exists so that I won't need to follow a chain of several
functions to find the answer to a simple question.  (And no, just
reading the code of project-try-vc is not enough, not unless you know
by heart what each one of the functions it calls does and returns.)

> > and we will not make sure they realize that relying on these details
> > is not a good idea.
> 
> Like I said, if you're not satisfied with how that recommendation is 
> conveyed, feel free to add clarifications. *Without* giving the example 
> of said values in the docstrings.

I think describing the values returned by public functions is a large
part of good documentation, it clarifies quite a lot in a very
efficient way (a picture is worth a thousand words).

> >> If my method of saying "don't rely on this format" is insufficient,
> >> please go ahead and add some more clarifications on that part. But
> >> without documenting this said value format.
> > 
> > You cannot usefully tell people to not rely on something without
> > describing that something.
> 
> And yet, somehow, parents around the world manage to tell their children 
> not to use swear words without swearing profusely themselves.

Sure, it isn't needed, not after the child him/herself swears, which
is the trigger for the parents telling them not to.

> > And there's the other disadvantage to
> > concealing information: obfuscation of the code.
> 
> The code is not the issue.

Of course it is!  Documentation is a significant aid in understanding
the code.

> >> If they create a new backend, they will read the code of the existing
> >> one.
> > 
> > That's not how we encourage extending Emacs.  Some minimal
> > documentation is needed before we can in good faith tell users to read
> > the code, and project.el currently falls short of that minimum, IMO.
> 
> You seem to be criticizing something else than what I said.

I'm criticizing the "they will read the code" part.

> And I never said we don't need documentation.

I never said you did.

> Okay, internal to project.el, if we want to be more precise.
> 
>  > AFAICT, project-current is backend-agnostic.
> 
> Except when we want to return some project instance where none were 
> detected.

Which is exactly what the text I added says.

Re: master 1e3b0f2: Improve doc strings of project.el

[Eli Zaretskii]

> Date: Sun, 12 Jul 2020 18:36:33 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: philip@warpmail.net, theo@thornhill.no, emacs-devel@gnu.org
> 
> > There's nothing hard in understanding what the current code returns: you 
> > navigate to the function's definition and look at the end.
> 
> Documentation exists so that I won't need to follow a chain of several
> functions to find the answer to a simple question.  (And no, just
> reading the code of project-try-vc is not enough, not unless you know
> by heart what each one of the functions it calls does and returns.)

Can we approach this from a different angle, please?  Can you tell why
you are so opposed to having this small piece of information in the
doc strings?  What harm can it do that you consider unacceptable?

I added that piece of information because without it I couldn't wrap
my head around what the code does, in particular when it tries to
determine which projects are equal to which.  I reckon if this was
difficult for me, there will be others who could benefit from having
that info.  I guess you don't have this problem, because you wrote
most of that code.  But even so, why is it so important not to divulge
that information?

Re: master 1e3b0f2: Improve doc strings of project.el

[Michael Albinus]

Dmitry Gutov <dgutov@yandex.ru> writes:

Hi Dmitry,

> Cache invalidation aside, when talking about Tramp I was more curious
> whether its caching would be enough to speed up repeated invocations
> of project-try-vc. Or even said invocations from files belonging to
> the same directory, for instance.
>
> Right now it relies on vc-root being cached, which allows it to skip
> some operations most of the time. But it also performs remote IO to
> read local variable values, and to fetch the list of git submodules.
>
> And if we allow customizing root markers, that would stop make
> project-vc call locate-dominating-file directly. I'm guess that would
> circumvent any existing caches in Tramp. Please correct me if I'm wrong.

Tramp caching is just about file information, nothing else. It does not
cache anything about vc git submodule lists or locate-dominating-file
results or whatever.

Best regards, Michael.

Re: master 1e3b0f2: Improve doc strings of project.el

[Dmitry Gutov]

On 12.07.2020 19:17, Eli Zaretskii wrote:

> Can we approach this from a different angle, please?  Can you tell why
> you are so opposed to having this small piece of information in the
> doc strings?  What harm can it do that you consider unacceptable?

Let's try this again.

This information is private (or "internal") similar to when we designate 
variables or function private: we don't want people to rely on it 
because a) that might lead to wrong decisions, b) that might lead to 
breakage because this information is subject to changing at will. For 
example, I'm considering turning the whole value into a list (it's 
currently a shallow cons) and adding the VC backend symbol to the 
project-try-vc's return value as the second element to save some CPU cycles.

Here are things we want third-party code to be able to do:

- Consume the public project APIs, which are currently limited to 
'project-current' and several generic functions defined for project 
instance values, to define new commands or supplement existing ones 
(e.g. to query the current project for its root or its list of files).

- Create new project backends. Note that we don't want *everybody* to do 
that because creating a well-performing backend is a significant amount 
of work (unless the user has very particular needs and works with small 
projects). But we want this to be a solid option, and it would be ideal 
if, say, a handful of backends become popular someday. To create a 
backend, one needs to choose the format of its instance value (which 
shouldn't match any of the existing formats), write its 
project-find-functions element, and define the project-root override 
that dispatches on its instance value. Preferably, also implement 
overrides for project-ignores and project-files.

Neither of these options requires knowing the exact format of 
project-try-vc's return value, if only to stay away from it. But the 
odds of a collision seem very low (and there are ways to reduce them 
even further).

Here are, on the other hand, things that people generally shouldn't do, 
and yet I have seen them done:

- Extract data from the project instance value directly, instead of 
going through e.g. 'project-root'. For instance, writing (cdr 
(project-current t)) instead of (project-root (project-current t)). The 
former is bad code.

- Write a new project-find-functions element which returns an instance 
value that belongs to a built-in backend (usually, the 'transient' one). 
AFAICT, this is usually done by cargo culting, without understanding of 
the consequences.

Both of these require knowing the exact value formats of project 
instances used by built-in backends. Thus, documenting them will 
encourage these usages.

I don't want to prohibit the last option outright, but we shouldn't 
leave any impression that it's legitimate either. And most problems that 
people try to solve this way (predominantly: customizing project root 
markers, AFAIK) should be filed as bug reports/feature requests.

> I added that piece of information because without it I couldn't wrap
> my head around what the code does, in particular when it tries to
> determine which projects are equal to which.

But it's the wrong direction.

I mean, you *can* look at the shape of the instance returned by 
project-try-vc (which is trivial to do: evaluate the form 
(project-current t) and look at the echo area), but it doesn't give you 
any guarantee: after all, we need to make sure 
project-switch-to-buffer's behavior makes sense for different backends, 
even ones we didn't write.

So documenting that one backend's instance format is not a solution.

Instead, it seems we have to document that said value should uniquely 
identify the returned project and be "stable": not change over time or 
when fetched from different buffers belonging to the same project. That 
also means that the value shouldn't contain any caches inside of it.

And once we add these requirements, the answer to "which projects are 
equal" translates to:

- The backend is the same,
- The instances describe the same project, but the exact fashion depends 
on the backend. As it would be anyway.

> I reckon if this was
> difficult for me, there will be others who could benefit from having
> that info.
This kind of internal information might be better suited for code 
comments. I'm not sure where you could put this particular bit of 
information, where it would both be relevant and yet not obvious from 
the nearby code, though.

Re: master 1e3b0f2: Improve doc strings of project.el

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > Yes, it is. Apparently you don't understand how cl-generic works.

  > Please drop the attitude.

+1.

Dmitry, it looks like you believe Eli was mistaken in what he said
about the return value of a certain function.

That could be so.  We are all human, as far as I know, so we all make
mistakes.  I have nothing to say about that function's return value;
I don't even know the function's name.

My point is that there are kind and harsh ways to say that someone
made a mistake.

"Apparently, you don't understand how this works" is a harsh way.  How
so?  Because it starts with evidence of a narrow mistake and generalizes
to accuse a much wider one.  That is exaggeration.

It is certainly possible to basically understand a topic and get one
detail wrong.  You can't justify the conclusion that a person doesn't
understand the topic, overall, just because of a mistake about a
detail.

What would be a kind way?  "With all due respect, I think you are mistaken
about how people use the return value of whatever-it-is."

I think that would have served the same purpose in the discussion of
this technical issue, while making it easy for others to agree with you
if you are right.

Would you please make an effort to change your habits that are harsh?






-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: master 1e3b0f2: Improve doc strings of project.el

[Eli Zaretskii]

> Cc: philip@warpmail.net, theo@thornhill.no, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Mon, 13 Jul 2020 04:51:33 +0300
> 
> On 12.07.2020 19:17, Eli Zaretskii wrote:
> 
> > Can we approach this from a different angle, please?  Can you tell why
> > you are so opposed to having this small piece of information in the
> > doc strings?  What harm can it do that you consider unacceptable?
> 
> Let's try this again.

Sadly, you never answered my questions.  I still hope that you do,
because I don't understand the staunch objections, to the degree that
causes extremely unkind responses, to improve the doc string by saying
something that is both simple and correct.

Instead of answering my practical questions, which could have led us
towards some acceptable compromise, you've elected to reiterate your
views, which just makes another round of the same dispute we've been
having, without any hope for reconciling the differences.  Because it
turns out we disagree even more than I thought, on the very principles
behind this, see below.

Please do consider answering my questions.  I think it is important to
do that, so that we will be able to usefully discuss similar issues in
the future.  We disagree about the principles, so the only hope is to
find some practical compromise with which we will be able to live.

> This information is private (or "internal") similar to when we designate 
> variables or function private: we don't want people to rely on it 
> because a) that might lead to wrong decisions, b) that might lead to 
> breakage because this information is subject to changing at will. For 
> example, I'm considering turning the whole value into a list (it's 
> currently a shallow cons) and adding the VC backend symbol to the 
> project-try-vc's return value as the second element to save some CPU cycles.

When the form of the return value changes, we will update the
documentation.  This kind of thing happens all the time in Emacs
development, and is nothing to be afraid of.  Especially if you
consider some information "private", which means we under no
obligation to preserve its form.  We even have an "Internals" chapter
in the ELisp manual (woefully incomplete, but not because we decided
not to tell those missing details).

This is why I'm frankly astonished by your opposition to document the
return value.  Once again: what's the catastrophe in doing that?

> Here are things we want third-party code to be able to do:
> 
> - Consume the public project APIs, which are currently limited to 
> 'project-current' and several generic functions defined for project 
> instance values, to define new commands or supplement existing ones 
> (e.g. to query the current project for its root or its list of files).

Where is this documented?  I don't think I see it.

> - Create new project backends. Note that we don't want *everybody* to do 
> that because creating a well-performing backend is a significant amount 
> of work (unless the user has very particular needs and works with small 
> projects). But we want this to be a solid option, and it would be ideal 
> if, say, a handful of backends become popular someday. To create a 
> backend, one needs to choose the format of its instance value (which 
> shouldn't match any of the existing formats), write its 
> project-find-functions element, and define the project-root override 
> that dispatches on its instance value. Preferably, also implement 
> overrides for project-ignores and project-files.

Where is it documented what needs to be done for creating a new
backend?  Again, I don't think I see it documented.

> Neither of these options requires knowing the exact format of 
> project-try-vc's return value, if only to stay away from it. But the 
> odds of a collision seem very low (and there are ways to reduce them 
> even further).

These are just some of the possible development directions.  There are
others, no less legitimate ones.  In particular, minor improvements to
the code, which don't change the overall design and implementation.
These and others do require to be aware of the form of the return
value, or at least they can be helped by knowing that.

Moreover, I'm quite sure that if you try to document the missing
pieces mentioned above in a way that is useful for potential
contributors, you will find soon enough that the policy of hiding
useful information about interfaces, even "internal" ones, works
against you: it makes it much harder to say something useful about how
the various objects and methods interact or should interact.  You will
eventually be forced to say "read the code" all the time.  Which means
the documentation cannot do one of its important jobs: help to
understand the code.

But more generally, it is not our prerogative to decide for others in
what directions they may or may not take the development in the
future.  This is not our pet project, this is a community project, and
any member of the community is welcome to propose changes to any part
of Emacs.  If the change is useful for the users, and is done cleanly,
we welcome and accept such changes -- this is how Emacs was and is
being developed for 40 years.  It is what brought us where we are
today.

So I very much object to the policy you describe above, which is that
you personally decide up front what kinds of changes are and aren't
legitimate, based on some principles that are not necessarily shared
by others (certainly aren't documented anywhere in the Emacs
contribution-related docs), and then unilaterally implement this rigid
policy, right down to the level of the smallest detail in the doc
strings.  This is an extremely radical and harsh policy, which I don't
think ever existed in Emacs, nor do I think it should exist.  If
accepted, it will IMO harm the Emacs development in the long run.

> Here are, on the other hand, things that people generally shouldn't do, 
> and yet I have seen them done:
> 
> - Extract data from the project instance value directly, instead of 
> going through e.g. 'project-root'. For instance, writing (cdr 
> (project-current t)) instead of (project-root (project-current t)). The 
> former is bad code.
> 
> - Write a new project-find-functions element which returns an instance 
> value that belongs to a built-in backend (usually, the 'transient' one). 
> AFAICT, this is usually done by cargo culting, without understanding of 
> the consequences.

See above: I disagree very much with saying flatly that we don't want
these.  These types of changes are not "verboten" in principle, they
should be each one of them judged on its own right, and if done
cleanly and efficiently, they are not in any way "wrong".

In a Free Software project, you cannot forbid people from trying
different development directions, even dangerous ones, you can only
warn them about the dangers by documenting them.  Which is one more
reason for having good, detailed documentation.  Refraining from
mentioning something because you think it is dangerous is the wrong
way, because people are curious and will find what you are trying to
hide.  When they do, no amount of "I didn't want you to know that, so
don't do that" will help, believe me.

> Both of these require knowing the exact value formats of project 
> instances used by built-in backends. Thus, documenting them will 
> encourage these usages.
> 
> I don't want to prohibit the last option outright, but we shouldn't 
> leave any impression that it's legitimate either. And most problems that 
> people try to solve this way (predominantly: customizing project root 
> markers, AFAIK) should be filed as bug reports/feature requests.

As I already said several times, the right way of doing this without
throwing the baby with the bathwater is to document the value and then
tell this form is subject to change without notice.  It is a very
simple solution that keeps both goals in our sights without losing any
of them.

> > I added that piece of information because without it I couldn't wrap
> > my head around what the code does, in particular when it tries to
> > determine which projects are equal to which.
> 
> But it's the wrong direction.
> 
> I mean, you *can* look at the shape of the instance returned by 
> project-try-vc (which is trivial to do: evaluate the form 
> (project-current t) and look at the echo area)

If the form of a return value can only be obtained by running the
code, it is IMO a clear sign that the documentation fails to do its
job.  Which is exactly why I decided to document it.

> but it doesn't give you any guarantee: after all, we need to make
> sure project-switch-to-buffer's behavior makes sense for different
> backends, even ones we didn't write.

Irrelevant: the information I added was about a single _existing_
backend.  If you have proposals how to say something sensible and
helpful about others, I'm all ears.

> So documenting that one backend's instance format is not a solution.

It is a solution for those who want to understand better how the
_existing_ code works.  Admittedly, it's a small step, but it's better
than nothing, and it's a step in the right direction.

> Instead, it seems we have to document that said value should uniquely 
> identify the returned project and be "stable": not change over time or 
> when fetched from different buffers belonging to the same project. That 
> also means that the value shouldn't contain any caches inside of it.
> 
> And once we add these requirements, the answer to "which projects are 
> equal" translates to:
> 
> - The backend is the same,
> - The instances describe the same project, but the exact fashion depends 
> on the backend. As it would be anyway.

Of course, none of this is described anywhere in project.el.  It isn't
even clear what objects are part of the design, what are the
requirements from each object, how to establish of some arbitrary Lisp
object is "an instance of a project", how to compare them, etc. etc.
In sum, the crucial information that is expected from any object
system is missing, and potential developers are evidently expected to
glean it all by reading the code.  How does that make sense, and how
is it TRT to refuse improvements of the documentation when what we
have is this imperfect situation?

> > I reckon if this was
> > difficult for me, there will be others who could benefit from having
> > that info.
> This kind of internal information might be better suited for code 
> comments.

But it is missing from comments as well.  It is simply not there.

> I'm not sure where you could put this particular bit of
> information, where it would both be relevant and yet not obvious from 
> the nearby code, though.

_Any_ place is better than _no_ place.  "Not sure where to put" is not
the same as "let's not document this".

Re: master 1e3b0f2: Improve doc strings of project.el

[Gregory Heytings via Emacs development discussions.]

>
>> This information is private (or "internal") similar to when we 
>> designate variables or function private: we don't want people to rely 
>> on it because a) that might lead to wrong decisions, b) that might lead 
>> to breakage because this information is subject to changing at will. 
>> For example, I'm considering turning the whole value into a list (it's 
>> currently a shallow cons) and adding the VC backend symbol to the 
>> project-try-vc's return value as the second element to save some CPU 
>> cycles.
>
> When the form of the return value changes, we will update the 
> documentation.  This kind of thing happens all the time in Emacs 
> development, and is nothing to be afraid of.  Especially if you consider 
> some information "private", which means we under no obligation to 
> preserve its form.  We even have an "Internals" chapter in the ELisp 
> manual (woefully incomplete, but not because we decided not to tell 
> those missing details).
>
> This is why I'm frankly astonished by your opposition to document the 
> return value.  Once again: what's the catastrophe in doing that?
>

I might be wrong, but I think Dmitry's viewpoint is that if he documents 
X, then X becomes part of the API.  Other developers will (or might) use X 
in their code, and he would not be completely free to change X anymore. 
He would have to keep X in a number of future versions of his code, 
document X as being obsolete for a while, create Y in parallel, and only 
later remove X from his code.

IOW, it is not only a matter of updating the documentation, it is a matter 
of updating all code that relied on that documentation.

(Note that this does NOT mean I agree with Dmitry.)

Gregory

Re: master 1e3b0f2: Improve doc strings of project.el

[Eli Zaretskii]

On July 13, 2020 10:11:21 AM GMT+03:00, "Gregory Heytings via Emacs development discussions." <emacs-devel@gnu.org> wrote:
> 
> >
> >> This information is private (or "internal") similar to when we 
> >> designate variables or function private: we don't want people to
> rely 
> >> on it because a) that might lead to wrong decisions, b) that might
> lead 
> >> to breakage because this information is subject to changing at
> will. 
> >> For example, I'm considering turning the whole value into a list
> (it's 
> >> currently a shallow cons) and adding the VC backend symbol to the 
> >> project-try-vc's return value as the second element to save some
> CPU 
> >> cycles.
> >
> > When the form of the return value changes, we will update the 
> > documentation.  This kind of thing happens all the time in Emacs 
> > development, and is nothing to be afraid of.  Especially if you
> consider 
> > some information "private", which means we under no obligation to 
> > preserve its form.  We even have an "Internals" chapter in the ELisp
> 
> > manual (woefully incomplete, but not because we decided not to tell 
> > those missing details).
> >
> > This is why I'm frankly astonished by your opposition to document
> the 
> > return value.  Once again: what's the catastrophe in doing that?
> >
> 
> I might be wrong, but I think Dmitry's viewpoint is that if he
> documents 
> X, then X becomes part of the API.  Other developers will (or might)
> use X 
> in their code, and he would not be completely free to change X
> anymore. 
> He would have to keep X in a number of future versions of his code, 
> document X as being obsolete for a while, create Y in parallel, and
> only 
> later remove X from his code.


I tried to explain why these fears have no basis: if we clearly say that some details are subject to change without notice, we can change them when we need to.

IOW, doing that does NOT make these details part of the public API which we promise shall remain stable.

Re: master 1e3b0f2: Improve doc strings of project.el

[Gregory Heytings via Emacs development discussions.]

>
>> I might be wrong, but I think Dmitry's viewpoint is that if he 
>> documents X, then X becomes part of the API.  Other developers will (or 
>> might) use X in their code, and he would not be completely free to 
>> change X anymore. He would have to keep X in a number of future 
>> versions of his code, document X as being obsolete for a while, create 
>> Y in parallel, and only later remove X from his code.
>
> I tried to explain why these fears have no basis: if we clearly say that 
> some details are subject to change without notice, we can change them 
> when we need to.
>
> IOW, doing that does NOT make these details part of the public API which 
> we promise shall remain stable.
>

I might be wrong, but in that case I think Dmitry's viewpoint is that this 
information should not be placed in the doc strings, but in the comments 
in the source file.

Again, I'm not Dmitry, and this does not mean I agree with him.

Gregory

Re: master 1e3b0f2: Improve doc strings of project.el

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1600 bytes --]

On Mon, Jul 13, 2020 at 07:11:21AM +0000, Gregory Heytings via Emacs development discussions. wrote:

[...]

> I might be wrong, but I think Dmitry's viewpoint is that if he
> documents X, then X becomes part of the API.  Other developers will
> (or might) use X in their code, and he would not be completely free
> to change X anymore [...]

Yes. But I think the core of the disagreement lies in *who*
decides where the line in the sand is. Dmitry seems to lean
to the side "the library designer does", while Eli to the one
"it's part of a collective process".

I'll leave this at that -- I'm aware that this is a huge issue,
and you'll find language designs more slanted to the one or to
the other direction, reflecting the one or the other culture.

Even the term "API", "application programming interface" conveys
the culture on the one side: here be the system programmers (better
paid, presumably), there be the application programmers. If one of
the latter *dares* to touch system things, (s)he's fired [1].

I wonder where the term "API" originated. I'd guess it comes from
the depth of the bowels of a 1960's to 1970's IBM, where you'd
have to apply for a variable name to a specialized department [2]. 

Cheers

[1] That was a bit tongue-in-cheek, and there *is* IMO a valid
   case for "separation of concerns", I do it all the time with
   myself. I just wanted to raise the contrast a bit.

[2] Again, lots of tongue-in-cheek and a barrel of salt apply.
   That thing with a variable name is from memory and might
   be grossly misremembered.

-- t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: master 1e3b0f2: Improve doc strings of project.el

[Eli Zaretskii]

On July 13, 2020 10:52:57 AM GMT+03:00, "Gregory Heytings via Emacs development discussions." <emacs-devel@gnu.org> wrote:
> 
> > IOW, doing that does NOT make these details part of the public API
> which 
> > we promise shall remain stable.
> >
> 
> I might be wrong, but in that case I think Dmitry's viewpoint is that
> this 
> information should not be placed in the doc strings, but in the
> comments 
> in the source file.

Maybe so, but the suggestion to move what I added to the doc strings into comments was never voiced by Dmitry until the very last message, and even there he is still firmly against documenting the form of the return value, no matter where.

Re: master 1e3b0f2: Improve doc strings of project.el

[Dmitry Gutov]

On 13.07.2020 05:51, Richard Stallman wrote:
>    > > Yes, it is. Apparently you don't understand how cl-generic works.
> 
>    > Please drop the attitude.
> 
> +1.
> 
> Dmitry, it looks like you believe Eli was mistaken in what he said
> about the return value of a certain function.

Really? You decided to admonish me about that phrase, but say nothing 
about Eli's actions that created the tension?

 > What would be a kind way?  "With all due respect, I think you are
 > mistaken about how people use the return value of whatever-it-is."

Yes, it's a good option. But kind communications cannot work if they are 
accompanied by unkind actions.

Re: master 1e3b0f2: Improve doc strings of project.el

[Eli Zaretskii]

> Cc: philip@warpmail.net, theo@thornhill.no, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Mon, 13 Jul 2020 13:47:00 +0300
> 
> Really? You decided to admonish me about that phrase, but say nothing 
> about Eli's actions that created the tension?

My actions weren't different from yours, and came after them.

Re: master 1e3b0f2: Improve doc strings of project.el

[Dmitry Gutov]

On 13.07.2020 16:00, Eli Zaretskii wrote:
> My actions weren't different from yours, and came after them.

You keep missing the point: when you misuse your authority, I have 
little recourse than to raise a stink.

Good job accusing me of being rude after that.

Re: master 1e3b0f2: Improve doc strings of project.el

[Eli Zaretskii]

> Cc: rms@gnu.org, philip@warpmail.net, theo@thornhill.no, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Mon, 13 Jul 2020 18:24:23 +0300
> 
> On 13.07.2020 16:00, Eli Zaretskii wrote:
> > My actions weren't different from yours, and came after them.
> 
> You keep missing the point: when you misuse your authority, I have 
> little recourse than to raise a stink.

I didn't misuse any authority.  My authority of pushing a commit is
exactly like yours or that of the next guy.

In Emacs development, people push changes they see fit all the time,
mostly silently, regardless of authority.  This is how Emacs
development always worked, whether we like it or not.

Re: master 1e3b0f2: Improve doc strings of project.el

[Dmitry Gutov]

On 13.07.2020 11:53, Eli Zaretskii wrote:
>> I might be wrong, but in that case I think Dmitry's viewpoint is that
>> this
>> information should not be placed in the doc strings, but in the
>> comments
>> in the source file.
> Maybe so, but the suggestion to move what I added to the doc strings into comments was never voiced by Dmitry until the very last message, and even there he is still firmly against documenting the form of the return value, no matter where.

One the contrary, I did say that this kind of information is better 
suited for comments, if we must have it.

For example, you can add it to project-try-vc somewhere. To me it seems 
gratuitous/too obvious from the implementation already, but I don't 
object to it on principle.

Re: master 1e3b0f2: Improve doc strings of project.el

[Dmitry Gutov]

On 13.07.2020 19:30, Eli Zaretskii wrote:
 > I didn't misuse any authority.  My authority of pushing a commit is
 > exactly like yours or that of the next guy.

That doesn't work like that when you revert a change I explicitly made 
and mentioned that it should be this way.

Because what's next? A competition in 'git revert'-s?

And no, we're not on equal merits here. I have authored and spent quite 
some time developing this package and thinking about its goals and 
design. And you are only kibitzing.

Re: master 1e3b0f2: Improve doc strings of project.el

[Eli Zaretskii]

> Cc: rms@gnu.org, philip@warpmail.net, theo@thornhill.no, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Mon, 13 Jul 2020 22:01:48 +0300
> 
> On 13.07.2020 19:30, Eli Zaretskii wrote:
>  > I didn't misuse any authority.  My authority of pushing a commit is
>  > exactly like yours or that of the next guy.
> 
> That doesn't work like that when you revert a change I explicitly made 
> and mentioned that it should be this way.

Only after you reverted mine.

> Because what's next? A competition in 'git revert'-s?

We already had it, and you won.  Congrats.

> And no, we're not on equal merits here. I have authored and spent quite 
> some time developing this package and thinking about its goals and 
> design. And you are only kibitzing.

Sigh.

Re: master 1e3b0f2: Improve doc strings of project.el

[Dmitry Gutov]

On 13.07.2020 07:40, Eli Zaretskii wrote:

> Sadly, you never answered my questions.  I still hope that you do,
> because I don't understand the staunch objections, to the degree that
> causes extremely unkind responses, to improve the doc string by saying
> something that is both simple and correct.

I have tried my best to be polite and accommodating in this thread, 
until you decided that your opinion is more important. And it's not the 
first time we're having this problem.

> Instead of answering my practical questions, which could have led us
> towards some acceptable compromise, you've elected to reiterate your
> views,

s/views/design goals/g

> which just makes another round of the same dispute we've been
> having, without any hope for reconciling the differences.  Because it
> turns out we disagree even more than I thought, on the very principles
> behind this, see below.

This sounds like you're saying that my opinion is invalid. I hope we 
manage to end up somewhere else than shouting "NO U".

> When the form of the return value changes, we will update the
> documentation.  This kind of thing happens all the time in Emacs
> development, and is nothing to be afraid of.

Having private/internal variables and private functions which we choose 
not to document or refer to in docstrings happens just as often.

> Especially if you
> consider some information "private", which means we under no
> obligation to preserve its form.  We even have an "Internals" chapter
> in the ELisp manual (woefully incomplete, but not because we decided
> not to tell those missing details).

If this stuff was really implemented in a way that is hard to make out 
for a would-be contributor (like, for example, the Display Engine is), 
then of course we should add a manual chapter with implementation details.

The information you want to document here, though, can be determined by 
anyone with basic Elisp skills in 10 seconds.

But in principle, yes, we could add a similar kind of "Internals" 
chapter to the manual that would outline how project.el works, and what 
is it for. If we decide that it's important enough.

And still, we wouldn't refer to that chapter from the docstring of 
'project-current'.

> This is why I'm frankly astonished by your opposition to document the
> return value.  Once again: what's the catastrophe in doing that?

Once again: it would encourage the uses that we don't want. And would do 
little to help the uses that we do want.

>> Here are things we want third-party code to be able to do:
>>
>> - Consume the public project APIs, which are currently limited to
>> 'project-current' and several generic functions defined for project
>> instance values, to define new commands or supplement existing ones
>> (e.g. to query the current project for its root or its list of files).
> 
> Where is this documented?  I don't think I see it.

Orthogonal, but fair. I should copy some of these explanations to the 
package's Commentary.

>> - Create new project backends. Note that we don't want *everybody* to do
>> that because creating a well-performing backend is a significant amount
>> of work (unless the user has very particular needs and works with small
>> projects). But we want this to be a solid option, and it would be ideal
>> if, say, a handful of backends become popular someday. To create a
>> backend, one needs to choose the format of its instance value (which
>> shouldn't match any of the existing formats), write its
>> project-find-functions element, and define the project-root override
>> that dispatches on its instance value. Preferably, also implement
>> overrides for project-ignores and project-files.
> 
> Where is it documented what needs to be done for creating a new
> backend?  Again, I don't think I see it documented.

Ditto.

>> Neither of these options requires knowing the exact format of
>> project-try-vc's return value, if only to stay away from it. But the
>> odds of a collision seem very low (and there are ways to reduce them
>> even further).
> 
> These are just some of the possible development directions.  There are
> others, no less legitimate ones.  In particular, minor improvements to
> the code, which don't change the overall design and implementation.
> These and others do require to be aware of the form of the return
> value, or at least they can be helped by knowing that.

They shouldn't be documented, however, in the docstring of the main 
entry point to the Project API.

> Moreover, I'm quite sure that if you try to document the missing
> pieces mentioned above in a way that is useful for potential
> contributors, you will find soon enough that the policy of hiding
> useful information about interfaces, even "internal" ones, works
> against you: it makes it much harder to say something useful about how
> the various objects and methods interact or should interact.  You will
> eventually be forced to say "read the code" all the time.  Which means
> the documentation cannot do one of its important jobs: help to
> understand the code.

But I'm not having this problem.

What remains problematic, on the other hand, is referring to "project 
instances" in general, because it's a novel notion for Emacs Lisp API 
(having arbitrary values), and it doesn't invoke the same mental image 
that our more common terms do.

> But more generally, it is not our prerogative to decide for others in
> what directions they may or may not take the development in the
> future.  This is not our pet project, this is a community project, and
> any member of the community is welcome to propose changes to any part
> of Emacs.  If the change is useful for the users, and is done cleanly,
> we welcome and accept such changes -- this is how Emacs was and is
> being developed for 40 years.  It is what brought us where we are
> today.

That's a fairly general response to my fairly specific points.

> So I very much object to the policy you describe above, which is that
> you personally decide up front what kinds of changes are and aren't
> legitimate,

I challenge your authority to counter my design decisions. They weren't 
done in a day.

> based on some principles that are not necessarily shared
> by others (certainly aren't documented anywhere in the Emacs
> contribution-related docs), and then unilaterally implement this rigid
> policy, right down to the level of the smallest detail in the doc
> strings.  This is an extremely radical and harsh policy, which I don't
> think ever existed in Emacs, nor do I think it should exist.  If
> accepted, it will IMO harm the Emacs development in the long run.

We've had private variables and functions in Emacs for years. Probably 
more than a decade, I'm not sure.

>> Here are, on the other hand, things that people generally shouldn't do,
>> and yet I have seen them done:
>>
>> - Extract data from the project instance value directly, instead of
>> going through e.g. 'project-root'. For instance, writing (cdr
>> (project-current t)) instead of (project-root (project-current t)). The
>> former is bad code.
>>
>> - Write a new project-find-functions element which returns an instance
>> value that belongs to a built-in backend (usually, the 'transient' one).
>> AFAICT, this is usually done by cargo culting, without understanding of
>> the consequences.
> 
> See above: I disagree very much with saying flatly that we don't want
> these.  These types of changes are not "verboten" in principle, they
> should be each one of them judged on its own right, and if done
> cleanly and efficiently, they are not in any way "wrong".

I invite you to read that again, first item in particular.

If you don't understand how it is a problem, you don't understand how 
this API works.

> In a Free Software project, you cannot forbid people from trying
> different development directions, even dangerous ones, you can only
> warn them about the dangers by documenting them.  Which is one more
> reason for having good, detailed documentation.  Refraining from
> mentioning something because you think it is dangerous is the wrong
> way, because people are curious and will find what you are trying to
> hide.  When they do, no amount of "I didn't want you to know that, so
> don't do that" will help, believe me.

I am physically unable to forbid them, and I'm not trying to.

But writing good documentation means mentioning relevant information, as 
well as avoiding littering the docs with irrelevant details.

>> I don't want to prohibit the last option outright, but we shouldn't
>> leave any impression that it's legitimate either. And most problems that
>> people try to solve this way (predominantly: customizing project root
>> markers, AFAIK) should be filed as bug reports/feature requests.
> 
> As I already said several times, the right way of doing this without
> throwing the baby with the bathwater is to document the value and then
> tell this form is subject to change without notice.  It is a very
> simple solution that keeps both goals in our sights without losing any
> of them.

If this results in client code that only functions when the current 
project backend is project-vc (and only the current version), but fails 
with all others, would you consider this a success?

>> but it doesn't give you any guarantee: after all, we need to make
>> sure project-switch-to-buffer's behavior makes sense for different
>> backends, even ones we didn't write.
> 
> Irrelevant: the information I added was about a single _existing_
> backend.  If you have proposals how to say something sensible and
> helpful about others, I'm all ears.

I have put some thorough descriptions in the grandparent email.

>> Instead, it seems we have to document that said value should uniquely
>> identify the returned project and be "stable": not change over time or
>> when fetched from different buffers belonging to the same project. That
>> also means that the value shouldn't contain any caches inside of it.
>>
>> And once we add these requirements, the answer to "which projects are
>> equal" translates to:
>>
>> - The backend is the same,
>> - The instances describe the same project, but the exact fashion depends
>> on the backend. As it would be anyway.
> 
> Of course, none of this is described anywhere in project.el.  It isn't
> even clear what objects are part of the design, what are the
> requirements from each object, how to establish of some arbitrary Lisp
> object is "an instance of a project",

To draw an analogy to a long-established feature:

How do you determine that an arbitrary function is a "completion table"?

>> I'm not sure where you could put this particular bit of
>> information, where it would both be relevant and yet not obvious from
>> the nearby code, though.
> 
> _Any_ place is better than _no_ place.  "Not sure where to put" is not
> the same as "let's not document this".

Right. That's why I chose these particular words. But let's not put that 
in the project-current's docstring.

Re: master 1e3b0f2: Improve doc strings of project.el

[Dmitry Gutov]

On 13.07.2020 22:41, Eli Zaretskii wrote:
>>   > I didn't misuse any authority.  My authority of pushing a commit is
>>   > exactly like yours or that of the next guy.
>>
>> That doesn't work like that when you revert a change I explicitly made
>> and mentioned that it should be this way.
> Only after you reverted mine.

Yes, I reverted a part of your commit and explained why. And said "thank 
you" for the rest of it.

Re: master 1e3b0f2: Improve doc strings of project.el

[Eli Zaretskii]

> Cc: philip@warpmail.net, theo@thornhill.no, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Mon, 13 Jul 2020 22:51:04 +0300
> 
> I have tried my best to be polite and accommodating in this thread, 

Actually, you've never been accommodating, as far as I'm concerned,
and lately stopped even trying to be polite.  Completely uncalled-for,
since I didn't do or say anything that a project maintainer isn't
supposed to say or do.

> I challenge your authority

Challenge away, if you must.  It won't change anything in how I'm
doing my job as the Emacs maintainer.  Because I have all the
intentions of doing that job the best I possibly can.  And that
requires me to be involved in any aspect of the Emacs development and
maintenance that I consider important as part of this job.  This
includes, but isn't limited to, the overall quality of our
documentation, down to individual doc strings when needed.  It also
includes any design and implementation aspects of any Emacs package or
C code that I decide is important for Emacs's stability, or its
future, or its being a project open to extensions, or for any other
relevant aspect.  These and other aspects are explicitly part of my
job and of my obligations towards the GNU Project, which named me the
maintainer.

So no matter what you think about my authority or lack thereof, I'm
doing my job here, and will continue doing it, as I understand it.
Your "challenges" will not stop me.  They will potentially cause a lot
of friction and unpleasant exchanges, but that is something I can do
nothing about, not without giving up what I consider part of my job.

The rest of your message simply reiterates what's been said already,
so I see no reason to respond to your other points one by one, as I
already replied to them or their ilk in the past.

P.S. I still hope you will change your mind, so that we could
cooperate in a much nicer manner.

Re: master 1e3b0f2: Improve doc strings of project.el

[Dmitry Gutov]

On 16.07.2020 19:34, Eli Zaretskii wrote:
>> Cc: philip@warpmail.net, theo@thornhill.no, emacs-devel@gnu.org
>> From: Dmitry Gutov <dgutov@yandex.ru>
>> Date: Mon, 13 Jul 2020 22:51:04 +0300
>>
>> I have tried my best to be polite and accommodating in this thread,
> 
> Actually, you've never been accommodating, as far as I'm concerned,
> and lately stopped even trying to be polite.

So trying to solve your problems and incorporate your ideas don't count 
as accommodating? I'll try to remember that.

And I accepted all of your edits and additions, right up until the point 
where they started to conflict with the design.

> Completely uncalled-for,
> since I didn't do or say anything that a project maintainer isn't
> supposed to say or do.

As a maintainer of an umbrella project, you're supposed to respect the 
contributors' efforts, and especially the maintainers of subsystems who 
have more understanding about the design and purpose of said subsystems.

But once you get an idea in your mind, you very rarely back down, even 
when confronted with your prior lack of knowledge of how things work, or 
how they are designed to be. Even after I explain those, and spend a lot 
of time doing that. This has happened on multiple occasions.

In this particular situation, I would have been happy to have your help. 
You are a good, thoughtful and diligent documentation writer (*).

But that stops being a benefit once you decide that things must be 
explained exactly how you managed to understand them, and no other way 
can be possibly adequate.

(*) I could have used more assistance in other aspects too, but so far 
the main technical hurdles have all been up to me.

> Challenge away, if you must.  It won't change anything in how I'm
> doing my job as the Emacs maintainer.  Because I have all the
> intentions of doing that job the best I possibly can.  <...>

That's a lot of words to say "things must be exactly as I say for the 
good of the project". The implication being that your experience is 
always best, or that you were chosen as the maintainer for being an 
expert on all aspects of Emacs (spoiler alert: that was not the reason).

You should consider doing something other than "the best you possibly 
can". Otherwise you stifle the alternative views here. And, more 
importantly, the approaches that fall outside of your experience.

Also recall this message: 
https://lists.gnu.org/archive/html/emacs-devel/2020-05/msg01711.html. 
Those weren't my words.

> The rest of your message simply reiterates what's been said already,
> so I see no reason to respond to your other points one by one, as I
> already replied to them or their ilk in the past.

If you think this paragraph sounds nice or polite, it probably does. But 
its meaning is anything but.

I'm disappointed to see that all this time spent on explanations has 
been in vain. You even skipped an essential question (or two) that I asked.

Re: master 1e3b0f2: Improve doc strings of project.el

[Dmitry Gutov]

On 16.07.2020 19:34, Eli Zaretskii wrote:
> P.S. I still hope you will change your mind, so that we could
> cooperate in a much nicer manner.

That doesn't work when you reject the possibility that you don't 
understand something, as well as all my attempts to help you learn.

Re: master 1e3b0f2: Improve doc strings of project.el

[Eli Zaretskii]

> From: Dmitry Gutov <dgutov@yandex.ru>
> Cc: emacs-devel@gnu.org
> Date: Thu, 16 Jul 2020 21:55:48 +0300
> 
> As a maintainer of an umbrella project, you're supposed to respect the 
> contributors' efforts, and especially the maintainers of subsystems who 
> have more understanding about the design and purpose of said subsystems.

I do.  But sometimes I'm not convinced, the disagreement cannot be
resolved, and there's no compromise anywhere in sight; in those rare
cases I insist on my opinion.  That is entirely my prerogative as the
maintainer, there's nothing new here.

> That's a lot of words to say "things must be exactly as I say for the 
> good of the project". The implication being that your experience is 
> always best, or that you were chosen as the maintainer for being an 
> expert on all aspects of Emacs (spoiler alert: that was not the reason).

You are putting words in my mouth that I didn't say.  Please don't.

Re: master 1e3b0f2: Improve doc strings of project.el

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Thu, 16 Jul 2020 21:56:50 +0300
> 
> On 16.07.2020 19:34, Eli Zaretskii wrote:
> > P.S. I still hope you will change your mind, so that we could
> > cooperate in a much nicer manner.
> 
> That doesn't work

It works with everyone else.

Re: master 1e3b0f2: Improve doc strings of project.el

[Dmitry Gutov]

On 16.07.2020 22:59, Eli Zaretskii wrote:
>>> P.S. I still hope you will change your mind, so that we could
>>> cooperate in a much nicer manner.
>> That doesn't work
> It works with everyone else.

Who hasn't left yet.

Or simply started contributing less and less.

Re: master 1e3b0f2: Improve doc strings of project.el

[Dmitry Gutov]

On 16.07.2020 22:57, Eli Zaretskii wrote:
>> From: Dmitry Gutov <dgutov@yandex.ru>
>> Cc: emacs-devel@gnu.org
>> Date: Thu, 16 Jul 2020 21:55:48 +0300
>>
>> As a maintainer of an umbrella project, you're supposed to respect the
>> contributors' efforts, and especially the maintainers of subsystems who
>> have more understanding about the design and purpose of said subsystems.
> 
> I do.  But sometimes I'm not convinced, the disagreement cannot be
> resolved, and there's no compromise anywhere in sight; in those rare
> cases I insist on my opinion.

I have made plenty of compromises, some in this thread alone. You never 
seem to appreciate them.

> That is entirely my prerogative as the
> maintainer, there's nothing new here.

And it is my obligation to care about the code I have in my purview to 
the best of my ability. Especially for the packages I created.

So you might have a chance to enjoy kicking me out of the project and 
availing yourself to the spoils.

>> That's a lot of words to say "things must be exactly as I say for the
>> good of the project". The implication being that your experience is
>> always best, or that you were chosen as the maintainer for being an
>> expert on all aspects of Emacs (spoiler alert: that was not the reason).
> 
> You are putting words in my mouth that I didn't say.  Please don't.

You just rephrased my message in your previous reply. In about as much 
favorable light.

Re: master 1e3b0f2: Improve doc strings of project.el

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

Emacs is not an "umbrella project" in the sense that you mean.  (That
is not the way the GNU Project works.)  It is a single project that is
meant to be coherent as a whole, and the Emacs maintainers are in
charge of the whole thing.

There are parts of Emacs that are developed and maintained by specific
people or groups.  Some of them take the form of "packages" while
others are simply areas of expertise.  The Emacs maintainers mostly
leave those areas to the people who work on them, to divide up the
task.

But that does not mean that the people or groups that work on parts
are independent in principle.

See https://gnu.org/gnu/gnu-structure.html for the structure
and administration of the GNU Project.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: master 1e3b0f2: Improve doc strings of project.el

[Dmitry Gutov]

On 17.07.2020 04:07, Richard Stallman wrote:
> Emacs is not an "umbrella project" in the sense that you mean.  (That
> is not the way the GNU Project works.)  It is a single project that is
> meant to be coherent as a whole, and the Emacs maintainers are in
> charge of the whole thing.

So, no provisions for respecting the author of the code, or deferring to 
the technical opinion of the maintainer of a particular part of Emacs? 
Even for non-critical issues?

One would think a project as old GNU would have considered non-technical 
aspects like that. If only to avoid ostracizing contributors.

 > There are parts of Emacs that are developed and maintained by specific
people or groups.  Some of them take the form of "packages" while
others are simply areas of expertise.  The Emacs maintainers mostly
leave those areas to the people who work on them, to divide up the
task.

This makes is sound like the bulk of the work is performed by people 
tasked with the said work by the maintainers. Which is very often not 
the case.

Re: master 1e3b0f2: Improve doc strings of project.el

[Dmitry Gutov]

On 13.07.2020 10:58, tomas@tuxteam.de wrote:
> Even the term "API", "application programming interface" conveys
> the culture on the one side: here be the system programmers (better
> paid, presumably), there be the application programmers. If one of
> the latter*dares*  to touch system things, (s)he's fired [1].

This is is pretty dated view: the "system things" are rarely in the 
picture. But they are as fine to touch as any, as long as that fits the 
abstraction you are defining.

The difference between system programmers and application programmers, I 
think, is the latter have found that abstractions are a good thing for a 
lot of domains, and have come up with certain rules for using them.

I can certainly understand how a system programmer might dislike having 
to deal with extra abstractions, but, again, certain jobs simply call 
for using them.

Re: master 1e3b0f2: Improve doc strings of project.el

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1723 bytes --]

On Fri, Jul 17, 2020 at 01:27:11PM +0300, Dmitry Gutov wrote:
> On 13.07.2020 10:58, tomas@tuxteam.de wrote:
> >Even the term "API", "application programming interface" conveys
> >the culture on the one side: here be the system programmers (better
> >paid, presumably), there be the application programmers. If one of
> >the latter*dares*  to touch system things, (s)he's fired [1].
> 
> This is is pretty dated view: the "system things" are rarely in the
> picture. But they are as fine to touch as any, as long as that fits
> the abstraction you are defining.

I was rather trying to highlight the contrast between (a) the library
designer(s) set the interface design and (b) the interface evolves
as a collective effort of designers and users. Reality will be a mix
of both, of course. The term API conveys a hierarchy -- clearly in
camp (a).

The "system programmer" thing comes from former times, yes.

> The difference between system programmers and application
> programmers, I think, is the latter have found that abstractions are
> a good thing for a lot of domains, and have come up with certain
> rules for using them.
> 
> I can certainly understand how a system programmer might dislike
> having to deal with extra abstractions, but, again, certain jobs
> simply call for using them.

I think we misunderstood each other. Abstractions are our daily
bread, of course. But the process of choosing which abstractions
are relevant and whether and where to punch holes in them is
open to debate.

Here we seem to have a conflict between your and Eli's views. I
sincerely hope you both don't take that to the personal level
and try to work out that conflict.

Cheers
-- tomás

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: master 1e3b0f2: Improve doc strings of project.el

[Dmitry Gutov]

On 17.07.2020 18:27, tomas@tuxteam.de wrote:
> I was rather trying to highlight the contrast between (a) the library
> designer(s) set the interface design and (b) the interface evolves
> as a collective effort of designers and users.

An interface's ability to evolve over time isn't entirely predicated on 
there being no limitations on its use. It can easily change as a result 
of user feedback anyway.

> Reality will be a mix
> of both, of course. The term API conveys a hierarchy -- clearly in
> camp (a).
> 
> The "system programmer" thing comes from former times, yes.

BTW, lower level programming is not devoid of abstractions as well. I 
could be wrong in some details here, but a file descriptor seems like a 
prime example of an abstraction.

The docs say it's represented by a number, but it's not a "real" number 
(you never do any arithmetic with it), and an fd can be backed by very 
different mediums: a file on disk, a network socket, a pipe, etc. And 
everything is a number in C anyway. It's an "opaque" value.

When using it in a piece of code, you don't always have to know what 
kind of file it is. Or you just know it's a socket, for instance. But 
you know there are certain things one can do with a file descriptor, 
like passing it to 'write' or 'read'.

Re: master 1e3b0f2: Improve doc strings of project.el

[tomas]

[-- Attachment #1: Type: text/plain, Size: 3193 bytes --]

On Sat, Jul 18, 2020 at 03:55:33AM +0300, Dmitry Gutov wrote:
> On 17.07.2020 18:27, tomas@tuxteam.de wrote:
> >I was rather trying to highlight the contrast between (a) the library
> >designer(s) set the interface design and (b) the interface evolves
> >as a collective effort of designers and users.
> 
> An interface's ability to evolve over time isn't entirely predicated
> on there being no limitations on its use. It can easily change as a
> result of user feedback anyway.

Definitely. But it's not an abrupt transition.

> >Reality will be a mix
> >of both, of course. The term API conveys a hierarchy -- clearly in
> >camp (a).
> >
> >The "system programmer" thing comes from former times, yes.
> 
> BTW, lower level programming is not devoid of abstractions as well.

Absolutely. We are in violent agreement. It's "turtles all the
way down [1]".

> I could be wrong in some details here, but a file descriptor seems
> like a prime example of an abstraction.

Well, that's at the Mother of Abstraction, aka the user space/kernel
space barrier :-)

This one is especially interesting because here you have different
applications which dont quite trust each other. You will see that
(sometimes ugly) pattern repeat whenever you have a similar situation
(web browser, I'm looking at you).

> The docs say it's represented by a number, but it's not a "real"
> number (you never do any arithmetic with it), and an fd can be
> backed by very different mediums: a file on disk, a network socket,
> a pipe, etc. And everything is a number in C anyway. It's an
> "opaque" value.

Yes, this is some kind of OO. But it leaks a couple of things:
first, it's a "small number" (i.e. not some random 64 bit
address, but the pattern of counting up from zero and of building
bitmaps of FDs is considered --mostly-- viable), i.e. the mental
model around is that the kernel "has" a table somewhere indexed
by FD. This detail wasn't probably intended to leak in the first
place, but was just "the obvious thing to do" in a world in which
machines with a 32 bit address space were considered serious iron.

So it somehow creatively leaked. Cf. the select() system call
for a (mis?)use of this leak in a very creative way which gave
us fun for at least 15 to 20 years. These days it's on its way
out. Times change.

> When using it in a piece of code, you don't always have to know what
> kind of file it is. Or you just know it's a socket, for instance.
> But you know there are certain things one can do with a file
> descriptor, like passing it to 'write' or 'read'.

Yes, definitely. Barring errors in the interface design (read()
returning zero bytes is "we're done" in the case of a file and
"currently nothing there, but do come back and retry later" for
a socket :-)

My point in all of that is that (some) leak in the abstractions
might lubricate change.

Writing in the docs "here be dragons, we might well change that
implementation from under you [2]" is fair, and IMO better than
"warranty void if opened" ;-)

Cheers

[1] https://xkcd.com/1416/
[2] ...but by all means, if you enjoy dragons, go ahead :)

-- tomás

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: master 1e3b0f2: Improve doc strings of project.el

[Eli Zaretskii]

> Date: Sat, 18 Jul 2020 11:05:46 +0200
> From: tomas@tuxteam.de
> Cc: emacs-devel@gnu.org
> 
> > I could be wrong in some details here, but a file descriptor seems
> > like a prime example of an abstraction.
> 
> Well, that's at the Mother of Abstraction, aka the user space/kernel
> space barrier :-)

IMO, there's nothing opaque about a file descriptor.  It is just a
number.  The abstraction here isn't in the descriptor itself, it's in
what it _represents_.  The descriptor is actually a _handle_ for an
object that itself is not exposed at all; you can only manipulate it
via the handle.

> Yes, this is some kind of OO. But it leaks a couple of things:
> first, it's a "small number" (i.e. not some random 64 bit
> address, but the pattern of counting up from zero and of building
> bitmaps of FDs is considered --mostly-- viable), i.e. the mental
> model around is that the kernel "has" a table somewhere indexed
> by FD. This detail wasn't probably intended to leak in the first
> place, but was just "the obvious thing to do" in a world in which
> machines with a 32 bit address space were considered serious iron.

That detail is not a general trait of a handle.  Another popular
example of a handle is a FILE pointer in C, where the details of the
object it points to are generally not public.  Here, the numerical
value of the handle is no longer small.

Yet another similar abstraction, much more close to our domain, is the
Lisp_Object representation on the C level: with a couple of
exceptions, each object is actually a handle of a C struct; once
again, the value is not necessarily small, especially if the object
tags use the MSBs.  Sometimes, very rarely, these handles are even
exposed to Lisp; one example is file-notify--add-watch-* functions.

So I think the handle notion and the related design patterns are only
tangentially relevant to the issue at hand.

P.S. The interpretation of a file descriptor as an index into some
small table doesn't hold in modern OSes anyway.  It was true for
MS-DOS, but modern OSes have moved away of that.

P.P.S. And no, not everything in C is a number, unless one also holds
that "everything in computing is a number".

Re: master 1e3b0f2: Improve doc strings of project.el

[Dmitry Gutov]

On 18.07.2020 13:10, Eli Zaretskii wrote:
>> Date: Sat, 18 Jul 2020 11:05:46 +0200
>> From: tomas@tuxteam.de
>> Cc: emacs-devel@gnu.org
>>
>>> I could be wrong in some details here, but a file descriptor seems
>>> like a prime example of an abstraction.
>>
>> Well, that's at the Mother of Abstraction, aka the user space/kernel
>> space barrier :-)
> 
> IMO, there's nothing opaque about a file descriptor.  It is just a
> number.  The abstraction here isn't in the descriptor itself, it's in
> what it _represents_.

Potato, pot-ah-to.

> The descriptor is actually a _handle_ for an
> object that itself is not exposed at all; you can only manipulate it
> via the handle.

So it's actually even more opaque.

>> Yes, this is some kind of OO. But it leaks a couple of things:
>> first, it's a "small number" (i.e. not some random 64 bit
>> address, but the pattern of counting up from zero and of building
>> bitmaps of FDs is considered --mostly-- viable), i.e. the mental
>> model around is that the kernel "has" a table somewhere indexed
>> by FD. This detail wasn't probably intended to leak in the first
>> place, but was just "the obvious thing to do" in a world in which
>> machines with a 32 bit address space were considered serious iron.
> 
> That detail is not a general trait of a handle.  Another popular
> example of a handle is a FILE pointer in C, where the details of the
> object it points to are generally not public.  Here, the numerical
> value of the handle is no longer small.

A FILE pointer is less interesting, given that the structure on the 
other end is usually more or less the same. Even if it's concealed.

> Yet another similar abstraction, much more close to our domain, is the
> Lisp_Object representation on the C level: with a couple of
> exceptions, each object is actually a handle of a C struct;

Exactly.

"There's nothing opaque about a project instance.  It is just a 
Lisp_Object."

> P.S. The interpretation of a file descriptor as an index into some
> small table doesn't hold in modern OSes anyway.  It was true for
> MS-DOS, but modern OSes have moved away of that.

Meaning that it's an abstraction that stood the test of time.

Re: master 1e3b0f2: Improve doc strings of project.el

[Dmitry Gutov]

On 18.07.2020 12:05, tomas@tuxteam.de wrote:

>> The docs say it's represented by a number, but it's not a "real"
>> number (you never do any arithmetic with it), and an fd can be
>> backed by very different mediums: a file on disk, a network socket,
>> a pipe, etc. And everything is a number in C anyway. It's an
>> "opaque" value.
> 
> Yes, this is some kind of OO. But it leaks a couple of things:
> first, it's a "small number" (i.e. not some random 64 bit
> address, but the pattern of counting up from zero and of building
> bitmaps of FDs is considered --mostly-- viable), i.e. the mental
> model around is that the kernel "has" a table somewhere indexed
> by FD.

So it leaks a little.

> This detail wasn't probably intended to leak in the first
> place, but was just "the obvious thing to do" in a world in which
> machines with a 32 bit address space were considered serious iron.

I think, at best, you are getting a sense that it is a pointer. But one 
you can't easily dereference.

And there can be just about any data structure behind that pointer.

It's not an ironclad abstraction (fds can have various kinds of 
properties, as well as slightly-incompatible behaviour, as you have 
pointed out). Still, it doesn't invite the caller to use whaterver data 
structure is behind it directly.

>> When using it in a piece of code, you don't always have to know what
>> kind of file it is. Or you just know it's a socket, for instance.
>> But you know there are certain things one can do with a file
>> descriptor, like passing it to 'write' or 'read'.
> 
> Yes, definitely. Barring errors in the interface design (read()
> returning zero bytes is "we're done" in the case of a file and
> "currently nothing there, but do come back and retry later" for
> a socket :-)

Right.

> My point in all of that is that (some) leak in the abstractions
> might lubricate change.

It might, I suppose. But Elisp is pretty transparent, so it's not 
possible to create a similar barrier anyway.

> Writing in the docs "here be dragons, we might well change that
> implementation from under you [2]" is fair, and IMO better than
> "warranty void if opened" ;-)

At least the doc should state the information that would help the client 
use the interface correctly first.

Re: master 1e3b0f2: Improve doc strings of project.el

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

A Unix file descriptor is a datatype which is semantically opaque and
has a representation which is literally opaque.  There is nothing
inside it that a user program can make sense of.  Once the developer
of the user program knows this much, there is nothing more to say
about it.

The package data structure is a very different case.  It is made up of
Lisp data structures which any Lisp program, and any Emacs user, could
look inside.

I gather that package.el provides a complete API of functions to do
everything anyone would wish, with this data structure.  If so, people
and programs invoking package.el don't need to know how that data
structure is made up.  There is no need to describe it in doc strings.

But people working on package.el in the future, and people trying to
understand it, do need to know its structure.  So it should have
comments which explain.  (I expect Linux has comments describing the
meaning of file descriptor data structures.)

We shouldn't wait to document that data structure, because any one of
us (and that includes you) might, at any moment, suddenly be unable to
continue working on Emacs.  With luck, you won't have to stop this
year -- but it will surely happen some day.

So please help people document this in comments.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: master 1e3b0f2: Improve doc strings of project.el

[Dmitry Gutov]

On 19.07.2020 05:27, Richard Stallman wrote:
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
> 
> A Unix file descriptor is a datatype which is semantically opaque and
> has a representation which is literally opaque.  There is nothing
> inside it that a user program can make sense of.  Once the developer
> of the user program knows this much, there is nothing more to say
> about it.

But there are reasons behind this state of affairs. It's a man-made 
design, not a force of nature.

My point is, having semantically opaque datatypes can be good for us, in 
certain applications.

> The package data structure is a very different case.  It is made up of
> Lisp data structures which any Lisp program, and any Emacs user, could
> look inside.
> 
> I gather that package.el provides a complete API of functions to do
> everything anyone would wish, with this data structure.  If so, people
> and programs invoking package.el don't need to know how that data
> structure is made up.  There is no need to describe it in doc strings.

This discussion is about project.el, not package.el.

But I think we agree on the general point: as long as people and 
programs provide an adequate API (which is our goal anyway), we don't 
need to describe in detail how its data structures (plural, in this 
case) are made up.

> But people working on package.el in the future, and people trying to
> understand it, do need to know its structure.  So it should have
> comments which explain.  (I expect Linux has comments describing the
> meaning of file descriptor data structures.)

I have nothing against comments, but in this case the data structures 
are pretty trivial and obvious from the functions that create them.

So I don't immediately see anything in there that would need 
clarification in comments, but would not stop people from doing so.

> We shouldn't wait to document that data structure, because any one of
> us (and that includes you) might, at any moment, suddenly be unable to
> continue working on Emacs.  With luck, you won't have to stop this
> year -- but it will surely happen some day.

Indeed.

Re: master 1e3b0f2: Improve doc strings of project.el

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > This discussion is about project.el, not package.el.

Sorry.  I know nothing about project.el.  I'm only suggesting
its data structures should be described by comments.

  > I have nothing against comments, but in this case the data structures 
  > are pretty trivial and obvious from the functions that create them.

It is important to document the meaning of each slot -- that may not
be obvious.  Which values are valid?

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)