RE: Opaque objects and Emacs documentation

RE: Opaque objects and Emacs documentation

[Drew Adams]

FWIW, since the question was asked -

I'm in favor of documenting pretty much all
Emacs-Lisp functions and variables _all the way
down_.

(Even lambda expressions sometimes deserve a doc
string.  And yes, a comment can also serve, but
a doc string is generally better for describing
what something is or does.  A comment is generally
good for describing design decisions or unobvious
code considerations.)

And this is so whether or not someone considers
this or that particular thing to be "internal",
whether just for now or for all time.

If you write code for yourself, any designation
of "internal" is just a note to yourself that
either (a) the code in question is asking to be
revisited and possibly changed later, perhaps
including its interfaces with other code, or (b)
the code in question is tricky or fragile, so
fiddling with it can be asking for trouble.

With free software, all code we write is code
for ourselves, because ourselves includes
everyone, now or later.  This means that all
labeling of "internal" is only a comment saying
(a) or (b), above.

Such labeling is _never_ a reason, with free
software, _not_ to describe something.  We're
not (should not be) trying to hide anything
from anyone, including ourselves.

There's too much labeling of stuff in Emacs as
"internal", IMO, and this has increased over
time.  For the most part it's not necessary or
useful.

Too often, I think, such branding as "internal"
is really a cop-out from someone who isn't
interested in the doc/help, or doesn't want to
take the time to describe things (e.g. in
English), or has trouble doing so, or just has
a mental "block" about doing so (similar to
math phobia), or even is unsure of the thing
to be described and wants to save the possible
embarrassment of offering an off-the-mark
description.  Such "feeling" motivations do
happen sometimes.

Such feelings can be understandable, but they
are never a reason why the things in question
_shouldn't_ be described.  They can be reasons
why the person having such feelings shouldn't
need to be the one to document those things.
They're just not a reason why things shouldn't
be documented at all.

The case of generated functions and variables
is a special one.  Generic functions are one
example of creating things programmatically.
`cl-defstruct' is another example.  Even things
like `define-minor-mode' are examples.

In these cases we need to (should, IMO) find
ways to:

1. Programmatically add _some_ useful doc, when
   feasible.  We've done that for things like
   `define-minor-mode', but there's still room
   for improvement.

2. Provide ways for the programmatically added
   doc to be manually tweaked (supplemented,
   replaced).

3. Actually _do_ such manual tweaking.  And
   to the extent that we haven't provided
   good solutions for #1 and #2, just provide
   the doc manually, whenever possible.

Lisp is not your average language.  And free
software doesn't have the same constraints
and purposes as proprietary software.  _Some_
of the typical encapsulation and rendering of
things as opaque/internal is motivated by a
spirit of protection of property/control that
doesn't apply to free software.

The point is not to not have abstractions or
use macros or higher-order functions.  It's
not to _force_ someone to dive deep for
understanding.  The point is to provide help
all the way up and down.  There's no good
reason _not_ to do that, IMO.

"Don't look! - INTERNAL" is, in particular,
not a good reason.  It's not a reason at all,
except perhaps in the context of protecting
private property, which we're not in the
business of doing, here.  Opaqueness should
be anathema to free software.

Just one opinion.  And thanks for posing the
question, Eli.

Re: Opaque objects and Emacs documentation

[Dmitry Gutov]

On 17.07.2020 18:41, Drew Adams wrote:
> "Don't look! - INTERNAL" is, in particular,
> not a good reason.  It's not a reason at all,
> except perhaps in the context of protecting
> private property, which we're not in the
> business of doing, here.  Opaqueness should
> be anathema to free software.

"You may look, but try not to touch" would perhaps be a better analogy.

Lisp indeed is a special language, and any concerns that my choice would 
actually be _hiding_ something from the users is a dramatic 
overstatement. Both the code and the returned values are trivial to examine.

RE: Opaque objects and Emacs documentation

[Drew Adams]

> > "Don't look! - INTERNAL" is, in particular,
> > not a good reason.  It's not a reason at all,
> > except perhaps in the context of protecting
> > private property, which we're not in the
> > business of doing, here.  Opaqueness should
> > be anathema to free software.
> 
> "You may look, but try not to touch" would perhaps be a better analogy.

Fine.  And a comment as to why is a plus.

> Lisp indeed is a special language, and any concerns that my choice
> would actually be _hiding_ something from the users is a dramatic
> overstatement. Both the code and the returned values are trivial to
> examine.

My post didn't speak about your choice for anything.
It's not about you.  I replied to Eli's request for
input on the general question.  An interesting and
useful question, IMO.

A general comment on your reply, here, is that the
trivialness of code and return values can be in the
eye of the reader.

And accompanying description rarely _hurts_.

It hurts when it's _wrong_ or misleading, and in
that case, especially if written by the code author,
such inaccuracy can be a sign that something isn't
understood as well as it should be.

That's another purpose doc/description serves: it
can be a check on our understanding.  This is one
of the reasons for careful _specifying_, whether
for requirements, design, tests, whatever.

Re: master 0339325: ; * lisp/progmodes/project.el (project-current): Doc fix.

[Dmitry Gutov]

On 12.07.2020 21:49, Eli Zaretskii wrote:
>   The \"transient\" project instance is a special kind of value
>   which denotes a project rooted in that directory and includes all
> -files under it except for ones that match standard ignores.
> +the files under the directory except for those that should be
> +ignored (per `project-ignores').

The more accurate way to say this would be

   except for those that are ignored
   in the default 'project-ignores' implementation.

But this is basically a tautology. If we wanted to describe transient's 
particular behavior we'd have to describe what it actually ignores (and 
it honors vc-directory-exclusion-list as well as 
grep-find-ignored-files). But that pretty clunky for one docstring. So 
perhaps that description should move somewhere. Into the Commentary, 
maybe. Or directly to the cl-defgeneric's docstring, to which 
project-current's docstring would after all refer to as "the default 
`project-ignores' implementation".

Re: master 0339325: ; * lisp/progmodes/project.el (project-current): Doc fix.

[Eli Zaretskii]

> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sun, 12 Jul 2020 23:07:46 +0300
> 
> On 12.07.2020 21:49, Eli Zaretskii wrote:
> >   The \"transient\" project instance is a special kind of value
> >   which denotes a project rooted in that directory and includes all
> > -files under it except for ones that match standard ignores.
> > +the files under the directory except for those that should be
> > +ignored (per `project-ignores').
> 
> The more accurate way to say this

What isn't accurate in what I wrote?  (The point being that 'standard
ignores" is something never mentioned in any other function or
variable in project.el, and so this term is unknown and confusing.)

>    except for those that are ignored
>    in the default 'project-ignores' implementation.
> 
> But this is basically a tautology. If we wanted to describe transient's 
> particular behavior we'd have to describe what it actually ignores (and 
> it honors vc-directory-exclusion-list as well as 
> grep-find-ignored-files). But that pretty clunky for one docstring.

Which is why I just mentioned the function which implements all that.

> So 
> perhaps that description should move somewhere. Into the Commentary, 
> maybe. Or directly to the cl-defgeneric's docstring, to which 
> project-current's docstring would after all refer to as "the default 
> `project-ignores' implementation".

You cannot usefully refer to the default implementation without saying
that this or that types of project use the default implementation,
which is IMO against your information encapsulation policies.

Re: master 0339325: ; * lisp/progmodes/project.el (project-current): Doc fix.

[Dmitry Gutov]

On 13.07.2020 06:48, Eli Zaretskii wrote:
>> From: Dmitry Gutov <dgutov@yandex.ru>
>> Date: Sun, 12 Jul 2020 23:07:46 +0300
>>
>> On 12.07.2020 21:49, Eli Zaretskii wrote:
>>>    The \"transient\" project instance is a special kind of value
>>>    which denotes a project rooted in that directory and includes all
>>> -files under it except for ones that match standard ignores.
>>> +the files under the directory except for those that should be
>>> +ignored (per `project-ignores').
>>
>> The more accurate way to say this
> 
> What isn't accurate in what I wrote?  (The point being that 'standard
> ignores" is something never mentioned in any other function or
> variable in project.el, and so this term is unknown and confusing.)

You were right to change it, but I'm talking about a further improvement.

>>     except for those that are ignored
>>     in the default 'project-ignores' implementation.
>>
>> But this is basically a tautology. If we wanted to describe transient's
>> particular behavior we'd have to describe what it actually ignores (and
>> it honors vc-directory-exclusion-list as well as
>> grep-find-ignored-files). But that pretty clunky for one docstring.
> 
> Which is why I just mentioned the function which implements all that.

Thing is, it's a generic function. It has multiple implementations, so 
saying 'per project-ignores' is not very informative.

>> So
>> perhaps that description should move somewhere. Into the Commentary,
>> maybe. Or directly to the cl-defgeneric's docstring, to which
>> project-current's docstring would after all refer to as "the default
>> `project-ignores' implementation".
> 
> You cannot usefully refer to the default implementation without saying
> that this or that types of project use the default implementation,
> which is IMO against your information encapsulation policies.

I think I see the reason for your choice. But I'm saying we can describe 
the behavior of project-ignores for that value.

Like we already described the behavior of 'project-root' for that value 
earlier in the same sentence ("denotes a project rooted in that directory").

While in general we might not want the consumers of project-current 
depend on particular behavior of either of these generic functions when 
using backends retrieved in the "normal" way, this is a special, 
fallback case that can inform the user of the behavior of said fallback 
backend. If we had a better place to describe that behavior in, though, 
we could move that there.

Re: master 0339325: ; * lisp/progmodes/project.el (project-current): Doc fix.

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Mon, 13 Jul 2020 14:09:54 +0300
> 
> >>     except for those that are ignored
> >>     in the default 'project-ignores' implementation.
> >>
> >> But this is basically a tautology. If we wanted to describe transient's
> >> particular behavior we'd have to describe what it actually ignores (and
> >> it honors vc-directory-exclusion-list as well as
> >> grep-find-ignored-files). But that pretty clunky for one docstring.
> > 
> > Which is why I just mentioned the function which implements all that.
> 
> Thing is, it's a generic function. It has multiple implementations, so 
> saying 'per project-ignores' is not very informative.

What I wrote talks only about the "transient" project, which AFAIU
uses the default implementation.  So I think it's as informative as
possible in this case.

But if you can come up with a better wording which doesn't lose useful
information, please do.

Btw, your implementation style makes it unusually hard to write good
doc strings.  I'm not quite sure why, but one possible reason is that
IMO you bring the generics too high, too close to the application
level, where doc strings should be more and more user-oriented.  When
this is coupled with your reluctance, to say the least, to disclose
details of what you consider to be opaque objects, it becomes hard to
say anything useful about many functions, because describing what
functions do usually involves talking about their inputs and outputs,
and how the former are processed into the latter.  The "for example"
method that I tried to use was about the only technique I could think
of to overcome those difficulties, and you reject even that.

Whether one agrees with your coding style or not, the difficulties it
presents to documenting our code are real, and I suggest that you
consider this disadvantage seriously, because it basically flies in
the face of long-standing traditions of Emacs self-documenting
features.

> > You cannot usefully refer to the default implementation without saying
> > that this or that types of project use the default implementation,
> > which is IMO against your information encapsulation policies.
> 
> I think I see the reason for your choice. But I'm saying we can describe 
> the behavior of project-ignores for that value.

Feel free to propose such a description, if you think such details are
important for a transient object.

> Like we already described the behavior of 'project-root' for that value 
> earlier in the same sentence ("denotes a project rooted in that directory").

That is actually quite vague.  It gives you some idea what such a
project means, but if you take it apart, it is just a fancy way of
saying that this project knows where its root is, and nothing else.

> While in general we might not want the consumers of project-current 
> depend on particular behavior of either of these generic functions when 
> using backends retrieved in the "normal" way, this is a special, 
> fallback case that can inform the user of the behavior of said fallback 
> backend. If we had a better place to describe that behavior in, though, 
> we could move that there.

The description of the transient project doesn't have to be detailed,
because it's, well, transient.  Its documentation needs to give some
idea what it is and how it is used; anything else is not a necessity,
IMO.

Re: master 0339325: ; * lisp/progmodes/project.el (project-current): Doc fix.

[Dmitry Gutov]

I'll try to keep this short.

On 13.07.2020 16:21, Eli Zaretskii wrote:
> Btw, your implementation style makes it unusually hard to write good
> doc strings.  I'm not quite sure why, but one possible reason is that
> IMO you bring the generics too high, too close to the application
> level, where doc strings should be more and more user-oriented.

The recent packages where you see this problem (and I agree that it's 
not ideal) are those which offer the possibility to customize complex 
behaviors. To an unprecedented degree, if we compare to older Emacs 
features.

Before xref and project.el, I think the closest one was minibuffer.el 
with its "completion tables" which can are often represented by 
anonymous functions. Which is another type of an opaque value.

Both xref and project.el could have used functions this way as well (a 
version doing that was initially proposed for xref). But any impression 
that such values would be easier to understand is illusory.

> When
> this is coupled with your reluctance, to say the least, to disclose
> details of what you consider to be opaque objects, it becomes hard to
> say anything useful about many functions, because describing what
> functions do usually involves talking about their inputs and outputs,
> and how the former are processed into the latter.

We're using cl-generic as an implementation technique. It's a more 
general implementation of dynamic dispatch, more commonly known to a lot 
of programmers from OOP.

> The "for example"
> method that I tried to use was about the only technique I could think
> of to overcome those difficulties, and you reject even that.

If we use the OOP analogy, you are trying to describe "object instances" 
in full detail, beyond the public interface, together with their 
"private fields", so to speak. It's a definite anti-pattern in OO community.

> Whether one agrees with your coding style or not, the difficulties it
> presents to documenting our code are real, and I suggest that you
> consider this disadvantage seriously, because it basically flies in
> the face of long-standing traditions of Emacs self-documenting
> features.

We could discuss alternative implementation approaches, but I think you 
will find none will fit the basic requirements of these packages.

Or at least that the possible options will require the client to treat 
the values as "opaque" exactly the same way.

Re: Opaque objects and Emacs documentation

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Fri, 17 Jul 2020 01:40:59 +0300
> 
> > Whether one agrees with your coding style or not, the difficulties it
> > presents to documenting our code are real, and I suggest that you
> > consider this disadvantage seriously, because it basically flies in
> > the face of long-standing traditions of Emacs self-documenting
> > features.
> 
> We could discuss alternative implementation approaches, but I think you 
> will find none will fit the basic requirements of these packages.
> 
> Or at least that the possible options will require the client to treat 
> the values as "opaque" exactly the same way.

Basically, you are saying that in your opinion this is as it should
be, and cannot be helped.

Which I think is against long-time Emacs tradition for documenting its
interfaces, and by that facilitating extensibility.  It is IMO wrong
to fill Emacs application levels with opaque objects which cannot be
usefully described; they should be a rare exception, but definitely
not the rule.

Therefore, we will most probably have many disputes in the future on
related issues which all boil down to this basic disagreement.

I wonder who else around here agrees with you on this, and invite
people who have an opinion to please speak up.  I've changed the
Subject to make it more descriptive.

Re: Opaque objects and Emacs documentation

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1175 bytes --]

On Fri, Jul 17, 2020 at 09:56:14AM +0300, Eli Zaretskii wrote:

[...]

> > Or at least that the possible options will require the client to treat 
> > the values as "opaque" exactly the same way.
> 
> Basically, you are saying that in your opinion this is as it should
> be, and cannot be helped.

I already weighted in, but in case that wasn't clear: while I think
that "layering" and "abstraction" is a useful structuring technique
when building up complex systems, the interfaces themselves have to
be subject to negotiation, as everything else.

This conflict is bound to happen time and again, because we software
folks tend to believe in (pick any) The True OO Way (TM), The True
Functional Way (also TM) or whatever.

At the end, it's the whole project what counts, and that's why a
calm and patient project leader is important. I think you're doing
a pretty awesome job, Eli.

And Dmitry -- as much respect as I have for your technical skills,
I think you're pushing it a bit too hard.

But that's just my opinion, and I'm known for having been wrong.

In any case, please, both of you: don't take all of that personally.

Cheers
-- t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Opaque objects and Emacs documentation

[Dmitry Gutov]

On 17.07.2020 11:13, tomas@tuxteam.de wrote:

> I already weighted in, but in case that wasn't clear: while I think
> that "layering" and "abstraction" is a useful structuring technique
> when building up complex systems, the interfaces themselves have to
> be subject to negotiation, as everything else.

How does this apply to this particular discussion? If you have a better 
suggestions for the interfaces, I'm all ears.

> And Dmitry -- as much respect as I have for your technical skills,
> I think you're pushing it a bit too hard.

I'm pushing because it's not the first time this kind of thing happened, 
and that has been negatively affecting my ability to contribute. I have 
to draw a line somewhere. Compromises aren't working, Eli simply skips 
past them.

And because that move: "please explain -- I write 3 screenfuls -- no, 
you didn't explain anything", is infuriating.

Re: Opaque objects and Emacs documentation

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1579 bytes --]

On Fri, Jul 17, 2020 at 01:40:54PM +0300, Dmitry Gutov wrote:
> On 17.07.2020 11:13, tomas@tuxteam.de wrote:
> 
> >I already weighted in, but in case that wasn't clear: while I think
> >that "layering" and "abstraction" is a useful structuring technique
> >when building up complex systems, the interfaces themselves have to
> >be subject to negotiation, as everything else.
> 
> How does this apply to this particular discussion? If you have a
> better suggestions for the interfaces, I'm all ears.

Hm. I haven't delved in the details, and I think it'd be useless:
Eli and you are both light-years smarter than me -- I just have
the feeling that Eli has a perspective there that you don't, just
because he's as deep in Emacs as few of us are; as current maintainer
his job is to favour overall consistency over perhaps bright, but
perhaps less understandable devices.

> >And Dmitry -- as much respect as I have for your technical skills,
> >I think you're pushing it a bit too hard.
> 
> I'm pushing because it's not the first time this kind of thing
> happened, and that has been negatively affecting my ability to
> contribute. I have to draw a line somewhere. Compromises aren't
> working, Eli simply skips past them.

I konw Eli can be stubborn :) But he's also extremely patient.

> And because that move: "please explain -- I write 3 screenfuls --
> no, you didn't explain anything", is infuriating.

I understand. Just give Eli the benefit of the doubt. I am certain
that there is no bad intention there. On both sides.

Cheers
-- t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Opaque objects and Emacs documentation

[Basil L. Contovounesios]

Eli Zaretskii <eliz@gnu.org> writes:

>> Cc: emacs-devel@gnu.org
>> From: Dmitry Gutov <dgutov@yandex.ru>
>> Date: Fri, 17 Jul 2020 01:40:59 +0300
>> 
>> > Whether one agrees with your coding style or not, the difficulties it
>> > presents to documenting our code are real, and I suggest that you
>> > consider this disadvantage seriously, because it basically flies in
>> > the face of long-standing traditions of Emacs self-documenting
>> > features.
>> 
>> We could discuss alternative implementation approaches, but I think you 
>> will find none will fit the basic requirements of these packages.
>> 
>> Or at least that the possible options will require the client to treat 
>> the values as "opaque" exactly the same way.
>
> Basically, you are saying that in your opinion this is as it should
> be, and cannot be helped.
>
> Which I think is against long-time Emacs tradition for documenting its
> interfaces, and by that facilitating extensibility.  It is IMO wrong
> to fill Emacs application levels with opaque objects which cannot be
> usefully described; they should be a rare exception, but definitely
> not the rule.
>
> Therefore, we will most probably have many disputes in the future on
> related issues which all boil down to this basic disagreement.
>
> I wonder who else around here agrees with you on this, and invite
> people who have an opinion to please speak up.  I've changed the
> Subject to make it more descriptive.

FWIW, I think Emacs documentation is one of the best things since pita
bread, and always hate to see it intentionally omitted.

There is nothing wrong with saying "the following are internal details
that shouldn't be relied on, but nevertheless here are their structure
and semantics".  In fact, I think it is infinitely better to do so.
There is little point or benefit in Emacs following conventions of other
communities, such as the aforementioned OO community.

With the exception of certain sensitive pieces of information such as
passwords in memory, I think Emacs benefits from being as open and
documented as reasonably possible.  There is nothing wrong with authors
not having the time, energy, or willingness to document everything they
write from the outset, but if other volunteers offer to complete the
documentation then I don't see why that should cause friction.

Multimethods are a very useful and powerful idiom, but the slight
increase in both source- and application-level complexity they add makes
it all the more important to document interfaces and intermediate
representations, even internal ones, for the benefit of users and
programmers alike.  Tooling can only go so far.

Thanks,

-- 
Basil

Re: Opaque objects and Emacs documentation

[Dmitry Gutov]

On 17.07.2020 13:37, Basil L. Contovounesios wrote:
> There is nothing wrong with saying "the following are internal details
> that shouldn't be relied on, but nevertheless here are their structure
> and semantics".  In fact, I think it is infinitely better to do so.

One should mind _where_ they write a sentence like this.

Re: Opaque objects and Emacs documentation

[Basil L. Contovounesios]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 17.07.2020 13:37, Basil L. Contovounesios wrote:
>> There is nothing wrong with saying "the following are internal details
>> that shouldn't be relied on, but nevertheless here are their structure
>> and semantics".  In fact, I think it is infinitely better to do so.
>
> One should mind _where_ they write a sentence like this.

Needless to say.

-- 
Basil

Re: Opaque objects and Emacs documentation

[Dmitry Gutov]

On 17.07.2020 09:56, Eli Zaretskii wrote:

>> Or at least that the possible options will require the client to treat
>> the values as "opaque" exactly the same way.
> 
> Basically, you are saying that in your opinion this is as it should
> be, and cannot be helped.

It shouldn't come as a surprise that I think there is no better 
technological choice. Otherwise I would have used it.

What should help is coming up with a better, consistent way to document 
these things.

> Which I think is against long-time Emacs tradition for documenting its
> interfaces, and by that facilitating extensibility.  It is IMO wrong
> to fill Emacs application levels with opaque objects which cannot be
> usefully described; they should be a rare exception, but definitely
> not the rule.

The problem here is that even if you document a particular value, it's 
_not useful_. It doesn't show you what you can or should do with it.

The main thing thing the user can do with that value is misuse it, by 
relying on its shape in the client code.

Re: Opaque objects and Emacs documentation

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Fri, 17 Jul 2020 13:53:35 +0300
> 
> It shouldn't come as a surprise that I think there is no better 
> technological choice. Otherwise I would have used it.

??? You are saying that the _only_ way to design project.el is to use
generics at that high level?  Just because sometimes a "project
instance" is a cons cell and sometimes some other Lisp structure?  I'm
surprised to hear that.

> What should help is coming up with a better, consistent way to document 
> these things.

I tried to find one, but couldn't.  If someone wants to propose a way,
I'm all ears.  I wrote those comments which started this thread
because it surprised me how difficult it was to try to keep the
limitations you impose on what should and what shouldn't be divulged,
and yet produce useful documentation of what the various public
functions do.  In all my 40 years of experience with Emacs, I have
never bumped into such a brick wall, with no way around it.

Later it became apparent that you are doing this on purpose, because
you have decided that certain directions of developing the package
should be made as hard as possible, something that I think is
completely against the spirit of Emacs development.  Readers of this
thread are invited to read

  https://lists.gnu.org/archive/html/emacs-devel/2020-07/msg00288.html

especially under "Here are things we want third-party code to be able
to do" and "Here are, on the other hand, things that people generally
shouldn't do".  Again, I'm interested to know how many of us here
share those views.

> > Which I think is against long-time Emacs tradition for documenting its
> > interfaces, and by that facilitating extensibility.  It is IMO wrong
> > to fill Emacs application levels with opaque objects which cannot be
> > usefully described; they should be a rare exception, but definitely
> > not the rule.
> 
> The problem here is that even if you document a particular value, it's 
> _not useful_. It doesn't show you what you can or should do with it.

It was very useful for me, and so I presume it could be useful for
others.  Even if you think it is not useful for you, the fact that
fellow developers tell you the contrary should be a reason to revisit
your views, and maybe allow for other views as well, even if you
disagree.

Documentation should strive to serve different ways of studying and
developing Emacs, not just a single way that you personally think is
The (only) Right Thing.

> The main thing thing the user can do with that value is misuse it, by 
> relying on its shape in the client code.

Once again: concealing information because someone could be silly
enough to misuse it punishes many valid uses on behalf of a few
invalid ones.  We should treat our users as responsible adults, even
if some of them aren't.  Those who aren't will eventually be amply
punished, or will recognize their mistakes and get their act together.

Re: Opaque objects and Emacs documentation

[Noam Postavsky]

On Fri, 17 Jul 2020 at 07:14, Eli Zaretskii <eliz@gnu.org> wrote:
>
> > Cc: emacs-devel@gnu.org
> > From: Dmitry Gutov <dgutov@yandex.ru>
> > Date: Fri, 17 Jul 2020 13:53:35 +0300
> >
> > It shouldn't come as a surprise that I think there is no better
> > technological choice. Otherwise I would have used it.
>
> ??? You are saying that the _only_ way to design project.el is to use
> generics at that high level?

That sounds like a bit of a misreading: Dmitry didn't say "_only_
way", he said "there is no better" way.

Re: Opaque objects and Emacs documentation

[Eli Zaretskii]

> From: Noam Postavsky <npostavs@gmail.com>
> Date: Fri, 17 Jul 2020 08:02:29 -0400
> Cc: Dmitry Gutov <dgutov@yandex.ru>, Emacs developers <emacs-devel@gnu.org>
> 
> On Fri, 17 Jul 2020 at 07:14, Eli Zaretskii <eliz@gnu.org> wrote:
> >
> > > Cc: emacs-devel@gnu.org
> > > From: Dmitry Gutov <dgutov@yandex.ru>
> > > Date: Fri, 17 Jul 2020 13:53:35 +0300
> > >
> > > It shouldn't come as a surprise that I think there is no better
> > > technological choice. Otherwise I would have used it.
> >
> > ??? You are saying that the _only_ way to design project.el is to use
> > generics at that high level?
> 
> That sounds like a bit of a misreading: Dmitry didn't say "_only_
> way", he said "there is no better" way.

OK, point taken.  But the question still stands, even with the
modified wording: is there really no way of comparable quality to
implement something like project.el except by keeping opaque objects
at those high levels?

Re: Opaque objects and Emacs documentation

[Dmitry Gutov]

On 17.07.2020 15:52, Eli Zaretskii wrote:
> OK, point taken.  But the question still stands, even with the
> modified wording: is there really no way of comparable quality to
> implement something like project.el except by keeping opaque objects
> at those high levels?

Where else would we keep "opaque objects"?

At lower levels, all objects are inevitably "transparent".

Re: Opaque objects and Emacs documentation

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Fri, 17 Jul 2020 16:09:14 +0300
> 
> On 17.07.2020 15:52, Eli Zaretskii wrote:
> > OK, point taken.  But the question still stands, even with the
> > modified wording: is there really no way of comparable quality to
> > implement something like project.el except by keeping opaque objects
> > at those high levels?
> 
> Where else would we keep "opaque objects"?

The question you are asking is irrelevant, and certainly doesn't
answer my question above.

You are saying that there's no comparable or better way of
implementing this functionality, and I question that.  I hope you or
someone else can explain why the way you use generics in project.el is
indeed so much better than the alternatives.

Re: Opaque objects and Emacs documentation

[Dmitry Gutov]

On 17.07.2020 16:56, Eli Zaretskii wrote:
> You are saying that there's no comparable or better way of
> implementing this functionality, and I question that.  I hope you or
> someone else can explain why the way you use generics in project.el is
> indeed so much better than the alternatives.

I have done a lot of explaining over the last several days. Including 
some answers relevant to this question.

None of that seems to have had any impact.

You are welcome to propose a patch that would implement the same 
observable functionality (a partial one would do), and I'll tell you 
what's wrong with it.

Re: Opaque objects and Emacs documentation

[Eli Zaretskii]

> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Fri, 17 Jul 2020 17:35:20 +0300
> Cc: npostavs@gmail.com, emacs-devel@gnu.org
> 
> You are welcome to propose a patch that would implement the same 
> observable functionality (a partial one would do), and I'll tell you 
> what's wrong with it.

The issue I suggested to discuss in this thread is a general one,
project.el is just an example.  Thus, asking for a patch is not going
to advance the discussion.  By contrast, an explanation why you think
project.el could not have successfully implemented in any other way
would advance the discussion, because it would show us the advantages
of such a design and implementation, to be weighed against the
disadvantages I pointed out.

Re: Opaque objects and Emacs documentation

[Dmitry Gutov]

On 17.07.2020 17:59, Eli Zaretskii wrote:
>> From: Dmitry Gutov <dgutov@yandex.ru>
>> Date: Fri, 17 Jul 2020 17:35:20 +0300
>> Cc: npostavs@gmail.com, emacs-devel@gnu.org
>>
>> You are welcome to propose a patch that would implement the same
>> observable functionality (a partial one would do), and I'll tell you
>> what's wrong with it.
> 
> The issue I suggested to discuss in this thread is a general one,
> project.el is just an example.  Thus, asking for a patch is not going
> to advance the discussion.

It would.

> By contrast, an explanation why you think
> project.el could not have successfully implemented in any other way
> would advance the discussion, because it would show us the advantages
> of such a design and implementation, to be weighed against the
> disadvantages I pointed out.

I explain something, you say "no, you haven't explained". This game is 
exhausting and unfair.

Re: Opaque objects and Emacs documentation

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

If we want to dowplay the current subsctructure of a piece of data,
that does not require mystifying that current subsctructure.  It is
easy to avoid that.

We define an accessor defsubst for each of the data it contains.  We
document its meaning.  We define a constructor function, and document
the meaning of each argument.  We manipulate them always through those
functions.  Voilà!

The subsctructure is clearly shown and yet changing it later is painless.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Opaque objects and Emacs documentation

[Dmitry Gutov]

On 18.07.2020 04:54, Richard Stallman wrote:
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
> 
> If we want to dowplay the current subsctructure of a piece of data,
> that does not require mystifying that current subsctructure.  It is
> easy to avoid that.
> 
> We define an accessor defsubst for each of the data it contains.  We
> document its meaning.  We define a constructor function, and document
> the meaning of each argument.  We manipulate them always through those
> functions.  Voilà!
> 
> The subsctructure is clearly shown and yet changing it later is painless.

This is pretty much what I'm doing here.

Only a little more complex, because there are multiple data types that 
implement the same interface.

Re: Opaque objects and Emacs documentation

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > We define an accessor defsubst for each of the data it contains.  We
  > > document its meaning.  We define a constructor function, and document
  > > the meaning of each argument.  We manipulate them always through those
  > > functions.  Voilà!
  > > 
  > > The subsctructure is clearly shown and yet changing it later is painless.

  > This is pretty much what I'm doing here.

  > Only a little more complex, because there are multiple data types that 
  > implement the same interface.

Eli, do you think the difference is significant?

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Opaque objects and Emacs documentation

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: eliz@gnu.org, npostavs@gmail.com, emacs-devel@gnu.org
> Date: Sat, 18 Jul 2020 22:27:22 -0400
> 
>   > > We define an accessor defsubst for each of the data it contains.  We
>   > > document its meaning.  We define a constructor function, and document
>   > > the meaning of each argument.  We manipulate them always through those
>   > > functions.  Voilà!
>   > > 
>   > > The subsctructure is clearly shown and yet changing it later is painless.
> 
>   > This is pretty much what I'm doing here.
> 
>   > Only a little more complex, because there are multiple data types that 
>   > implement the same interface.
> 
> Eli, do you think the difference is significant?

I'm probably missing something, because I don't even see the
similarity.  Let me explain.

What you, Richard, describe is a paradigm we use in several places in
Emacs.  One example is the POSITION object originally used for
describing mouse clicks.  Another example is ispell.el's
ispell-dictionary-alist.  Yet another example is the
window-configuration object manipulated by the likes of
current-window-configuration and set-window-configuration.

In these and other similar examples, we do use defsubst-type accessors
to the fields or slots of the object.  But we also have the following
important traits of the respective objects:

  . a way of creating an object of that type
  . a predicate to test whether an arbitrary Lisp value is an object
    of that type
  . a function to compare two objects of this type

Some of the objects (the first 2 examples above) have their Lisp
structure completely documented, in which case the predicate and the
comparison functions are not strictly needed (but we still provide
them, at least in some cases).  When the structure is not documented,
the predicate and the comparison functions are IMO a necessity, and
there should be either an exhaustive set of accessors for the public
fields of the object, or at least a documented list of those fields
and how to access them.

Taken together, these measures present a useful, usable, and fairly
complete description of an abstraction.

I can see none of that in project.el regarding the "project instance"
abstraction.  Not even a description of what a "project instance" is,
conceptually, or what it entails, neither in terms of the "outside
world" (i.e. the domain of its applicability), nor in terms of the
Lisp implementation.  And I can only guess what is considered the
analogy of the accessors you were alluding to, because it isn't
documented, either.  (Or maybe the idea is that none of the project's
fields are public? but then what is project-root about?)

So I don't see how these two techniques can be usefully compared, let
alone proclaimed "pretty much" the same.

The reason we were given for this lack of crucial documentation in
project.el is the desire to leave the API open to extensions.  IOW,
if, for example, we say today that a project entails a root directory,
a collection of files in that directory and its subdirectories, and
optionally a set of external tools, such as a VCS, to provide some
project-related services, then it will make it harder to add in the
future projects of a very different kind.  Which is a valid concern,
and the reason I started this thread.  Because having user commands
that allude to "project instance" without even a minimal explanation
of what that is makes our documentation of such commands almost
useless, and the question that bothers me is how to keep the quality
of our documentation at its usual high level while using these or
similar design and implementation patterns.

Bottom line: unless I'm missing something important, I think the
difference is significant, and downplaying it is not a good way of
coming up with useful solutions for this important issue.

Re: Opaque objects and Emacs documentation

[Dmitry Gutov]

On 19.07.2020 17:48, Eli Zaretskii wrote:
>    . a predicate to test whether an arbitrary Lisp value is an object
>      of that type

A predicate would look like:

(defun project-instance-p (pr)
   (condition-case nil
       (progn
         (project-roots pr)
         t)
     (error
      nil)))

But I don't see how it's going to help you write documentation.

And no code is going to call it, so it's not defined.

>    . a function to compare two objects of this type

As discussed, it's a little function called 'equal'.

If we find out we need something more complex, we'll add it.

Re: Opaque objects and Emacs documentation

[Eli Zaretskii]

> Cc: npostavs@gmail.com, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Tue, 21 Jul 2020 22:00:15 +0300
> 
> On 19.07.2020 17:48, Eli Zaretskii wrote:
> >    . a predicate to test whether an arbitrary Lisp value is an object
> >      of that type
> 
> A predicate would look like:
> 
> (defun project-instance-p (pr)
>    (condition-case nil
>        (progn
>          (project-roots pr)
>          t)
>      (error
>       nil)))

I actually thought about cl-defgeneric etc., since obviously each type
of project should have its own implementation.

> But I don't see how it's going to help you write documentation.

It makes the object system more complete.  Suppose some application
would like to check it gets a valid object, before dispatching?

> >    . a function to compare two objects of this type
> 
> As discussed, it's a little function called 'equal'.

Again, I envisioned a cl-defgeneric here.  Using 'equal' only works
for simple forms, but if we are willing to limit ourselves to such
simple forms, why do we need generics at all?

Re: Opaque objects and Emacs documentation

[Dmitry Gutov]

On 21.07.2020 22:36, Eli Zaretskii wrote:

>> A predicate would look like:
>>
>> (defun project-instance-p (pr)
>>     (condition-case nil
>>         (progn
>>           (project-roots pr)
>>           t)
>>       (error
>>        nil)))
> 
> I actually thought about cl-defgeneric etc., since obviously each type
> of project should have its own implementation.

There's no point. If it implements project-root, it's a project.

>> But I don't see how it's going to help you write documentation.
> 
> It makes the object system more complete.  Suppose some application
> would like to check it gets a valid object, before dispatching?

I can imagine some marginal utility in that, but not enough to mandate 
every backend to implement a project-instance-p method which simply 
returns t.

>>>     . a function to compare two objects of this type
>>
>> As discussed, it's a little function called 'equal'.
> 
> Again, I envisioned a cl-defgeneric here.  Using 'equal' only works
> for simple forms,

Lists, cl-structs, eieio classes, etc, can also be equal.

Re: Opaque objects and Emacs documentation

[Tomas Hlavaty]

On Tue 21 Jul 2020 at 22:36, Eli Zaretskii <eliz@gnu.org> wrote:
>> >    . a function to compare two objects of this type
>> 
>> As discussed, it's a little function called 'equal'.
>
> Again, I envisioned a cl-defgeneric here.  Using 'equal' only works
> for simple forms, but if we are willing to limit ourselves to such
> simple forms, why do we need generics at all?

equal seems to also work for classes in emacs lisp (are they simple
forms?), because they are implemented on top of structs which emacs lisp
equal understands.

this is different than in common lisp

in emacs lisp:

   (defclass foo () ((hi :initarg :hi)))
   (equal (make-instance 'foo :hi 42) (make-instance 'foo :hi 42))
   => t

in common lisp:

   CL-USER> (defclass foo () ((hi :initarg :hi)))
   #<STANDARD-CLASS COMMON-LISP-USER::FOO>
   CL-USER> (equal (make-instance 'foo :hi 42) (make-instance 'foo :hi 42))
   NIL

there don't seem to be opaque objects in emacs lisp

   ;; -*- lexical-binding: t -*-
   (defun make-counter (n) (lambda () (incf n)))
   (equal (make-counter 42) (make-counter 42))
   => t
   (equal (make-counter 42) (funcall (make-counter 42)))
   => nil

what are opaque objects in emacs lisp?

what are non-simple forms?

CLOS seems to be spreading in Emacs which is a shame because it is
better to avoid it when writing robust and lasting software.

Re: Opaque objects and Emacs documentation

[Tomas Hlavaty]

On Tue 21 Jul 2020 at 22:00, Dmitry Gutov <dgutov@yandex.ru> wrote:
> On 19.07.2020 17:48, Eli Zaretskii wrote:
>>    . a predicate to test whether an arbitrary Lisp value is an object
>>      of that type
>
> A predicate would look like:
>
> (defun project-instance-p (pr)
>    (condition-case nil
>        (progn
>          (project-roots pr)
>          t)
>      (error
>       nil)))

that's ugly

it's better not to use errors for control flow

what happens if project-roots raises error on legitimate project
instance?

Re: Opaque objects and Emacs documentation

[Dmitry Gutov]

On 21.07.2020 23:49, Tomas Hlavaty wrote:
> what happens if project-roots raises error on legitimate project
> instance?

If project-root doesn't work, it's not a legitimate instance. It is the 
responsibility of each backend to make sure this method works.

Re: Opaque objects and Emacs documentation

[Tomas Hlavaty]

On Tue 21 Jul 2020 at 23:59, Dmitry Gutov <dgutov@yandex.ru> wrote:
> On 21.07.2020 23:49, Tomas Hlavaty wrote:
>> what happens if project-roots raises error on legitimate project
>> instance?
>
> If project-root doesn't work, it's not a legitimate instance. It is the 
> responsibility of each backend to make sure this method works.

that's pretty strong and flawed constraint

you are also conflating what project instance is and what project-root
does

you are restricting project instances to such cases where only trivial
implementations of project-root cannot ever raise an error

Re: Opaque objects and Emacs documentation

[Dmitry Gutov]

On 22.07.2020 00:20, Tomas Hlavaty wrote:
> you are also conflating what project instance is and what project-root
> does

A project is what a project does. It's a protocol.

> you are restricting project instances to such cases where only trivial
> implementations of project-root cannot ever raise an error

Well, we could differentiate by one particular error ("not 
implemented"). But that's a detail.

Re: Opaque objects and Emacs documentation

[Dmitry Gutov]

On 19.07.2020 17:48, Eli Zaretskii wrote:
> Because having user commands
> that allude to "project instance" without even a minimal explanation
> of what that is makes our documentation of such commands almost
> useless, and the question that bothers me is how to keep the quality
> of our documentation at its usual high level while using these or
> similar design and implementation patterns.

I started replying to say that you are welcome to propose improvements 
for better description of allowed semantics, but then again, you 
probably meant something like more details about which cons cell 
contains which values.

In the meantime, I have added more text to Commentary in project.el. 
Please take a look whenever convenient.

Re: Opaque objects and Emacs documentation

[Eli Zaretskii]

> Cc: npostavs@gmail.com, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Tue, 21 Jul 2020 22:08:47 +0300
> 
> On 19.07.2020 17:48, Eli Zaretskii wrote:
> > Because having user commands
> > that allude to "project instance" without even a minimal explanation
> > of what that is makes our documentation of such commands almost
> > useless, and the question that bothers me is how to keep the quality
> > of our documentation at its usual high level while using these or
> > similar design and implementation patterns.
> 
> I started replying to say that you are welcome to propose improvements 
> for better description of allowed semantics, but then again, you 
> probably meant something like more details about which cons cell 
> contains which values.

Actually, no.  For user commands it is better to describe a project in
terms of outside world: files, directories, repositories, etc.

> In the meantime, I have added more text to Commentary in project.el. 
> Please take a look whenever convenient.

It is orders of magnitude better than what we had before.  Thank you
very much for working on that.

Re: Opaque objects and Emacs documentation

[Dmitry Gutov]

On 21.07.2020 22:38, Eli Zaretskii wrote:

>> I started replying to say that you are welcome to propose improvements
>> for better description of allowed semantics, but then again, you
>> probably meant something like more details about which cons cell
>> contains which values.
> 
> Actually, no.  For user commands it is better to describe a project in
> terms of outside world: files, directories, repositories, etc.

Right.

Well, we're still growing the capabilities, and with that, the notion of 
what constitutes a project, is expanding bit by bit.

>> In the meantime, I have added more text to Commentary in project.el.
>> Please take a look whenever convenient.
> 
> It is orders of magnitude better than what we had before.  Thank you
> very much for working on that.

Welcome!

But I basically retold (sometimes copied) the contents of the recent 
discussion. Which I said I'd do.

Re: Opaque objects and Emacs documentation

[John Yates]

[-- Attachment #1: Type: text/plain, Size: 2682 bytes --]

Wrote this up but not sure where in the thread to inject it.

Here is my understanding of Dmitry's design:

A backend's hook function, when presented a context (is this
always a directory?), is expected to answer the implicit
question "Do you recognize this as a form of project for which
you are prepared to take responsibility?".  The function returns
nil if no, non-nil if yes.  Returning non-nil effectively asserts
that the presented context is an instance of the form of projects
handled by that backend.  The value returned ia a backend private
identification of that project instance (i.e. an intance handle).

"Takig responsibility" entails being prepared to support a -
potentially growing over time - set of project methods.  The
non-nil object returned by a backend hook function will be
presented at the invocation of a project method.  This allows a
backend to know the specific project instance upon which it is
being asked to operate.

The project.el dispatcher never interprets an instance handle.
It merely holds onto it and then presents it to a backend as part
of a method invocation.  The handle only needs to capture (in a
backend private fashion) enough context to allow carrying out the
various methods.

As an engineering concern the backend should be able to execute
the expected methods in an acceptably responsive manner.  For a
traditional directory tree-based project identified by some VCS
feature on a local file system a mere pointer to the project's
root directory may be sufficient.  Other forms of project (say
a more complex topology than a single directory tree, complex
permissions issues, remote connections, etc) may prompt a backend
to capture more information in its project instance handles.
But that is an entirely backend private design decision.

On Tue, Jul 21, 2020 at 3:09 PM Dmitry Gutov <dgutov@yandex.ru> wrote:

> On 19.07.2020 17:48, Eli Zaretskii wrote:
> > Because having user commands
> > that allude to "project instance" without even a minimal explanation
> > of what that is makes our documentation of such commands almost
> > useless, and the question that bothers me is how to keep the quality
> > of our documentation at its usual high level while using these or
> > similar design and implementation patterns.
>
> I started replying to say that you are welcome to propose improvements
> for better description of allowed semantics, but then again, you
> probably meant something like more details about which cons cell
> contains which values.
>
> In the meantime, I have added more text to Commentary in project.el.
> Please take a look whenever convenient.
>
>

-- 
John Yates
505 Tremont St, #803
Boston, MA 02116

[-- Attachment #2: Type: text/html, Size: 3586 bytes --]

Re: Opaque objects and Emacs documentation

[Dmitry Gutov]

On 21.07.2020 23:29, John Yates wrote:
> The project.el dispatcher never interprets an instance handle.
> It merely holds onto it and then presents it to a backend as part
> of a method invocation.  The handle only needs to capture (in a
> backend private fashion) enough context to allow carrying out the
> various methods.

That is right.

There is really very little use in knowing what such an instance handle 
contains and how, since the caller will just need to pass it forward to 
one of the project methods.

Re: Opaque objects and Emacs documentation

[Eli Zaretskii]

> From: John Yates <john@yates-sheets.org>
> Date: Tue, 21 Jul 2020 16:29:54 -0400
> Cc: Eli Zaretskii <eliz@gnu.org>, Richard Stallman <rms@gnu.org>, npostavs@gmail.com, 
> 	Emacs developers <emacs-devel@gnu.org>
> 
> Wrote this up but not sure where in the thread to inject it.
> 
> Here is my understanding of Dmitry's design:
> [...]

Thanks.

This is all understood and agreed to.  The issue here is not what the
design is, but how to document the commands, functions, and interfaces
under such a design.

Re: Opaque objects and Emacs documentation

[Dmitry Gutov]

On 18.07.2020 04:54, Richard Stallman wrote:
> We define an accessor defsubst for each of the data it contains.  We
> document its meaning.  We define a constructor function, and document
> the meaning of each argument.  We manipulate them always through those
> functions.  Voilà!

Here's a question that will perhaps help us solve this dispute.

Consider the docstring of said constructor function. What should it say 
its return value is?

Re: Opaque objects and Emacs documentation

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Consider the docstring of said constructor function. What should it say 
  > its return value is?

One natural possibility:

  Return an empty project object.  You need to store values in it with ...

Another:

  Return a project object with name MAME and containing files FILES.

I am guessing here.  I don't know what these projects do.
I don't know what data a project object should have.


-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Opaque objects and Emacs documentation

[Dmitry Gutov]

On 20.07.2020 05:44, Richard Stallman wrote:
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
> 
>    > Consider the docstring of said constructor function. What should it say
>    > its return value is?
> 
> One natural possibility:
> 
>    Return an empty project object.  You need to store values in it with ...
> 
> Another:
> 
>    Return a project object with name MAME and containing files FILES.
> 
> I am guessing here.  I don't know what these projects do.
> I don't know what data a project object should have.

Okay, see, neither option is going to work.

We have a "factory" function (called 'project-current') which delegates 
project instance creation to a hook. Which is what makes it extensible. 
The only argument is DIRECTORY, meaning the current directory. The 
functions in the hook get called in turn, and the first one that returns 
non-nil is going to be used. Its return value must be a "project 
instance" which satisfies a certain protocol defined by several 
cl-generic methods (think "interface" in Java, or "typeclass" in 
Haskell, or "protocol" in Clojure, or multimethods elsewhere). One 
method is mandatory to be defined for all project implementations, the 
rest have default implementations.

Thanks to the cl-generic infrastructure, the datatype representing the 
current project can be almost anything, and the structure of the fields 
inside can be arbitrary. This is sorted out by each individual backend 
which knows how to access fields inside "project instances" belonging to 
it, and which implements the aforementioned protocol for that instance type.

The two built-in backends use very simple structures (right now: conses 
like (TYPE-TAG . PROJECT-ROOT)), which is very obvious from their 
implementations. The core of the dispute is whether project-current's 
docstring should document the return value in detail (giving an example 
of built-in project instances's structure). Eli is insisting on it 
because (AFAICT) that suits his thought process and he thinks that will 
help users/clients to make sense of how things work.

I am adamantly against this because it's against the design (clients 
should program against the public interface, not against the 
implementation), and is likely to encourage incorrect programs.

Of course, we can add this information elsewhere, e.g. in the comments 
of the "backend" functions (which are the elements of the hook I 
mentioned above). I haven't done so because, well, the implementations 
are fairly transparent and the structures are trivial. But I wouldn't 
object against somebody doing that.

 > It is important to document the meaning of each slot -- that may not
 > be obvious.  Which values are valid?

The space of possible structures of valid project instances is 
theoretically infinite.

Re: Opaque objects and Emacs documentation

[tomas]

[-- Attachment #1: Type: text/plain, Size: 887 bytes --]

Hi,

On Tue, Jul 21, 2020 at 04:09:18AM +0300, Dmitry Gutov wrote:

[...]

I don't think it's intentional, but please reconsider this:

> [...] The core of the dispute is whether
> project-current's docstring should document the return value in
> detail (giving an example of built-in project instances's
> structure). Eli is insisting on it because (AFAICT) that suits his
> thought process and he thinks that will help users/clients to make
> sense of how things work.

in a few words: Eli's thought process vs...

> I am adamantly against this because it's against the design (clients
> should program against the public interface, not against the
> implementation), and is likely to encourage incorrect programs.

... The Right Way To Do It (TM).

It comes across as a little... biased. Which is understandable,
but not a good position when bargaining, nevertheless ;-)

Cheers
-- t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Opaque objects and Emacs documentation

[Dmitry Gutov]

On 21.07.2020 11:57, tomas@tuxteam.de wrote:

> I don't think it's intentional, but please reconsider this:

Reconsider the phrasing?

>> [...] The core of the dispute is whether
>> project-current's docstring should document the return value in
>> detail (giving an example of built-in project instances's
>> structure). Eli is insisting on it because (AFAICT) that suits his
>> thought process and he thinks that will help users/clients to make
>> sense of how things work.
> 
> in a few words: Eli's thought process vs...

That's how I understand it. And there is nothing bad in documenting code 
along the lines of someone's thoughts. It's generally a good thing. But 
not when it defeats the purpose of how the code is written.

>> I am adamantly against this because it's against the design (clients
>> should program against the public interface, not against the
>> implementation), and is likely to encourage incorrect programs.
> 
> ... The Right Way To Do It (TM).
> 
> It comes across as a little... biased. Which is understandable,
> but not a good position when bargaining, nevertheless ;-)

The whole situation is biased, in multiple respects.

Re: Opaque objects and Emacs documentation

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1552 bytes --]

On Tue, Jul 21, 2020 at 01:14:16PM +0300, Dmitry Gutov wrote:
> On 21.07.2020 11:57, tomas@tuxteam.de wrote:
> 
> >I don't think it's intentional, but please reconsider this:
> 
> Reconsider the phrasing?
> 
> >>[...] The core of the dispute is whether
> >>project-current's docstring should document the return value in
> >>detail (giving an example of built-in project instances's
> >>structure). Eli is insisting on it because (AFAICT) that suits his
> >>thought process and he thinks that will help users/clients to make
> >>sense of how things work.
> >
> >in a few words: Eli's thought process vs...
> 
> That's how I understand it. And there is nothing bad in documenting
> code along the lines of someone's thoughts. It's generally a good
> thing. But not when it defeats the purpose of how the code is
> written.
> 
> >>I am adamantly against this because it's against the design (clients
> >>should program against the public interface, not against the
> >>implementation), and is likely to encourage incorrect programs.
> >
> >... The Right Way To Do It (TM).
> >
> >It comes across as a little... biased. Which is understandable,
> >but not a good position when bargaining, nevertheless ;-)
> 
> The whole situation is biased, in multiple respects.

Yes, and I do feel your pain. Don't take this as a personal attack,
that'd the last thing I wish for. It's just that I do understand
also Eli's position, and I feel that it has merit -- it is unfair
to depict it as "just Eli's thoughts".

Cheers
-- t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Opaque objects and Emacs documentation

[Eli Zaretskii]

> Cc: eliz@gnu.org, npostavs@gmail.com, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Tue, 21 Jul 2020 04:09:18 +0300
> 
> We have a "factory" function (called 'project-current') which
> delegates project instance creation to a hook. Which is what makes
> it extensible. The only argument is DIRECTORY, meaning the current
> directory. The functions in the hook get called in turn, and the
> first one that returns non-nil is going to be used. Its return value
> must be a "project instance" which satisfies a certain protocol
> defined by several cl-generic methods (think "interface" in Java, or
> "typeclass" in Haskell, or "protocol" in Clojure, or multimethods
> elsewhere). One method is mandatory to be defined for all project
> implementations, the rest have default implementations.
> 
> Thanks to the cl-generic infrastructure, the datatype representing
> the current project can be almost anything, and the structure of the
> fields inside can be arbitrary. This is sorted out by each
> individual backend which knows how to access fields inside "project
> instances" belonging to it, and which implements the aforementioned
> protocol for that instance type.

The type of "project" returned by project-current can indeed be
anything, but each hook function returns a specific type, or maybe a
small set of specific types.  So the type returned by each hook could
be documented in the doc string of that hook in terms suggested by
Richard (or something similar).  For example, project-try-vc could
have a doc string describing that for the project instance it returns.
Similarly, the "transient" project instance returned by
project-current itself, when a project doesn't yet exist, is also
known, and its structure could be similarly documented without
impediment to extensibility.

> The two built-in backends use very simple structures (right now: conses like (TYPE-TAG . PROJECT-ROOT)), which is very obvious from their implementations.

Whether the structure is obvious from the implementation may or may
not be true (and the author of the code is usually not the best judge
of that), but doesn't solve the issue at hand, IMO.  A good
documentation of an interface should allow a developer to write code
that uses the interface without looking at the interface's
implementation.  If it is necessary to consult the implementation,
that is an indication of some deficiency in the docs, and we should
try to avoid that as much as possible.

Re: Opaque objects and Emacs documentation

[Dmitry Gutov]

On 21.07.2020 17:34, Eli Zaretskii wrote:
> So the type returned by each hook could
> be documented in the doc string of that hook in terms suggested by
> Richard (or something similar).

It could. It would not be a significant problem.

> Similarly, the "transient" project instance returned by
> project-current itself, when a project doesn't yet exist, is also
> known, and its structure could be similarly documented without
> impediment to extensibility.

If you say that, could you give an example of something that *would* be 
an impediment to extensibility? I'd like to see where we stand now.

> Whether the structure is obvious from the implementation may or may
> not be true (and the author of the code is usually not the best judge
> of that), but doesn't solve the issue at hand, IMO.

So have we moved on from trying to document the examples in the 
docstrings of project-find-functions or project-current?

> A good
> documentation of an interface should allow a developer to write code
> that uses the interface without looking at the interface's
> implementation.

Right. But there won't be any third-party callers of project-try-vc, 
this function's only purpose is to be inside project-find-functions.

Or at least there shouldn't be any until we're reasonably certain we 
want to support it as a public function, and we understand in what 
circumstances it might be called.

So as things currently stand, the responsibility for it being correct 
and accepting the right argument lies on its author, and not on any 
third-party developers.

> If it is necessary to consult the implementation,
> that is an indication of some deficiency in the docs, and we should
> try to avoid that as much as possible.

No disagreement there, as long as we're talking about public functions.

Re: Opaque objects and Emacs documentation

[Eli Zaretskii]

> Cc: rms@gnu.org, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Tue, 21 Jul 2020 21:56:11 +0300
> 
> On 21.07.2020 17:34, Eli Zaretskii wrote:
> > So the type returned by each hook could
> > be documented in the doc string of that hook in terms suggested by
> > Richard (or something similar).
> 
> It could. It would not be a significant problem.
> 
> > Similarly, the "transient" project instance returned by
> > project-current itself, when a project doesn't yet exist, is also
> > known, and its structure could be similarly documented without
> > impediment to extensibility.
> 
> If you say that, could you give an example of something that *would* be 
> an impediment to extensibility?

What you said: describing what generics returns, or what
project-current returns in general.

> > Whether the structure is obvious from the implementation may or may
> > not be true (and the author of the code is usually not the best judge
> > of that), but doesn't solve the issue at hand, IMO.
> 
> So have we moved on from trying to document the examples in the 
> docstrings of project-find-functions or project-current?

If those are the rules of the game, yes.  IOW, if it's not okay to
describe the possible forms of the object in the doc string of
project-find-functions, but okay to describe them in the individual
doc strings of each hook that can be put there, then it could be an
acceptable compromise, at least from my POV.

> > A good
> > documentation of an interface should allow a developer to write code
> > that uses the interface without looking at the interface's
> > implementation.
> 
> Right. But there won't be any third-party callers of project-try-vc, 
> this function's only purpose is to be inside project-find-functions.

I'm thinking about additional "authors" who'd like to add
functionality to existing project backends.

> So as things currently stand, the responsibility for it being correct 
> and accepting the right argument lies on its author, and not on any 
> third-party developers.

Yes, but "its author" doesn't have to be a single person, and doesn't
have to be the same person who wrote the initial implementations.

> > If it is necessary to consult the implementation,
> > that is an indication of some deficiency in the docs, and we should
> > try to avoid that as much as possible.
> 
> No disagreement there, as long as we're talking about public functions.

Are we still under the rule that any function without 2 dashes in its
name is public?  If so, then I think we have only discussed public
functions in this and related threads.

Re: Opaque objects and Emacs documentation

[Dmitry Gutov]

On 21.07.2020 22:33, Eli Zaretskii wrote:

>>> Similarly, the "transient" project instance returned by
>>> project-current itself, when a project doesn't yet exist, is also
>>> known, and its structure could be similarly documented without
>>> impediment to extensibility.
>>
>> If you say that, could you give an example of something that *would* be
>> an impediment to extensibility?
> 
> What you said: describing what generics returns, or what
> project-current returns in general.

No, describing "in general" is good. What is bad is describing "in 
particular" when a function is "general" and can return values that are 
different from the examples. project-current is "general".

And speaking of "transient", it's not helpful to say it returns a cons 
(transient . root) because that doesn't say anything about the project 
behavior anyway (which is the important part).

>>> Whether the structure is obvious from the implementation may or may
>>> not be true (and the author of the code is usually not the best judge
>>> of that), but doesn't solve the issue at hand, IMO.
>>
>> So have we moved on from trying to document the examples in the
>> docstrings of project-find-functions or project-current?
> 
> If those are the rules of the game, yes.  IOW, if it's not okay to
> describe the possible forms of the object in the doc string of
> project-find-functions, but okay to describe them in the individual
> doc strings of each hook that can be put there, then it could be an
> acceptable compromise, at least from my POV.

Very well. The "transient" project is not on the hook, though.

>>> A good
>>> documentation of an interface should allow a developer to write code
>>> that uses the interface without looking at the interface's
>>> implementation.
>>
>> Right. But there won't be any third-party callers of project-try-vc,
>> this function's only purpose is to be inside project-find-functions.
> 
> I'm thinking about additional "authors" who'd like to add
> functionality to existing project backends.

They won't call it either. If they do, the function is likely to have 
changed significantly from its current state.

>>> If it is necessary to consult the implementation,
>>> that is an indication of some deficiency in the docs, and we should
>>> try to avoid that as much as possible.
>>
>> No disagreement there, as long as we're talking about public functions.
> 
> Are we still under the rule that any function without 2 dashes in its
> name is public?  If so, then I think we have only discussed public
> functions in this and related threads.

OK, I have a question then. Does every built-in member of a public hook 
need to be public?

Re: Opaque objects and Emacs documentation

[Eli Zaretskii]

> Cc: rms@gnu.org, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Wed, 22 Jul 2020 00:31:04 +0300
> 
> >> If you say that, could you give an example of something that *would* be
> >> an impediment to extensibility?
> > 
> > What you said: describing what generics returns, or what
> > project-current returns in general.
> 
> No, describing "in general" is good. What is bad is describing "in 
> particular" when a function is "general" and can return values that are 
> different from the examples. project-current is "general".

That's because "general" is overloaded, and you understand it in a
sense different from the one in which I used it.  Just drop "in
general" from what I wrote, and you will have a much better
approximation to what I meant.

> And speaking of "transient", it's not helpful to say it returns a cons 
> (transient . root) because that doesn't say anything about the project 
> behavior anyway (which is the important part).

A backend that receives such an object will need to be prepared for
it.

> > If those are the rules of the game, yes.  IOW, if it's not okay to
> > describe the possible forms of the object in the doc string of
> > project-find-functions, but okay to describe them in the individual
> > doc strings of each hook that can be put there, then it could be an
> > acceptable compromise, at least from my POV.
> 
> Very well. The "transient" project is not on the hook, though.

Yes, but the code which returns it is a kind of "default
implementation" for bootstrapping projects.  So it is definitely in
the same class of objects as the vc project.

> >> Right. But there won't be any third-party callers of project-try-vc,
> >> this function's only purpose is to be inside project-find-functions.
> > 
> > I'm thinking about additional "authors" who'd like to add
> > functionality to existing project backends.
> 
> They won't call it either. If they do, the function is likely to have 
> changed significantly from its current state.

Even if they don't call it (and I'm unconvinced), they will need to
deal with the return value, so some documentation about it will be
useful.

> >> No disagreement there, as long as we're talking about public functions.
> > 
> > Are we still under the rule that any function without 2 dashes in its
> > name is public?  If so, then I think we have only discussed public
> > functions in this and related threads.
> 
> OK, I have a question then. Does every built-in member of a public hook 
> need to be public?

The main point here is that the particular types of objects the
existing backends produce should be documented somewhere.

Re: Opaque objects and Emacs documentation

[Dmitry Gutov]

On 22.07.2020 17:28, Eli Zaretskii wrote:

>>>> If you say that, could you give an example of something that *would* be
>>>> an impediment to extensibility?
>>>
>>> What you said: describing what generics returns, or what
>>> project-current returns in general.
>>
>> No, describing "in general" is good. What is bad is describing "in
>> particular" when a function is "general" and can return values that are
>> different from the examples. project-current is "general".
> 
> That's because "general" is overloaded, and you understand it in a
> sense different from the one in which I used it.  Just drop "in
> general" from what I wrote, and you will have a much better
> approximation to what I meant.

If I drop both instances of the word "general" from the sentence I 
quoted, there will be nothing left.

Perhaps you can read my description and learn something new from it?

>> And speaking of "transient", it's not helpful to say it returns a cons
>> (transient . root) because that doesn't say anything about the project
>> behavior anyway (which is the important part).
> 
> A backend that receives such an object will need to be prepared for
> it.

The backend returned that value, so it is surely prepared.

>>> If those are the rules of the game, yes.  IOW, if it's not okay to
>>> describe the possible forms of the object in the doc string of
>>> project-find-functions, but okay to describe them in the individual
>>> doc strings of each hook that can be put there, then it could be an
>>> acceptable compromise, at least from my POV.
>>
>> Very well. The "transient" project is not on the hook, though.
> 
> Yes, but the code which returns it is a kind of "default
> implementation" for bootstrapping projects.  So it is definitely in
> the same class of objects as the vc project.

Yes. If the same class as "shouldn't be documented in too much detail in 
project-current or project-find-functions".

>>>> Right. But there won't be any third-party callers of project-try-vc,
>>>> this function's only purpose is to be inside project-find-functions.
>>>
>>> I'm thinking about additional "authors" who'd like to add
>>> functionality to existing project backends.
>>
>> They won't call it either. If they do, the function is likely to have
>> changed significantly from its current state.
> 
> Even if they don't call it (and I'm unconvinced), they will need to
> deal with the return value, so some documentation about it will be
> useful.

The function ends with (cons 'vc root).

Is there really much uncertainty about what kind of value is returns?

>>>> No disagreement there, as long as we're talking about public functions.
>>>
>>> Are we still under the rule that any function without 2 dashes in its
>>> name is public?  If so, then I think we have only discussed public
>>> functions in this and related threads.
>>
>> OK, I have a question then. Does every built-in member of a public hook
>> need to be public?
> 
> The main point here is that the particular types of objects the
> existing backends produce should be documented somewhere.

Check out 'M-x describe-function RET project-root RET', the 
"Implementations:" section.

Re: Opaque objects and Emacs documentation

[Eli Zaretskii]

> Cc: rms@gnu.org, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Wed, 22 Jul 2020 18:44:18 +0300
> 
> >>> What you said: describing what generics returns, or what
> >>> project-current returns in general.
> >>
> >> No, describing "in general" is good. What is bad is describing "in
> >> particular" when a function is "general" and can return values that are
> >> different from the examples. project-current is "general".
> > 
> > That's because "general" is overloaded, and you understand it in a
> > sense different from the one in which I used it.  Just drop "in
> > general" from what I wrote, and you will have a much better
> > approximation to what I meant.
> 
> If I drop both instances of the word "general" from the sentence I 
> quoted, there will be nothing left.

In _my_ sentence above there's only one "general".  the other one is
"generics", a different word.

> Perhaps you can read my description and learn something new from it?

I thought we could keep this discussion civilized for a change?

> >> And speaking of "transient", it's not helpful to say it returns a cons
> >> (transient . root) because that doesn't say anything about the project
> >> behavior anyway (which is the important part).
> > 
> > A backend that receives such an object will need to be prepared for
> > it.
> 
> The backend returned that value, so it is surely prepared.

No, it's project-current that did.

> > Even if they don't call it (and I'm unconvinced), they will need to
> > deal with the return value, so some documentation about it will be
> > useful.
> 
> The function ends with (cons 'vc root).
> 
> Is there really much uncertainty about what kind of value is returns?

Didn't we agree that understanding what an interface does shouldn't
require reading the code?

Anyway, it sounds like this discussion again veered in directions I
don't want to follow, so I will stop here.

Re: Opaque objects and Emacs documentation

[Dmitry Gutov]

On 22.07.2020 19:30, Eli Zaretskii wrote:

>> If I drop both instances of the word "general" from the sentence I
>> quoted, there will be nothing left.
> 
> In _my_ sentence above there's only one "general".  the other one is
> "generics", a different word.

Same root, similar purpose.

Describing what generics return is fine, the project-current's docstring 
already does that, actually. The issue is in the manner of the 
description. You insist on describing the data structure (as opposed to 
describing associated behaviors), but the callers of that function 
should never need it.

>>>> And speaking of "transient", it's not helpful to say it returns a cons
>>>> (transient . root) because that doesn't say anything about the project
>>>> behavior anyway (which is the important part).
>>>
>>> A backend that receives such an object will need to be prepared for
>>> it.
>>
>> The backend returned that value, so it is surely prepared.
> 
> No, it's project-current that did.

project-current received it from the backend and returned it to the caller.

Re: Opaque objects and Emacs documentation

[Alan Mackenzie]

Hello, Dmitry.

On Tue, Jul 21, 2020 at 04:09:18 +0300, Dmitry Gutov wrote:
> On 20.07.2020 05:44, Richard Stallman wrote:
> > [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> > [[[ whether defending the US Constitution against all enemies,     ]]]
> > [[[ foreign or domestic, requires you to follow Snowden's example. ]]]

> >    > Consider the docstring of said constructor function. What should it say
> >    > its return value is?

> > One natural possibility:

> >    Return an empty project object.  You need to store values in it with ...

> > Another:

> >    Return a project object with name MAME and containing files FILES.

> > I am guessing here.  I don't know what these projects do.
> > I don't know what data a project object should have.

> Okay, see, neither option is going to work.

> We have a "factory" function (called 'project-current') which delegates 
> project instance creation to a hook. Which is what makes it extensible. 
> The only argument is DIRECTORY, meaning the current directory. The 
> functions in the hook get called in turn, and the first one that returns 
> non-nil is going to be used. Its return value must be a "project 
> instance" which satisfies a certain protocol defined by several 
> cl-generic methods (think "interface" in Java, or "typeclass" in 
> Haskell, or "protocol" in Clojure, or multimethods elsewhere). One 
> method is mandatory to be defined for all project implementations, the 
> rest have default implementations.

> Thanks to the cl-generic infrastructure, the datatype representing the 
> current project can be almost anything, and the structure of the fields 
> inside can be arbitrary.

You seem to be writing that as though it were a good thing.  From the
perspective of somebody who's likely to want to debug things like this in
the future, it sounds like a Bad Thing indeed.  What precisely forces you
to use such difficult and arcane facilities?  What does it gain you that
you couldn't get out of plain Emacs Lisp (without things like
cl-generic)?

> This is sorted out by each individual backend which knows how to access
> fields inside "project instances" belonging to it, and which implements
> the aforementioned protocol for that instance type.

Trouble is, when I'm trying to debug a problem involving such code, that
style of coding adds an order of magnitude onto the time and effort
needed.

> The two built-in backends use very simple structures (right now: conses 
> like (TYPE-TAG . PROJECT-ROOT)), which is very obvious from their 
> implementations. The core of the dispute is whether project-current's 
> docstring should document the return value in detail (giving an example 
> of built-in project instances's structure). Eli is insisting on it 
> because (AFAICT) that suits his thought process and he thinks that will 
> help users/clients to make sense of how things work.

if it is not documented in detail, it will add another order of magnitude
onto the time to debug it; anybody needing to know the structure of a
return value in detail, possibly to check it's valid, would first have to
read and understand the intricate details of the source code, and
effectively recreate in her private notes the details missing from the
doc string.

That isn't fun.  Not so long ago I gave up on an Edebug bug because the
proliferation of inadequately documented cl- functions in edebug.el just
made it too tedious to continue.

> I am adamantly against this because it's against the design (clients 
> should program against the public interface, not against the 
> implementation), and is likely to encourage incorrect programs.

That's known as "security by obscurity", I think.  I think your fear is
misfounded.  For a start, if the public interface is adequate, people
will use it in preference to dissecting internal structures.  When gaps
are found in that interface, people will use the internal details to
craft a workaround whether the documentation is there or not.  (I have
done this at least once in the past.)

> Of course, we can add this information elsewhere, e.g. in the comments 
> of the "backend" functions (which are the elements of the hook I 
> mentioned above). I haven't done so because, well, the implementations 
> are fairly transparent and the structures are trivial. But I wouldn't 
> object against somebody doing that.

I think that is the job of the package's author/maintainer.

>  > It is important to document the meaning of each slot -- that may not
>  > be obvious.  Which values are valid?

> The space of possible structures of valid project instances is 
> theoretically infinite.

It remains important to document that infinite space.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Opaque objects and Emacs documentation

[Dmitry Gutov]

Hi Alan,

On 21.07.2020 22:36, Alan Mackenzie wrote:

> What precisely forces you
> to use such difficult and arcane facilities?  What does it gain you that
> you couldn't get out of plain Emacs Lisp (without things like
> cl-generic)?

Sorry, could you just read this thread? Or the description of the package.

As I said, the closest alternative would be using functions with 
OO-style dispatch arguments, akin to completion tables. I don't think 
you'd like that option better.

>> This is sorted out by each individual backend which knows how to access
>> fields inside "project instances" belonging to it, and which implements
>> the aforementioned protocol for that instance type.
> 
> Trouble is, when I'm trying to debug a problem involving such code, that
> style of coding adds an order of magnitude onto the time and effort
> needed.

Sometimes, when we need more advanced capabilities for our programs, 
that can indeed result in less trivial debugging. The alternative would 
be not writing said program at all.

> if it is not documented in detail, it will add another order of magnitude
> onto the time to debug it; anybody needing to know the structure of a
> return value in detail, possibly to check it's valid, would first have to
> read and understand the intricate details of the source code, and
> effectively recreate in her private notes the details missing from the
> doc string.

You can only know it's valid by looking at the "Implementations:" list 
in 'M-x describe-function RET project-root RET'.

You won't know it is valid just by reading some examples of return values.

> That isn't fun.  Not so long ago I gave up on an Edebug bug because the
> proliferation of inadequately documented cl- functions in edebug.el just
> made it too tedious to continue.

Edebug is difficult. So far I have no idea why that is: perhaps because 
of the coding style, but more likely just due to the problem domain.

But we indeed have old code that is unnecessarily difficult. At some 
point, I had a lot of trouble following the code in etags.el. Lots of 
implicit state and dynamic variables everywhere.

Eli did some cleanups after that, but I'm yet to form an opinion on how 
much they changed things.

>> I am adamantly against this because it's against the design (clients
>> should program against the public interface, not against the
>> implementation), and is likely to encourage incorrect programs.
> 
> That's known as "security by obscurity", I think.

No, abstraction and security are largely orthogonal.

> I think your fear is
> misfounded.  For a start, if the public interface is adequate, people
> will use it in preference to dissecting internal structures.

That means documenting the public interface adequately first. And for 
that, the description needs to be abstraction-oriented.

> When gaps
> are found in that interface, people will use the internal details to
> craft a workaround whether the documentation is there or not.  (I have
> done this at least once in the past.)

Either inadequate public interface, or inadequate documentation, or 
programmer laziness (or "cowboy-ism"). I wouldn't presume to say which 
was the cause more, but I've seen this in practice pertaining to the 
package under discussion.

Re: Opaque objects and Emacs documentation

[Eli Zaretskii]

> Cc: rms@gnu.org, eliz@gnu.org, npostavs@gmail.com, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Wed, 22 Jul 2020 00:17:18 +0300
> 
> The alternative would be not writing said program at all.

Technically, there's another alternative: implement said programs via
different design patterns.  But that's a separate discussion.

Re: Opaque objects and Emacs documentation

[Dmitry Gutov]

On 22.07.2020 17:31, Eli Zaretskii wrote:
>> The alternative would be not writing said program at all.
> Technically, there's another alternative: implement said programs via
> different design patterns.  But that's a separate discussion.

How about we avoid unsubstantiated statements here?

Re: Opaque objects and Emacs documentation

[Eli Zaretskii]

> Cc: acm@muc.de, rms@gnu.org, npostavs@gmail.com, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Wed, 22 Jul 2020 18:33:55 +0300
> 
> On 22.07.2020 17:31, Eli Zaretskii wrote:
> >> The alternative would be not writing said program at all.
> > Technically, there's another alternative: implement said programs via
> > different design patterns.  But that's a separate discussion.
> 
> How about we avoid unsubstantiated statements here?

Which ones?

Re: Opaque objects and Emacs documentation

[Dmitry Gutov]

On 22.07.2020 19:22, Eli Zaretskii wrote:
>> Cc:acm@muc.de,rms@gnu.org,npostavs@gmail.com,emacs-devel@gnu.org
>> From: Dmitry Gutov<dgutov@yandex.ru>
>> Date: Wed, 22 Jul 2020 18:33:55 +0300
>>
>> On 22.07.2020 17:31, Eli Zaretskii wrote:
>>>> The alternative would be not writing said program at all.
>>> Technically, there's another alternative: implement said programs via
>>> different design patterns.  But that's a separate discussion.
>> How about we avoid unsubstantiated statements here?
> Which ones?

Your message above basically said that my statement was invalid. Without 
any supporting arguments. And you even cut it out of context.

Please stop that.

And as for a "separate discussion", I already invited you to have it (to 
try to actually prove your opinion there), and you bowed out.

Re: Opaque objects and Emacs documentation

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Its return value must be a "project 
  > instance" which satisfies a certain protocol defined by several 
  > cl-generic methods (think "interface" in Java, or "typeclass" in 
  > Haskell, or "protocol" in Clojure, or multimethods elsewhere).

This protocol of operations and what jobs they are supposed to do
is what defines he project instance data structure.  So that is
a good way to document the data structure.

However, I wonder if there are some data that we can expect every
project instace to have..  For instance, does every project instace
have a name?  Is there an operation to ask for its name?

What are the simple data items that every project instance should
have?

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Opaque objects and Emacs documentation

[Dmitry Gutov]

On 23.07.2020 07:04, Richard Stallman wrote:
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
> 
>    > Its return value must be a "project
>    > instance" which satisfies a certain protocol defined by several
>    > cl-generic methods (think "interface" in Java, or "typeclass" in
>    > Haskell, or "protocol" in Clojure, or multimethods elsewhere).
> 
> This protocol of operations and what jobs they are supposed to do
> is what defines he project instance data structure.  So that is
> a good way to document the data structure.
> 
> However, I wonder if there are some data that we can expect every
> project instace to have..  For instance, does every project instace
> have a name?  Is there an operation to ask for its name?

The root directory is one such point of data. So the project-root method 
must be defined (it's an operation, part of the protocol).

Currently it serves the role of "project name" as well. We might add a 
new, separate method for it later, but it will be optional.

> What are the simple data items that every project instance should
> have?

So far this is it. The rest of the methods have default implementations.

Re: Opaque objects and Emacs documentation

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 1961 bytes --]

On Sun, Jul 19, 2020, 23:02 Dmitry Gutov <dgutov@yandex.ru> wrote:

> On 18.07.2020 04:54, Richard Stallman wrote:
> > We define an accessor defsubst for each of the data it contains.  We
> > document its meaning.  We define a constructor function, and document
> > the meaning of each argument.  We manipulate them always through those
> > functions.  Voilà!
>
> Here's a question that will perhaps help us solve this dispute.
>
> Consider the docstring of said constructor function. What should it say
> its return value is?
>

IMHO, a most reasonable way to answer this question, for any function, but
particularly for generic ones:

"Return an object which one can pass to/responds to 'foo' and 'bar', which
see".

It's pretty helpful already. If you can add things to be more helpful, like
mentioning a type (which may be just a deftype) you should, but oftentimes
duck typing and generic functions means you're better off talking in terms
of operations.

In the docstring of methods of the generic functions you can be more
specific as to how the object returned responds to foo and bar.

This is fundamentally different from the  Java OO paradigm. In Common Lisp,
function is king.

But generic functions are not at all the only way to successfully design
abstractions in CL. IME of about 15 years, they're good because they mix
well with defclass via method combinations (of which Emacs's version of
generics only has the one). Method combinations "understand" the object
hierarchy. When you're not using them with defclass, they become so-so, on
par or sometimes even slightly behind many other techniques.

As a very rough rule of thumb, if you're not using 'call-next-method'
anywhere, or don't see a good reason why, you've probably overengineered it.

PS: i'm just replying to this bit of context, no idea how it fits in the
general discussion and whose position this helps, if anyone's...

João

>

[-- Attachment #2: Type: text/html, Size: 2900 bytes --]

Re: Opaque objects and Emacs documentation

[Dmitry Gutov]

On 17.07.2020 15:02, Noam Postavsky wrote:
>> ??? You are saying that the_only_  way to design project.el is to use
>> generics at that high level?
> That sounds like a bit of a misreading: Dmitry didn't say "_only_
> way", he said "there is no better" way.

I even gave an example of an alternative approach that we would have 
taken without cl-generic. And how it's no better.

Re: Opaque objects and Emacs documentation

[Dmitry Gutov]

On 17.07.2020 14:14, Eli Zaretskii wrote:
>> Cc: emacs-devel@gnu.org
>> From: Dmitry Gutov <dgutov@yandex.ru>
>> Date: Fri, 17 Jul 2020 13:53:35 +0300
>>
>> It shouldn't come as a surprise that I think there is no better
>> technological choice. Otherwise I would have used it.
> 
> ??? You are saying that the _only_ way to design project.el is to use
> generics at that high level?  Just because sometimes a "project
> instance" is a cons cell and sometimes some other Lisp structure?  I'm
> surprised to hear that.

This logic is backwards. The generics are here because we make good use 
of them.

The use of cons cells or some other structures, is a choice, basically 
arbitrary, that can change at will. That is another reason why I don't 
like to see the low-level description in the docstring of project-current.

>> What should help is coming up with a better, consistent way to document
>> these things.
> 
> I tried to find one, but couldn't.  If someone wants to propose a way,
> I'm all ears.  I wrote those comments which started this thread
> because it surprised me how difficult it was to try to keep the
> limitations you impose on what should and what shouldn't be divulged,
> and yet produce useful documentation of what the various public
> functions do.  In all my 40 years of experience with Emacs, I have
> never bumped into such a brick wall, with no way around it.

I'm saying it's nothing new: completion tables, for instance, have been 
quite as abstract.

But for some reason you are fine with docstrings that say "ARG is a 
completion table", or "returns a completion table"?

> Later it became apparent that you are doing this on purpose, because
> you have decided that certain directions of developing the package
> should be made as hard as possible,

That is an unfair misstatement of my words.

> something that I think is
> completely against the spirit of Emacs development.  Readers of this
> thread are invited to read
> 
>    https://lists.gnu.org/archive/html/emacs-devel/2020-07/msg00288.html
> 
> especially under "Here are things we want third-party code to be able
> to do" and "Here are, on the other hand, things that people generally
> shouldn't do".  Again, I'm interested to know how many of us here
> share those views.

Those are not examples of "developing the package". Those are examples 
of using it. Some of them - incorrectly.

>>> Which I think is against long-time Emacs tradition for documenting its
>>> interfaces, and by that facilitating extensibility.  It is IMO wrong
>>> to fill Emacs application levels with opaque objects which cannot be
>>> usefully described; they should be a rare exception, but definitely
>>> not the rule.
>>
>> The problem here is that even if you document a particular value, it's
>> _not useful_. It doesn't show you what you can or should do with it.
> 
> It was very useful for me, and so I presume it could be useful for
> others.

It has been useful to you to find an answer to a minor question: "how 
projects are compared, which projects are equal?". Which is not 
something most people will even think about.

And even answering that question for the project-vc case doesn't give 
you a general answer. I don't see how you are content with only having 
that answer for a special case anyway.

> Even if you think it is not useful for you, the fact that
> fellow developers tell you the contrary should be a reason to revisit
> your views, and maybe allow for other views as well, even if you
> disagree.

Perhaps if someone else said "I wanted to do a (valid thing X), couldn't 
understand how to do it, and this piece of information would have made 
it easier"?

No such declarations so far.

> Documentation should strive to serve different ways of studying and
> developing Emacs, not just a single way that you personally think is
> The (only) Right Thing.

I already said you can add code comments. Preferably somewhere which is 
not the main entry point of the API.

I also talked about the possibility to have this documented in the 
manual, etc (in a chapter targeted at developers).

You have skipped and dismissed it all, and now make contrived accusations.

>> The main thing thing the user can do with that value is misuse it, by
>> relying on its shape in the client code.
> 
> Once again: concealing information because someone could be silly
> enough to misuse it punishes many valid uses on behalf of a few
> invalid ones.

Which valid uses, though?

> We should treat our users as responsible adults, even
> if some of them aren't.  Those who aren't will eventually be amply
> punished, or will recognize their mistakes and get their act together.

Our users are not all "responsible adults", or to put it differently, 
are not all professional programmers. Even even those who are, are at 
very different levels of skill.

And even when writing documentation for professional programmers, it is 
always considered a good taste to structure it so that it encourages 
writing good code, and discourages writing bad one.

Re: Opaque objects and Emacs documentation

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Fri, 17 Jul 2020 16:42:55 +0300
> 
> >> It shouldn't come as a surprise that I think there is no better
> >> technological choice. Otherwise I would have used it.
> > 
> > ??? You are saying that the _only_ way to design project.el is to use
> > generics at that high level?  Just because sometimes a "project
> > instance" is a cons cell and sometimes some other Lisp structure?  I'm
> > surprised to hear that.
> 
> This logic is backwards. The generics are here because we make good use 
> of them.

That wasn't the issue.  The issue is whether we should use these
techniques in a way that makes it hard or impossible to document our
functions and commands.  Even if technically this is the best design
(and I don't yet see why this should be so, nor see any arguments
presented to justify that), we may decide that the price is too heavy.

> The use of cons cells or some other structures, is a choice, basically 
> arbitrary, that can change at will. That is another reason why I don't 
> like to see the low-level description in the docstring of project-current.

As was already written several times, internal implementation details
can be documented without any real impediment of future development.
So this argument is invalid.

> > I tried to find one, but couldn't.  If someone wants to propose a way,
> > I'm all ears.  I wrote those comments which started this thread
> > because it surprised me how difficult it was to try to keep the
> > limitations you impose on what should and what shouldn't be divulged,
> > and yet produce useful documentation of what the various public
> > functions do.  In all my 40 years of experience with Emacs, I have
> > never bumped into such a brick wall, with no way around it.
> 
> I'm saying it's nothing new: completion tables, for instance, have been 
> quite as abstract.

That one bad example exist doesn't mean we want others.

> But for some reason you are fine with docstrings that say "ARG is a 
> completion table", or "returns a completion table"?

Who said I was fine with them?  Rome wasn't built in a day.

> > Later it became apparent that you are doing this on purpose, because
> > you have decided that certain directions of developing the package
> > should be made as hard as possible,
> 
> That is an unfair misstatement of my words.

I provided the URL for your words, so that everyone will be able to
make their own minds, in case I've indeed misstated them.

> >    https://lists.gnu.org/archive/html/emacs-devel/2020-07/msg00288.html
> > 
> > especially under "Here are things we want third-party code to be able
> > to do" and "Here are, on the other hand, things that people generally
> > shouldn't do".  Again, I'm interested to know how many of us here
> > share those views.
> 
> Those are not examples of "developing the package". Those are examples 
> of using it.

Consuming the package's API and adding backends is development of
project.el.  (It is also "using" it, but not on the user level, so
saying that is "using" just muddies the waters, but doesn't affect the
main issue in any way.)

> >> The problem here is that even if you document a particular value, it's
> >> _not useful_. It doesn't show you what you can or should do with it.
> > 
> > It was very useful for me, and so I presume it could be useful for
> > others.
> 
> It has been useful to you to find an answer to a minor question: "how 
> projects are compared, which projects are equal?". Which is not 
> something most people will even think about.

Why not?  Of course they will.

Besides, the question about what exactly is a "project instance" is
relevant in many places in the package, just look how many doc strings
mention that phrase.

> And even answering that question for the project-vc case doesn't give 
> you a general answer. I don't see how you are content with only having 
> that answer for a special case anyway.

That's the only case that we have for now, so it's highly relevant and
useful.

> > Even if you think it is not useful for you, the fact that
> > fellow developers tell you the contrary should be a reason to revisit
> > your views, and maybe allow for other views as well, even if you
> > disagree.
> 
> Perhaps if someone else said "I wanted to do a (valid thing X), couldn't 
> understand how to do it, and this piece of information would have made 
> it easier"?
> 
> No such declarations so far.

Since when we write documentation only when someone asks a question?
Documentation is supposed to be proactive, it should answer the
important questions before they are asked.

> > Documentation should strive to serve different ways of studying and
> > developing Emacs, not just a single way that you personally think is
> > The (only) Right Thing.
> 
> I already said you can add code comments. Preferably somewhere which is 
> not the main entry point of the API.

I'm sure you understand how low is my motivation to do any work on
documenting project.el.  Look what happened last time: I improved
several doc strings, and suddenly I have a dispute that lasts for a
month, and ends up accusing me in all the sins on Earth, including
that I cause contributors to flee and don't understand the code I'm
reading.  I'm only human, and have a busy schedule; unpleasant jobs
get pushed back and delayed.

> > Once again: concealing information because someone could be silly
> > enough to misuse it punishes many valid uses on behalf of a few
> > invalid ones.
> 
> Which valid uses, though?

Any and all of them.

> > We should treat our users as responsible adults, even
> > if some of them aren't.  Those who aren't will eventually be amply
> > punished, or will recognize their mistakes and get their act together.
> 
> Our users are not all "responsible adults", or to put it differently, 
> are not all professional programmers. Even even those who are, are at 
> very different levels of skill.

That's exactly the crux of our disagreement, right there.  I don't
agree with such paternalistic views of our contributors and users.

> And even when writing documentation for professional programmers, it is 
> always considered a good taste to structure it so that it encourages 
> writing good code, and discourages writing bad one.

We encourage writing good code, but not through hiding important
information.

Re: Opaque objects and Emacs documentation

[Dmitry Gutov]

On 17.07.2020 17:22, Eli Zaretskii wrote:

> That wasn't the issue.  The issue is whether we should use these
> techniques in a way that makes it hard or impossible to document our
> functions and commands.  Even if technically this is the best design
> (and I don't yet see why this should be so, nor see any arguments
> presented to justify that), we may decide that the price is too heavy.

I can hardly understand how we might choose to reject the possibility to 
implement certain useful features only because return values of some 
functions become more difficult to document.

>> The use of cons cells or some other structures, is a choice, basically
>> arbitrary, that can change at will. That is another reason why I don't
>> like to see the low-level description in the docstring of project-current.
> 
> As was already written several times, internal implementation details
> can be documented without any real impediment of future development.

In an appropriate place. Say, the "internals" documentation, wherever it 
may reside.

> So this argument is invalid.

Of course.

>>> I tried to find one, but couldn't.  If someone wants to propose a way,
>>> I'm all ears.  I wrote those comments which started this thread
>>> because it surprised me how difficult it was to try to keep the
>>> limitations you impose on what should and what shouldn't be divulged,
>>> and yet produce useful documentation of what the various public
>>> functions do.  In all my 40 years of experience with Emacs, I have
>>> never bumped into such a brick wall, with no way around it.
>>
>> I'm saying it's nothing new: completion tables, for instance, have been
>> quite as abstract.
> 
> That one bad example exist doesn't mean we want others.

You seem to be on a crusade against abstraction. Never mind that it is 
the only way to provide features with strong extensibility.

>> But for some reason you are fine with docstrings that say "ARG is a
>> completion table", or "returns a completion table"?
> 
> Who said I was fine with them?  Rome wasn't built in a day.

And Carthage must be destroyed, apparently.

>> Those are not examples of "developing the package". Those are examples
>> of using it.
> 
> Consuming the package's API and adding backends is development of
> project.el. 

No, it isn't. It's not development of the API, or of the built-in backends.

> (It is also "using" it, but not on the user level, so
> saying that is "using" just muddies the waters, but doesn't affect the
> main issue in any way.)

If the presence of user-level commands inside project.el confuses you, 
we can split them into a different file.

>>>> The problem here is that even if you document a particular value, it's
>>>> _not useful_. It doesn't show you what you can or should do with it.
>>>
>>> It was very useful for me, and so I presume it could be useful for
>>> others.
>>
>> It has been useful to you to find an answer to a minor question: "how
>> projects are compared, which projects are equal?". Which is not
>> something most people will even think about.
> 
> Why not?  Of course they will.

That has never come up in several years.

> Besides, the question about what exactly is a "project instance" is
> relevant in many places in the package, just look how many doc strings
> mention that phrase.

Saying that is a cons (and even giving an example), does nothing to 
address that question. This seems to be the main issue you fail to 
understand here.

>> And even answering that question for the project-vc case doesn't give
>> you a general answer. I don't see how you are content with only having
>> that answer for a special case anyway.
> 
> That's the only case that we have for now, so it's highly relevant and
> useful.

Two built-in cases. One case in Stefan's backend which you have never 
seen. Some backends out there, either existing or future ones.

But you need to be able to reason about all of them.

>> Perhaps if someone else said "I wanted to do a (valid thing X), couldn't
>> understand how to do it, and this piece of information would have made
>> it easier"?
>>
>> No such declarations so far.
> 
> Since when we write documentation only when someone asks a question?

You are refuting something I haven't said.

> Documentation is supposed to be proactive, it should answer the
> important questions before they are asked.

Exactly. The important ones.

>>> Documentation should strive to serve different ways of studying and
>>> developing Emacs, not just a single way that you personally think is
>>> The (only) Right Thing.
>>
>> I already said you can add code comments. Preferably somewhere which is
>> not the main entry point of the API.
> 
> I'm sure you understand how low is my motivation to do any work on
> documenting project.el.

Whatever your motivation is, saying I object to any and all 
documentation is false.

> Look what happened last time: I improved
> several doc strings, and suddenly I have a dispute that lasts for a
> month,

I accepted your multiple edits, and changed a couple of functions and 
descriptions myself to better support your personal goals.

And I removed a couple of sentences you wrote. Which you have been 
unable to leave alone, or accept my perspective.

> and ends up accusing me in all the sins on Earth, including
> that I cause contributors to flee and don't understand the code I'm
> reading.  I'm only human, and have a busy schedule; unpleasant jobs
> get pushed back and delayed.

There is no hurry.

>>> Once again: concealing information because someone could be silly
>>> enough to misuse it punishes many valid uses on behalf of a few
>>> invalid ones.
>>
>> Which valid uses, though?
> 
> Any and all of them.

That's vague.

>> And even when writing documentation for professional programmers, it is
>> always considered a good taste to structure it so that it encourages
>> writing good code, and discourages writing bad one.
> 
> We encourage writing good code, but not through hiding important
> information.

Having private/undocumented functions and variables is a common practice 
even in our project.

Re: Opaque objects and Emacs documentation

[John Yates]

[-- Attachment #1: Type: text/plain, Size: 1784 bytes --]

On Fri, Jul 17, 2020 at 7:14 AM Eli Zaretskii <eliz@gnu.org> wrote:
> Later it became apparent that you are doing this on purpose, because
> you have decided that certain directions of developing the package
> should be made as hard as possible, something that I think is
> completely against the spirit of Emacs development.  Readers of this
> thread are invited to read
>
>   https://lists.gnu.org/archive/html/emacs-devel/2020-07/msg00288.html
>
> especially under "Here are things we want third-party code to be able
> to do" and "Here are, on the other hand, things that people generally
> shouldn't do".  Again, I'm interested to know how many of us here
> share those views.
I have used emacs for nearly 30 years but have rarely ventured into elisp
development much
beyond maintaining my .emacs.  So I look to emacs documentation as a user,
not a developer.

OTOH I have been programming for just shy of 50 years.  Over that time I
have continually
increased the size and scope of projects I could tackle primarily by
looking for ways to
build and compose ever better abstractions.  Sometimes that has taken the
form of adopting
a new programming language, sometimes a new methodology or disciple.

Conversely, one of the banes of my existence has been leaky abstractions.
For better
or worse, based maining on the untyped nature of lisp and its culture of
describing how
to interpret structures of various shapes, my overriding impression of the
lisp world in
general and of elisp in particular is that they are rife with leaky
abstractions.

I commend Dmitry's desire to provide an opaque abstraction for a project.
Sadly, in
this instance I suspect that such an effort is an attempt to swim against
an impossibly
strong tide.  (If you cannot beat them, join them :-)

/john

[-- Attachment #2: Type: text/html, Size: 2442 bytes --]

RE: Opaque objects and Emacs documentation

[Drew Adams]

> ... looking for ways to build and compose ever better
> abstractions.  Sometimes that has taken the form of
> adopting a new programming language, sometimes a new
> methodology or disciple.

> ... one of the banes of my existence has been leaky
> abstractions.  For better or worse, based main[ly] on
> the untyped nature of lisp and its culture of
> describing how to interpret structures of various
> shapes, my overriding impression of the lisp world in
> general and of elisp in particular is that they are
> rife with leaky abstractions.

FWIW, I can sympathize with that.  I don't think anyone
is arguing for less abstraction or in favor of leaky
abstractions.  And I agree about the problems you cite.

Lisp is not Haskell, for sure.  It does provide constructs
for building abstractions, notably higher-order functions
and Lisp macros.  But because it's imperative/procedural,
and in particular it's used/intended for operating on code
as data, it's less amenable to declarative specification
and static reasoning.

Practical Lisp (as opposed to a purely functional Lisp)
is a dynamic language that lets you reach all the way down
to bits and active procedures, including ones that are
doing the reaching.  It lets you examine your own guts,
and even operate on them without anesthesia.  Yikes!

For better and worse, that's Lisp.

> I commend Dmitry's desire to provide an opaque
> abstraction for a project.

I have nothing to say about that particular attempt or
that particular project.  My comments are about the
general question of documenting code, for both users
and developers.

But again, I think the abstraction / no-abstraction
argument is a false dichotomy here.

Re: Opaque objects and Emacs documentation

[Eli Zaretskii]

> From: John Yates <john@yates-sheets.org>
> Date: Fri, 17 Jul 2020 12:56:52 -0400
> Cc: Emacs developers <emacs-devel@gnu.org>, Dmitry Gutov <dgutov@yandex.ru>
> 
> I have used emacs for nearly 30 years but have rarely ventured into elisp development much
> beyond maintaining my .emacs.  So I look to emacs documentation as a user, not a developer.
> 
> OTOH I have been programming for just shy of 50 years.  Over that time I have continually
> increased the size and scope of projects I could tackle primarily by looking for ways to
> build and compose ever better abstractions.  Sometimes that has taken the form of adopting
> a new programming language, sometimes a new methodology or disciple.
> 
> Conversely, one of the banes of my existence has been leaky abstractions.  For better
> or worse, based maining on the untyped nature of lisp and its culture of describing how
> to interpret structures of various shapes, my overriding impression of the lisp world in
> general and of elisp in particular is that they are rife with leaky abstractions.
> 
> I commend Dmitry's desire to provide an opaque abstraction for a project.  Sadly, in
> this instance I suspect that such an effort is an attempt to swim against an impossibly
> strong tide.  (If you cannot beat them, join them :-)

The issue at hand is not whether or not to use the abstractions.
Rather, it is how to use abstractions in a way that doesn't prevent
good documentation.

Re: Opaque objects and Emacs documentation

[Dmitry Gutov]

On 17.07.2020 22:04, Eli Zaretskii wrote:
> The issue at hand is not whether or not to use the abstractions.
> Rather, it is how to use abstractions in a way that doesn't prevent
> good documentation.

Or: how to document said abstractions better.

Re: Opaque objects and Emacs documentation

[Dmitry Gutov]

On 17.07.2020 19:56, John Yates wrote:

> OTOH I have been programming for just shy of 50 years.  Over that time I 
> have continually
> increased the size and scope of projects I could tackle primarily by 
> looking for ways to
> build and compose ever better abstractions.  Sometimes that has taken 
> the form of adopting
> a new programming language, sometimes a new methodology or disciple.
> 
> Conversely, one of the banes of my existence has been leaky 
> abstractions.  For better
> or worse, based maining on the untyped nature of lisp and its culture of 
> describing how
> to interpret structures of various shapes, my overriding impression of 
> the lisp world in
> general and of elisp in particular is that they are rife with leaky 
> abstractions.
> 
> I commend Dmitry's desire to provide an opaque abstraction for a 
> project.

Thanks, John.

> Sadly, in
> this instance I suspect that such an effort is an attempt to swim 
> against an impossibly
> strong tide.  (If you cannot beat them, join them :-)

I don't know if it's true.

project.el has seen a certain level of adoption for the last few years, 
and some very nice contributions recently. All without using the 
approach to documentation that Eli is advocating for.

Re: Opaque objects and Emacs documentation

[Gregory Heytings via Emacs development discussions.]

>
>> Which I think is against long-time Emacs tradition for documenting its 
>> interfaces, and by that facilitating extensibility.  It is IMO wrong to 
>> fill Emacs application levels with opaque objects which cannot be 
>> usefully described; they should be a rare exception, but definitely not 
>> the rule.
>
> The problem here is that even if you document a particular value, it's 
> _not useful_. It doesn't show you what you can or should do with it.
>
> The main thing thing the user can do with that value is misuse it, by 
> relying on its shape in the client code.
>

IMO the main goal of writing Emacs documentation (and software 
documentation in general) is not to help its users, but to help all those 
who in the future, when you will have moved to something else, will work 
on the code you wrote.  For them it is definitely very useful to 
understand the implementation choices you made, the internal 
representations you chose to use, and so forth.  In fact, such a 
documentation is probably also useful for yourself, if you leave your code 
temporarily, and come back to it after a while.

Gregory

Re: Opaque objects and Emacs documentation

[Dmitry Gutov]

On 17.07.2020 14:53, Gregory Heytings via Emacs development discussions. 
wrote:

> IMO the main goal of writing Emacs documentation (and software 
> documentation in general) is not to help its users, but to help all 
> those who in the future, when you will have moved to something else, 
> will work on the code you wrote.

So you have never encountered a difference between internal 
documentation and public documentation?

> For them it is definitely very useful 
> to understand the implementation choices you made, the internal 
> representations you chose to use, and so forth.  In fact, such a 
> documentation is probably also useful for yourself, if you leave your 
> code temporarily, and come back to it after a while.

Even coming back to it in a few years, I'm pretty sure I can spare 
several seconds to call project-try-vc and see which value it returns.

The most valuable documentation is not this kind:

   // add 1 to x
   x += 1

but that which describes original choices, and the reasons for them. Or 
an overview of how the system works. We're not discussing either of 
those options here.

Re: Opaque objects and Emacs documentation

[Gregory Heytings via Emacs development discussions.]

>
>> IMO the main goal of writing Emacs documentation (and software 
>> documentation in general) is not to help its users, but to help all 
>> those who in the future, when you will have moved to something else, 
>> will work on the code you wrote.
>
> So you have never encountered a difference between internal 
> documentation and public documentation?
>

Of course I did.  But as it happens, this difference does not exist in 
Emacs.  Remember that one of the direct inspirations of Emacs were Lisp 
machines, in which the user can read and modify almost every piece of code 
on the fly, from the lowest to the highest level.  In such a system, there 
can be no difference between "internal" and "external" documentation.  I 
understand that it can be difficult to adapt to this way of thinking when 
one comes from another programming tradition.

>
> The most valuable documentation is [...] that which describes original 
> choices, and the reasons for them. Or an overview of how the system 
> works. We're not discussing either of those options here.
>

It's one way to document things, but certainly not the only one.  I'm not 
sure I agree that it's the "most valuable" one.  When you have to work on 
code written by someone else, you rarely do so on the highest level, and 
having a picture of the overall organization is not directly useful. 
Having a clear and well-written documentation for each function is 
directly useful.

Gregory

RE: Opaque objects and Emacs documentation

[Drew Adams]

> Of course I did.  But as it happens, this difference does not exist in
> Emacs.  Remember that one of the direct inspirations of Emacs were Lisp
> machines, in which the user can read and modify almost every piece of
> code on the fly, from the lowest to the highest level.  In such a system,
> there can be no difference between "internal" and "external" documentation.
> I understand that it can be difficult to adapt to this way of thinking
> when one comes from another programming tradition.

FWIW, I agree.

The only uses of labeling something "internal" for Emacs
are (1) to indicate some relative risk/expectation of
future change in implementation and (2) to indicate some
relative risk in changing some fragile or tricky code.

And even in those case, IMO just a binary label "internal"
vs "external" is not the way to go.  Just document in more
words what's involved - which kind of risk, maybe exactly
what the risks are or the future expectations/intentions
are. 

IOW, "relative" is key here - it's not about an unexplained
absolute "hands-off" or "don't look".  We aren't limited
to just hanging an "Off Limits - No Admittance" sign.  We
can tell our future selves just what's involved, and why.

Re: Opaque objects and Emacs documentation

[Dmitry Gutov]

On 17.07.2020 17:26, Gregory Heytings via Emacs development discussions. 
wrote:
> Of course I did.  But as it happens, this difference does not exist in 
> Emacs.  Remember that one of the direct inspirations of Emacs were Lisp 
> machines, in which the user can read and modify almost every piece of 
> code on the fly, from the lowest to the highest level.  In such a 
> system, there can be no difference between "internal" and "external" 
> documentation.

Why not?

Given that Lisp allows one to expect any value, and jump to any 
implementation, and debug any function, I would say that actually 
_lowers_ the need to document things, in general, not the other way 
around, like in environments which you can't inspect and thus have to 
rely solely on documentation.

As for internal vs external difference, I believe it's still here, as 
with any programming language. The latter should be much shorter. There 
is no need to dump all internal details on somebody who just wants to 
use a library. It's simply counter-productive.

Re: Opaque objects and Emacs documentation

[Gregory Heytings via Emacs development discussions.]

[-- Attachment #1: Type: text/plain, Size: 1541 bytes --]


>
>> Of course I did.  But as it happens, this difference does not exist in 
>> Emacs.  Remember that one of the direct inspirations of Emacs were Lisp 
>> machines, in which the user can read and modify almost every piece of 
>> code on the fly, from the lowest to the highest level.  In such a 
>> system, there can be no difference between "internal" and "external" 
>> documentation.
>
> Why not?
>

I have no intention to start another thread / dispute with you.  I merely 
tried to provide you with some elements that explain why things are as 
they are.

>
> Given that Lisp allows one to expect any value, and jump to any 
> implementation, and debug any function, I would say that actually 
> _lowers_ the need to document things, in general, not the other way 
> around, like in environments which you can't inspect and thus have to 
> rely solely on documentation.
>

That's your opinion, these are not facts.  The fact that Emacs's code is 
both readable by every Emacs user *and* extremely well documented, and 
that for many closed source programs the contrary holds for both points, 
shows that your opinion does not make sense.  Moreover the immediate 
consequence of your opinion would be that the many people who have spent 
an enormous amount of time to document Emacs as carefully is possible, and 
who knew very well that the code was just there, were all mistaken and 
have wasted their time.  The fact that Emacs is still there after so many 
years shows that they have not.

Gregory

Re: Opaque objects and Emacs documentation

[Yuan Fu]

> On Jul 17, 2020, at 3:22 PM, Dmitry Gutov <dgutov@yandex.ru> wrote:
> 
> On 17.07.2020 17:26, Gregory Heytings via Emacs development discussions. wrote:
>> Of course I did.  But as it happens, this difference does not exist in Emacs.  Remember that one of the direct inspirations of Emacs were Lisp machines, in which the user can read and modify almost every piece of code on the fly, from the lowest to the highest level.  In such a system, there can be no difference between "internal" and "external" documentation.
> 
> Why not?
> 
> Given that Lisp allows one to expect any value, and jump to any implementation, and debug any function, I would say that actually _lowers_ the need to document things, in general, not the other way around, like in environments which you can't inspect and thus have to rely solely on documentation.

For the return value, I think documenting the return value is better than not doing so. Sure, I can evaluate the function and see the return value, but reading off the documentation is much easier. Why make other people’s life harder?

For documentation in general, I think the right thing to do is to document the function, clearly state that this is “internal” and shouldn’t be depended upon, and explain the reason why (or just say “see doctoring of xxx for why”). I don’t see why saying nothing is better than being explicit in documentation.

> 
> As for internal vs external difference, I believe it's still here, as with any programming language. The latter should be much shorter. There is no need to dump all internal details on somebody who just wants to use a library. It's simply counter-productive.
> 

As it happens to Emacs (and other free software), users often _are_ developers. Library authors should document their code to make it as easy as possible for others to look at the code and understand what it does, and start hacking on it. IOW, one writes documentation not merely for users, but for his fellow developers. (I guess this doesn’t apply to personal projects, but Emacs is as collaborative as it gets, so the point should still hold.)

Yuan

RE: Opaque objects and Emacs documentation

[Drew Adams]

> one writes documentation not merely
> for users, but for his fellow developers.
> (I guess this doesn’t apply to personal
> projects

It does.  Why?  Because we lose some degree
of familiarity with the code, and especially
with undocumented and unobvious "whys".

Communication not only from one human to
another, but from one human state, at one
time, to another state of the same human,
at another time, perhaps in another context.

Re: Opaque objects and Emacs documentation

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > to help all those 
  > who in the future, when you will have moved to something else, will work 
  > on the code you wrote.  For them it is definitely very useful to 
  > understand the implementation choices you made, the internal 
  > representations you chose to use, and so forth.  In fact, such a 
  > documentation is probably also useful for yourself, if you leave your code 
  > temporarily, and come back to it after a while.

Those explanations are very important, but they do not have to be in the
doc strings and manuals,  If including them in doc strings and manuals
is not particularly useful, we put them in comments in the source files.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Opaque objects and Emacs documentation

[Stefan Monnier]

> Basically, you are saying that in your opinion this is as it should
> be, and cannot be helped.
>
> Which I think is against long-time Emacs tradition for documenting its
> interfaces, and by that facilitating extensibility.  It is IMO wrong
> to fill Emacs application levels with opaque objects which cannot be
> usefully described; they should be a rare exception, but definitely
> not the rule.

I believe in this specific case, the issue is that the shape can and
will change and not only over time.  The only thing defined is an
interface and any code which relies on more than this interface
(e.g. a function which relies on completion-tables being lists of
strings) will break sooner or later.

Of course, that doesn't prevent the doc from giving *examples* to show
the intended behavior.


        Stefan

Re: Opaque objects and Emacs documentation

[Andy Moreton]

On Fri 17 Jul 2020, Stefan Monnier wrote:

>> Basically, you are saying that in your opinion this is as it should
>> be, and cannot be helped.
>>
>> Which I think is against long-time Emacs tradition for documenting its
>> interfaces, and by that facilitating extensibility.  It is IMO wrong
>> to fill Emacs application levels with opaque objects which cannot be
>> usefully described; they should be a rare exception, but definitely
>> not the rule.
>
> I believe in this specific case, the issue is that the shape can and
> will change and not only over time.  The only thing defined is an
> interface and any code which relies on more than this interface
> (e.g. a function which relies on completion-tables being lists of
> strings) will break sooner or later.
>
> Of course, that doesn't prevent the doc from giving *examples* to show
> the intended behavior.

I have read this (rather long) thread, and its discussion of all kinds
of low level detail, but it seems the bigger picture is missing.

Documentation is needed for three distinct audiences:
a) Emacs users
b) Developers adding new methods to provide a new project implementation
   which conform to the project API (genric functions)
c) Developers extending the project.el infrastructure.
   This is already covered by the commentry in the code (I hope).

Taking the first two cases in turn:

a) Users:
Currently there is no documentation that I can see in the manuals for
the project feature for users. As an ordinary user, it is not clear what
problem project.el solves, or how it helps users with their work.

So, please try to extend the docs and the manual for users to state:
 - what is a project ?
 - what is the "current project" ?
 - what is the "transient project" (mentioned in project-current) ?
 - why is this useful ? How does it help the user ?
 - what is the interface for users ?
 - how do users discover this interface ?

The doc string for project-root uses the term "current project" without
defining it. What is the "current project" ? How is it chosen ? The
*help* buffer for project-root docs also shows 3 method implementations,
all of which are undocumented: documenting these methods would be more
helpful.


b) Project backend developers
The commentary at the top of project.el describes the project interface
fairly loosely. The docs need to describe the interface more precisely:
 - which functions must be implemented ?
 - which functions are optional ?
 - what should a developer consider when choosing which optional
   functions to implement ?
 - what is important to consider when implementing these functions ?
 - if return values are opaque types consumed by other functions, then
   the docs must clearly state which interface functions consume this
   data.

There has been lengthy discussion about documenting the data types used
in the project interface for each implementation. If the interface
behaviour is precisely specified, the exact data types are perhaps less
important. However it can still be helpful to describe the data types
for developers learning how the pieces fit together, even if it is not
part of the API contract.

                                -=o0o=-

As an ordinary user, project.el requires significant investment of time
and effort to discover if it might be useful or not. Most users do not
read the elisp sources, and are not necessarily familiar with CL generics.

Documenting project.el so that learning process is as short as possible
would be a great service to emacs users.

    AndyM

Re: Opaque objects and Emacs documentation

[Andy Moreton]

On Fri 24 Jul 2020, Andy Moreton wrote:

> On Fri 17 Jul 2020, Stefan Monnier wrote:
>
>>> Basically, you are saying that in your opinion this is as it should
>>> be, and cannot be helped.
>>>
>>> Which I think is against long-time Emacs tradition for documenting its
>>> interfaces, and by that facilitating extensibility.  It is IMO wrong
>>> to fill Emacs application levels with opaque objects which cannot be
>>> usefully described; they should be a rare exception, but definitely
>>> not the rule.
>>
>> I believe in this specific case, the issue is that the shape can and
>> will change and not only over time.  The only thing defined is an
>> interface and any code which relies on more than this interface
>> (e.g. a function which relies on completion-tables being lists of
>> strings) will break sooner or later.
>>
>> Of course, that doesn't prevent the doc from giving *examples* to show
>> the intended behavior.
>
> I have read this (rather long) thread, and its discussion of all kinds
> of low level detail, but it seems the bigger picture is missing.
>
> Documentation is needed for three distinct audiences:
> a) Emacs users
> b) Developers adding new methods to provide a new project implementation
>    which conform to the project API (genric functions)
> c) Developers extending the project.el infrastructure.
>    This is already covered by the commentry in the code (I hope).
>
> Taking the first two cases in turn:
>
> a) Users:
> Currently there is no documentation that I can see in the manuals for
> the project feature for users. As an ordinary user, it is not clear what
> problem project.el solves, or how it helps users with their work.

Having sent this, I then found that there is a "Working with Projects"
section in the manual, but which is not easily found from the top level
and is not indexed so it can be found by looking for "project".

    AndyM

Re: Opaque objects and Emacs documentation

[Eli Zaretskii]

> From: Andy Moreton <andrewjmoreton@gmail.com>
> Date: Fri, 24 Jul 2020 00:40:38 +0100
> 
> Having sent this, I then found that there is a "Working with Projects"
> section in the manual, but which is not easily found from the top level
> and is not indexed so it can be found by looking for "project".

What do you mean by "not indexed"?  There are several index entries
for that section; type "i project TAB" to see them.  If you have
suggestions for more index entries, please tell what entries you think
we should add or modify.

Thanks.

Re: Opaque objects and Emacs documentation

[Andy Moreton]

On Fri 24 Jul 2020, Eli Zaretskii wrote:

>> From: Andy Moreton <andrewjmoreton@gmail.com>
>> Date: Fri, 24 Jul 2020 00:40:38 +0100
>> 
>> Having sent this, I then found that there is a "Working with Projects"
>> section in the manual, but which is not easily found from the top level
>> and is not indexed so it can be found by looking for "project".
>
> What do you mean by "not indexed"?  There are several index entries
> for that section; type "i project TAB" to see them.  If you have
> suggestions for more index entries, please tell what entries you think
> we should add or modify.

If you don't use tab completion then it is well buried:

"C-h r i project RET ,,,,,,,,," to get to the "Working with Projects"
node.

I have never needed tab completion on info entries before, as it usually
takes you to the relevant entry first. It would be better if the
indexing landed at the "Working with Projects" node before the other
entries.

    AndyM

Re: Opaque objects and Emacs documentation

[Eli Zaretskii]

> From: Andy Moreton <andrewjmoreton@gmail.com>
> Date: Fri, 24 Jul 2020 12:23:24 +0100
> 
> If you don't use tab completion then it is well buried:
> 
> "C-h r i project RET ,,,,,,,,," to get to the "Working with Projects"
> node.
> 
> I have never needed tab completion on info entries before, as it usually
> takes you to the relevant entry first.

I suggest to start using TAB.  It is very hard to make sure whatever
you are looking for is always the first index hit, especially for a
"popular" and general-purpose word such as "project".  We could tweak
the index entries to produce that effect, but IME it's highly probable
that this will deteriorate with time, as new index entries are added.

Btw, if you say "i projects RET", you do get to that section.

Re: Opaque objects and Emacs documentation

[Dmitry Gutov]

On 24.07.2020 02:24, Andy Moreton wrote:

> Taking the first two cases in turn:
> 
> a) Users:
> Currently there is no documentation that I can see in the manuals for
> the project feature for users. As an ordinary user, it is not clear what
> problem project.el solves, or how it helps users with their work.

IME, "ordinary users" don't read the manual. At least, not as the first 
stop. But sure, we'll need to add some more to it.

> So, please try to extend the docs and the manual for users to state:
>   - what is a project ?
>   - what is the "current project" ?
>   - what is the "transient project" (mentioned in project-current) ?
>   - why is this useful ? How does it help the user ?
>   - what is the interface for users ?
>   - how do users discover this interface ?
> 
> The doc string for project-root uses the term "current project" without
> defining it. What is the "current project" ? How is it chosen ?

Doesn't the Commentary cover most (all?) of the above?

BTW, did you read the latest version of it in master?

> The
> *help* buffer for project-root docs also shows 3 method implementations,
> all of which are undocumented: documenting these methods would be more
> helpful.

Fair point.

But would those docs say anything more than

   Return the root directory of a "transient" project

   Return the root directory of a "vcs-backed" project

?

> 
> b) Project backend developers
> The commentary at the top of project.el describes the project interface
> fairly loosely. The docs need to describe the interface more precisely:
>   - which functions must be implemented ?
>   - which functions are optional ?
>   - what should a developer consider when choosing which optional
>     functions to implement ?
>   - what is important to consider when implementing these functions ?
>   - if return values are opaque types consumed by other functions, then
>     the docs must clearly state which interface functions consume this
>     data.

I think Commentary covers the above now. Again, the latest version.

> As an ordinary user, project.el requires significant investment of time
> and effort to discover if it might be useful or not. Most users do not
> read the elisp sources, and are not necessarily familiar with CL generics.

You might want to try pressing 'C-x p C-h'.

It's a good quick overview, even if you'll have to guess at the 
semantics of "current project".

Re: Opaque objects and Emacs documentation

[Andy Moreton]

On Fri 24 Jul 2020, Dmitry Gutov wrote:

> On 24.07.2020 02:24, Andy Moreton wrote:
>
>> Taking the first two cases in turn:
>> a) Users:
>> Currently there is no documentation that I can see in the manuals for
>> the project feature for users. As an ordinary user, it is not clear what
>> problem project.el solves, or how it helps users with their work.
>
> IME, "ordinary users" don't read the manual. At least, not as the first stop.
> But sure, we'll need to add some more to it.

Indeed. The index in the manual needs improving so that "i project RET" finds
the "Working with Projects" node.

>> So, please try to extend the docs and the manual for users to state:
>>   - what is a project ?
>>   - what is the "current project" ?
>>   - what is the "transient project" (mentioned in project-current) ?
>>   - why is this useful ? How does it help the user ?
>>   - what is the interface for users ?
>>   - how do users discover this interface ?
>> The doc string for project-root uses the term "current project" without
>> defining it. What is the "current project" ? How is it chosen ?
>
> Doesn't the Commentary cover most (all?) of the above?

The manual does cover most of this. User facing docs need to focus on
what the backends can do more than the project.el machinery, so as more
backends are added, some of the project.el descripion should probably
migrate to the elisp manual (as it is of more interest to developers).

> BTW, did you read the latest version of it in master?

Yes - belatedly, as I only found the info node after posting my previous
message.

>> The
>> *help* buffer for project-root docs also shows 3 method implementations,
>> all of which are undocumented: documenting these methods would be more
>> helpful.
>
> Fair point.
>
> But would those docs say anything more than
>
>   Return the root directory of a "transient" project
>
>   Return the root directory of a "vcs-backed" project

A one-liner to give a sense of what they do is fine: that is still more
helpful than a list of entries that all say "undocumented".

>> b) Project backend developers
>> The commentary at the top of project.el describes the project interface
>> fairly loosely. The docs need to describe the interface more precisely:
>>   - which functions must be implemented ?
>>   - which functions are optional ?
>>   - what should a developer consider when choosing which optional
>>     functions to implement ?
>>   - what is important to consider when implementing these functions ?
>>   - if return values are opaque types consumed by other functions, then
>>     the docs must clearly state which interface functions consume this
>>     data.
>
> I think Commentary covers the above now. Again, the latest version.

I read project.el from top of tree before posting: it was not clear what
the minimal set of methods that are required, nor what are the
constraints on how the implementations should behave.

>> As an ordinary user, project.el requires significant investment of time
>> and effort to discover if it might be useful or not. Most users do not
>> read the elisp sources, and are not necessarily familiar with CL generics.
>
> You might want to try pressing 'C-x p C-h'.

A useful entry point, but only once you know it exists! Making things
discoverable is important for getting new users up to speed.

    AndyM

Re: Opaque objects and Emacs documentation

[Eli Zaretskii]

> From: Andy Moreton <andrewjmoreton@gmail.com>
> Date: Fri, 24 Jul 2020 02:22:16 +0100
> 
> The index in the manual needs improving so that "i project RET" finds
> the "Working with Projects" node.

It does, on the 2nd hit.

Re: Opaque objects and Emacs documentation

[Andy Moreton]

On Fri 24 Jul 2020, Eli Zaretskii wrote:

>> From: Andy Moreton <andrewjmoreton@gmail.com>
>> Date: Fri, 24 Jul 2020 02:22:16 +0100
>> 
>> The index in the manual needs improving so that "i project RET" finds
>> the "Working with Projects" node.
>
> It does, on the 2nd hit.

Not for me: please show the key sequence for that.

    AndyM

Re: Opaque objects and Emacs documentation

[Eli Zaretskii]

> From: Andy Moreton <andrewjmoreton@gmail.com>
> Date: Sat, 25 Jul 2020 17:33:49 +0100
> 
> >> The index in the manual needs improving so that "i project RET" finds
> >> the "Working with Projects" node.
> >
> > It does, on the 2nd hit.
> 
> Not for me: please show the key sequence for that.

i project RET ,

Re: Opaque objects and Emacs documentation

[Andy Moreton]

On Sat 25 Jul 2020, Eli Zaretskii wrote:

>> From: Andy Moreton <andrewjmoreton@gmail.com>
>> Date: Sat, 25 Jul 2020 17:33:49 +0100
>> 
>> >> The index in the manual needs improving so that "i project RET" finds
>> >> the "Working with Projects" node.
>> >
>> > It does, on the 2nd hit.
>> 
>> Not for me: please show the key sequence for that.
>
> i project RET ,

Nope, that goes to the end of node "(emacs) Project File Commands", even
from "emacs -Q".

   AndyM

Re: Opaque objects and Emacs documentation

[Eli Zaretskii]

> From: Andy Moreton <andrewjmoreton@gmail.com>
> Date: Sat, 25 Jul 2020 21:18:53 +0100
> 
> > i project RET ,
> 
> Nope, that goes to the end of node "(emacs) Project File Commands", even
> from "emacs -Q".

And how is that a problem?

Re: Opaque objects and Emacs documentation

[Andy Moreton]

On Sun 26 Jul 2020, Eli Zaretskii wrote:

>> From: Andy Moreton <andrewjmoreton@gmail.com>
>> Date: Sat, 25 Jul 2020 21:18:53 +0100
>> 
>> > i project RET ,
>> 
>> Nope, that goes to the end of node "(emacs) Project File Commands", even
>> from "emacs -Q".
>
> And how is that a problem?

Because it does not match the question asked. You claimed that the 2nd
hit in the search for "project" would land at the "Working with
Projects" node, but does not do so.

It is not a problem, but your answer was not accurate.

    AndyM

Re: Opaque objects and Emacs documentation

[Eli Zaretskii]

> From: Andy Moreton <andrewjmoreton@gmail.com>
> Date: Sun, 26 Jul 2020 10:16:59 +0100
> 
> >> > i project RET ,
> >> 
> >> Nope, that goes to the end of node "(emacs) Project File Commands", even
> >> from "emacs -Q".
> >
> > And how is that a problem?
> 
> Because it does not match the question asked. You claimed that the 2nd
> hit in the search for "project" would land at the "Working with
> Projects" node, but does not do so.
> 
> It is not a problem, but your answer was not accurate.

I apologize for being inaccurate (it was accurate in Emacs 27, where
there's a single section devoted to project.el functionality).

My point, though, is that for the use case where the user has no idea
what can be done with project.el, and asks him/herself "what is this
Project thing", getting to a subsection of that section in a couple of
keystrokes is "good enough".

Re: Opaque objects and Emacs documentation

[Dmitry Gutov]

On 24.07.2020 04:22, Andy Moreton wrote:

> I read project.el from top of tree before posting: it was not clear what
> the minimal set of methods that are required, nor what are the
> constraints on how the implementations should behave.

Quoting from there:

   ;; `project-root' must be defined for every project.
   ;; `project-files' can be overridden for performance purposes.
   ;; `project-ignores' and `project-external-roots' describe the project
   ;; files and its relations to external directories.  `project-files'
   ;; should be consistent with `project-ignores'.

   <...>

   ;; - Define new methods for some or all generic functions for this
   ;; backend using `cl-defmethod'.  A `project-root' method is
   ;; mandatory, `project-files' is recommended, the rest are optional.

>>> As an ordinary user, project.el requires significant investment of time
>>> and effort to discover if it might be useful or not. Most users do not
>>> read the elisp sources, and are not necessarily familiar with CL generics.
>>
>> You might want to try pressing 'C-x p C-h'.
> 
> A useful entry point, but only once you know it exists! Making things
> discoverable is important for getting new users up to speed.

 From the Commentary again:

   ;; Type `C-x p C-h' to see all available commands and bindings.

Re: Opaque objects and Emacs documentation

[Andy Moreton]

On Fri 24 Jul 2020, Dmitry Gutov wrote:

> On 24.07.2020 04:22, Andy Moreton wrote:
>>>> As an ordinary user, project.el requires significant investment of time
>>>> and effort to discover if it might be useful or not. Most users do not
>>>> read the elisp sources, and are not necessarily familiar with CL generics.
>>>
>>> You might want to try pressing 'C-x p C-h'.
>> A useful entry point, but only once you know it exists! Making things
>> discoverable is important for getting new users up to speed.
>
> From the Commentary again:
>
>   ;; Type `C-x p C-h' to see all available commands and bindings.

The commentry is aimed at devleopers who have the elisp source available.
Users might not have the lisp sources installed, and even if it is
available are less likely to read it.

    AndyM

Re: Opaque objects and Emacs documentation

[Eli Zaretskii]

> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Fri, 24 Jul 2020 03:43:41 +0300
> 
> Doesn't the Commentary cover most (all?) of the above?

Documenting stuff in the commentary is better than nothing, but it is
sub-optimal: our Help commands don't scan comments, so the information
that is only in comments is less discoverable, certainly by users.

Therefore, I think any information important to users should be in doc
strings and the manual, not just in comments.

Re: Opaque objects and Emacs documentation

[Dmitry Gutov]

On 24.07.2020 08:45, Eli Zaretskii wrote:
>> From: Dmitry Gutov <dgutov@yandex.ru>
>> Date: Fri, 24 Jul 2020 03:43:41 +0300
>>
>> Doesn't the Commentary cover most (all?) of the above?
> 
> Documenting stuff in the commentary is better than nothing, but it is
> sub-optimal: our Help commands don't scan comments, so the information
> that is only in comments is less discoverable, certainly by users.

We do have 'M-x describe-package', though (C-h P).

> Therefore, I think any information important to users should be in doc
> strings and the manual, not just in comments.

I agree, but it's not like we can describe what the "current project" is 
in concrete terms in a user-facing command.

As for the manual, we already talked about this. Hopefully when you 
start on the alternative project backend, it can persuade you to include 
more higher level language in the descriptions.

RE: Opaque objects and Emacs documentation

[Drew Adams]

> >> Doesn't the Commentary cover most (all?) of the above?
> >
> > Documenting stuff in the commentary is better than nothing, but it is
> > sub-optimal: our Help commands don't scan comments, so the information
> > that is only in comments is less discoverable, certainly by users.
> 
> We do have 'M-x describe-package', though (C-h P).

And there is `finder-commentary'.  It could
be bound to a help key (`C-h <something>').

The version of `finder-commentary' I have in
`finder+.el' names the buffer `*Commentary, <file>*'
where <file> is the file the Commentary is from,
not just `*Finder-package*'.  That labels it well,
and lets you easily have more than one such help
buffer.

https://www.emacswiki.org/emacs/download/finder%2b.el

https://www.emacswiki.org/emacs/FinderMode

> > Therefore, I think any information important to
> > users should be in doc strings and the manual,
> > not just in comments.

I agree with that.  (But I also think that a helpful
Commentary is a good thing, and I miss the Commentary
sections of yesteryear.)

Re: Opaque objects and Emacs documentation

[Eli Zaretskii]

> Date: Fri, 24 Jul 2020 17:21:36 +0000 (UTC)
> From: Drew Adams <drew.adams@oracle.com>
> Cc: andrewjmoreton@gmail.com, emacs-devel@gnu.org
> 
> > We do have 'M-x describe-package', though (C-h P).
> 
> And there is `finder-commentary'.  It could
> be bound to a help key (`C-h <something>').

This is all missing the point: we also have "M-x grep".

The point is not to be able to search comments or display them, the
point is to be able to find relevant commands/variables/functions by
regexp or keyword search, using a single command.