docstrings and elisp reference

docstrings and elisp reference

[Jean-Christophe Helary]

I was thinking (naively?) that having the docstrings on one side and the reference on the other side was not a very efficient way to maintain documentation. Is everything in the documentation actually written by hand based on the docstrings? Wouldn't it be nicer to have good docstrings and use them directly in the documentation to avoid duplication of work? Is there any reason I'm not seeing why this is not happening?

Jean-Christophe 

Re: docstrings and elisp reference

[Tino Calancha]

On Tue, 6 Jun 2017, Jean-Christophe Helary wrote:

> I was thinking (naively?) that having the docstrings on one side and the reference on the other side was not a very efficient way to maintain documentation. Is everything in the documentation actually written by hand based on the docstrings? Wouldn't it be nicer to have good docstrings and use them directly in the documentation to avoid duplication of work? Is there any reason I'm not seeing why this is not happening?
Hi,

you might wish to read the following thread:
https://lists.gnu.org/archive/html/emacs-devel/2016-04/msg00452.html

Re: docstrings and elisp reference

[Jean-Christophe Helary]

> 2017/06/06 9:09、Tino Calancha <tino.calancha@gmail.com>のメール:
> 
>> I was thinking (naively?) that having the docstrings on one side and the reference on the other side was not a very efficient way to maintain documentation. Is everything in the documentation actually written by hand based on the docstrings? Wouldn't it be nicer to have good docstrings and use them directly in the documentation to avoid duplication of work? Is there any reason I'm not seeing why this is not happening?

> Hi,
> 
> you might wish to read the following thread:
> https://lists.gnu.org/archive/html/emacs-devel/2016-04/msg00452.html

Wow, did I open a can of worms... :)

Fascinating thread.

Just a few comments.

Having a mechanism that generates automatic documentation (a la javadoc) would be extremely valuable. By just having that "reference" it would suggest developers to adopt a more standardized format/style.

The elisp reference should definitely extend the descriptions by adding exemples so that people learn from reading (which is admittedly difficult with the docstrings only). Maybe not go as far as a cookbook, but a bit more than what we have would be very welcome (there is a *huge* gap between the Introduction and the Reference).

My initial worry is that the day we start l10n (and we'll get there eventually :), we're going to have a huge amount of redundancy. But the flip side is that we're going to have also a huge amount of feedback on both the docstrings and the elisp reference...

Jean-Christophe 

Re: docstrings and elisp reference

[Eli Zaretskii]

> From: Jean-Christophe Helary <jean.christophe.helary@gmail.com>
> Date: Tue, 6 Jun 2017 14:10:19 +0900
> 
> Having a mechanism that generates automatic documentation (a la javadoc) would be extremely valuable.

It could be valuable, but IMO only if we exercise a lot of discipline
in producing those comments.  Because every project which uses this
methodology that I looked at ends up providing _abysmally_ inadequate
documentation.  As one very typical example, look at the GTK docs,
e.g.  here:

  https://developer.gnome.org/gtk4/stable/GtkWindow.html

Re: docstrings and elisp reference

[Stephen Leake]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Jean-Christophe Helary <jean.christophe.helary@gmail.com>
>> Date: Tue, 6 Jun 2017 14:10:19 +0900
>> 
>> Having a mechanism that generates automatic documentation (a la
>> javadoc) would be extremely valuable.
>
> It could be valuable, but IMO only if we exercise a lot of discipline
> in producing those comments.  Because every project which uses this
> methodology that I looked at ends up providing _abysmally_ inadequate
> documentation.

+1

Doc strings are for short reference information for single functions/variables.

texinfo docs are for more indepth explanations of several related functions/features.

They are complementary; they should not duplicate each other.

-- 
-- Stephe

RE: docstrings and elisp reference

[Drew Adams]

> >> Having a mechanism that generates automatic documentation (a la
> >> javadoc) would be extremely valuable.
> >
> > It could be valuable, but IMO only if we exercise a lot of discipline
> > in producing those comments.  Because every project which uses this
> > methodology that I looked at ends up providing _abysmally_ inadequate
> > documentation.
> 
> +1
> 
> Doc strings are for short reference information for single
> functions/variables.
> 
> texinfo docs are for more indepth explanations of several related
> functions/features.
> 
> They are complementary; they should not duplicate each other.

Yes.

And this is not JavaDoc.  The Elisp manual is not purely
and simply an API reference manual.

And now this question is being rehashed again.  Everything
said this time around has already been said before.  Please
consult the previous thread, which was already cited.

(If you feel you really have something *new* to say, feel free.)

Re: docstrings and elisp reference

[Etienne Prud’homme]

Drew Adams <drew.adams@oracle.com> writes:

> And this is not JavaDoc.  The Elisp manual is not purely
> and simply an API reference manual.

It’s worth mentioning that JavaDoc documentation style is particularly
useful for Object-Oriented Programming and strongly typed languages.  I
think it’s even mandatory with OOP to be readable.

However, I also think Jean-Christophe makes a good point about
documentation generation.  Not with duplication, but semantic support.
While documentation support is awesome in Emacs with GNU libraries, it’s
not always so with third-party documentation tools.  I’m thinking about
Zeal (and to a very limited extent Dash that is not free).

Those tools are highly effective for semantic indexation for newcomers
since they offer a simple interface for hundred FLOSS libraries.  GNU
projects are almost nonexistent.

I’ve been trying in the past to port GNU projects documentation and I
finally gave up.  I find Texinfo to be very limited when it comes to
semantic support.  It’s really hard to extract meaningful definitions
from texi files.

Re: docstrings and elisp reference

[Jean-Christophe Helary]

It was really not my intent to raise the temperature of this list to that level. Apologies for that. But the discussion is really interesting.

1) I'll read the Doc Strings and Manuals in the GNU Coding Standards.

2) my remark about javadoc was not intended to suggest a replacement of the current documentation, but to have a document that has all the docstring information (Emacs API) in one place. It would be an alternative to the info system, with exactly the same contents. It would be a great tool to find API information and also very practical to check if the rules defined in the GNU Coding Standards are respected.

3) my remark about duplication was full answered in the discussion and I'll have even better insights when I complete 1) above. So I'll just put that aside for the moment. My perspective was not exclusively l10n but also ease of access to information for beginners, etc.

4) regarding "semantic support" (although I'm not 100% sure what Etienne is referring to here) it seems to me that there are a number of not so hard things to do on the (some ?) texi conversion templates. I've been working on a CSS for the manuals and I found the HTML conversion templates remarkably underused (I'm currently discussing this on help@texinfo). Even while maintaining legacy HTML support, there are plenty of areas where CSS selectors could be added to considerably enhance the output. There are even some regressions, for ex, the 4.8 version I was unknowingly working with until this morning added a class="defun" to definitions, and that class information is gone in 6.3 (replaced by a generic <dl> tag).

I understand I'm going in a lot of directions at the moment, sorry for that.

Jean-Christophe 

> Jun 8, 2017 11:29、Etienne Prud’homme <e.e.f.prudhomme@gmail.com>のメール:
> 
> Drew Adams <drew.adams@oracle.com> writes:
> 
>> And this is not JavaDoc.  The Elisp manual is not purely
>> and simply an API reference manual.
> 
> It’s worth mentioning that JavaDoc documentation style is particularly
> useful for Object-Oriented Programming and strongly typed languages.  I
> think it’s even mandatory with OOP to be readable.
> 
> However, I also think Jean-Christophe makes a good point about
> documentation generation.  Not with duplication, but semantic support.
> While documentation support is awesome in Emacs with GNU libraries, it’s
> not always so with third-party documentation tools.  I’m thinking about
> Zeal (and to a very limited extent Dash that is not free).
> 
> Those tools are highly effective for semantic indexation for newcomers
> since they offer a simple interface for hundred FLOSS libraries.  GNU
> projects are almost nonexistent.
> 
> I’ve been trying in the past to port GNU projects documentation and I
> finally gave up.  I find Texinfo to be very limited when it comes to
> semantic support.  It’s really hard to extract meaningful definitions
> from texi files.
> 

Re: docstrings and elisp reference

[Chad Brown]

> On 7Jun, 2017, at 20:48, Jean-Christophe Helary <jean.christophe.helary@gmail.com> wrote:
> 
> 2) my remark about javadoc was not intended to suggest a replacement of the current documentation, but to have a document that has all the docstring information (Emacs API) in one place. It would be an alternative to the info system, with exactly the same contents. It would be a great tool to find API information and also very practical to check if the rules defined in the GNU Coding Standards are respected.

Could you perhaps talk about what you see as the benefit to “an alternative to the info system, with exactly the same contents”? 

For example, I could imagine some small value to being able to see info manuals incorporated into systems that have already included documentation sets (like Zeal, mentioned earlier), but that also represents a loss, since learning how to use info in particular — and learning how the “self-documenting” part of emacs works in general — is a potent step in mastering emacs. Also, as you might have noticed, emacs tends to prefer to incorporate other systems, rather than the other way around. :-)

Certainly, there’s a fair bit of friction and a lot of inertia around both info and texifnfo, but it looks like you’re already looking into that — thanks in advance for that, by the way. 

I hope that helps,
~Chad

Re: docstrings and elisp reference

[Jean-Christophe Helary]

> On Jun 8, 2017, at 14:39, Chad Brown <yandros@gmail.com> wrote:
> 
> 
>> On 7Jun, 2017, at 20:48, Jean-Christophe Helary <jean.christophe.helary@gmail.com> wrote:
>> 
>> 2) my remark about javadoc was not intended to suggest a replacement of the current documentation, but to have a document that has all the docstring information (Emacs API) in one place. It would be an alternative to the info system, with exactly the same contents. It would be a great tool to find API information and also very practical to check if the rules defined in the GNU Coding Standards are respected.
> 
> Could you perhaps talk about what you see as the benefit to “an alternative to the info system, with exactly the same contents”? 

An HTML page with the whole API with visual cues.

Jean-Christophe 

Re: docstrings and elisp reference

[Eli Zaretskii]

> From: Jean-Christophe Helary <jean.christophe.helary@gmail.com>
> Date: Thu, 8 Jun 2017 17:12:37 +0900
> 
> > Could you perhaps talk about what you see as the benefit to “an alternative to the info system, with exactly the same contents”? 
> 
> An HTML page with the whole API with visual cues.

Can you elaborate on the "visual cues" part?

Thanks.

Re: docstrings and elisp reference

[Jean-Christophe Helary]

> On Jun 9, 2017, at 0:11, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> From: Jean-Christophe Helary <jean.christophe.helary@gmail.com>
>> Date: Thu, 8 Jun 2017 17:12:37 +0900
>> 
>>> Could you perhaps talk about what you see as the benefit to “an alternative to the info system, with exactly the same contents”? 
>> 
>> An HTML page with the whole API with visual cues.
> 
> Can you elaborate on the "visual cues" part?

I mean a data set that makes full use of CSS selectors and provides visual information about where are the function names, the arguments, the types of arguments, etc. by using something like syntax highlighting and things like this.

But the contents would be exclusively the docstrings, with 2 way links to the elisp reference. I guess when we develop an extraction process for l10n, it will be possible to implement.

Jean-Christophe

Re: docstrings and elisp reference

[Chad Brown]

[-- Attachment #1: Type: text/plain, Size: 1987 bytes --]


> On 8Jun, 2017, at 08:42, Jean-Christophe Helary <jean.christophe.helary@gmail.com> wrote:
> 
> 
>> On Jun 9, 2017, at 0:11, Eli Zaretskii <eliz@gnu.org> wrote:
>> 
>>> From: Jean-Christophe Helary <jean.christophe.helary@gmail.com>
>>> Date: Thu, 8 Jun 2017 17:12:37 +0900
>>> 
>>>> Could you perhaps talk about what you see as the benefit to “an alternative to the info system, with exactly the same contents”? 
>>> 
>>> An HTML page with the whole API with visual cues.
>> 
>> Can you elaborate on the "visual cues" part?
> 
> I mean a data set that makes full use of CSS selectors and provides visual information about where are the function names, the arguments, the types of arguments, etc. by using something like syntax highlighting and things like this.
> 
> But the contents would be exclusively the docstrings, with 2 way links to the elisp reference. I guess when we develop an extraction process for l10n, it will be possible to implement.

You might be interested in some work that Nic Ferrier did on Info and HTML a while back. There’s a (long) emacs-devel thread on the topic that ranged around a bit. This looks like a reasonable place to start in the archives, on “HTML-Info design"

	https://lists.gnu.org/archive/html/emacs-devel/2014-12/msg01901.html <https://lists.gnu.org/archive/html/emacs-devel/2014-12/msg01901.html>

Two more literal starting points for the discussion, perhaps not quite as helpful, are here:
	
	https://lists.gnu.org/archive/html/emacs-devel/2014-12/msg01672.html <https://lists.gnu.org/archive/html/emacs-devel/2014-12/msg01672.html>
	https://lists.gnu.org/archive/html/emacs-devel/2014-12/msg00347.html <https://lists.gnu.org/archive/html/emacs-devel/2014-12/msg00347.html>

There used to be a good-looking web-based example of Nic’s work, but it seems to be down now; trying to access it gets me errors that suggest a caching server sitting in front of a down or non-existant origin server.

~Chad

[-- Attachment #2: Type: text/html, Size: 3382 bytes --]

Re: docstrings and elisp reference

[Etienne Prud’homme]

[-- Attachment #1: Type: text/plain, Size: 293 bytes --]

Jean-Christophe Helary <jean.christophe.helary@gmail.com> writes:

> It was really not my intent to raise the temperature of this list to
> that level. Apologies for that. But the discussion is really
> interesting.

Maybe raising gmane temperature, but we’re good on list.gnu.org.


[-- Attachment #2: Gmane Temperature --]
[-- Type: image/png, Size: 4826 bytes --]

Re: docstrings and elisp reference

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > 2) my remark about javadoc was not intended to suggest a
  > replacement of the current documentation, but to have a document
  > that has all the docstring information (Emacs API) in one
  > place. It would be an alternative to the info system, with exactly
  > the same contents. It would be a great tool to find API
  > information and also very practical to check if the rules defined
  > in the GNU Coding Standards are respected.

All the doc strings are found together in one place: Emacs.
That's a program, not a document.  But why in practice would it be
better to have them all in one document?

It should not be hard to write a Lisp program which will write
that document into a file, straight from Emacs.


-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: docstrings and elisp reference

[Eli Zaretskii]

> From: Etienne Prud’homme <e.e.f.prudhomme@gmail.com>
> Date: Wed, 07 Jun 2017 22:29:47 -0400
> Cc: Stephen Leake <stephen_leake@stephe-leake.org>,
> 	emacs-devel <emacs-devel@gnu.org>
> 
> However, I also think Jean-Christophe makes a good point about
> documentation generation.  Not with duplication, but semantic support.
> While documentation support is awesome in Emacs with GNU libraries, it’s
> not always so with third-party documentation tools.  I’m thinking about
> Zeal (and to a very limited extent Dash that is not free).

Are there any other similar free tools?  Zeal seems a one-man project,
that is not developed too actively.  What's more, at least the docs it
has for Bash and ELisp are exactly the respective Texinfo manuals,
with the same text and the same node structure.  What is the advantage
of using a different browser for the exact same text?

> Those tools are highly effective for semantic indexation for newcomers
> since they offer a simple interface for hundred FLOSS libraries.  GNU
> projects are almost nonexistent.

Most of the docs offered with Zeal are about languages and other
similar systems, so docs of GNU packages among them is more like the
odd one out than the rule.  And given the contents, which is exactly
the Info manual, I don't see why bother.  Am I missing something?

And I don't think I understand what you mean by "semantic indexation".
Can you explain that using as example the ELisp manual available for
Zeal?

> I’ve been trying in the past to port GNU projects documentation and I
> finally gave up.  I find Texinfo to be very limited when it comes to
> semantic support.  It’s really hard to extract meaningful definitions
> from texi files.

This should be discussed on the Texinfo list, not here.  The current
Texinfo translator is highly customizable, so it could be that it can
be adapted to these needs much better.  And the fact that the 2 GNU
manuals there are obviously translations of Texinfo seems to
contradict what you say here.

Re: docstrings and elisp reference

[Etienne Prud’homme]

Eli Zaretskii <eliz@gnu.org> writes:

> Are there any other similar free tools?  Zeal seems a one-man project,
> that is not developed too actively.

There’s helm-dash.  While the name makes it look like it depends on
Dash, it’s only using the Dash documentation format.

Being a one man project doesn’t exclude it from being popular.  I see
4,941 stars on Github.  It’s particularly used (with Dash too) in the
Front-End development field.

> What's more, at least the docs it has for Bash and ELisp are exactly
> the respective Texinfo manuals, with the same text and the same node
> structure.  What is the advantage of using a different browser for the
> exact same text?

That’s my point.  I must confess that the available documentation for
Elisp has better support now for choosing whether we search for a macro,
command, function, etc.  It used to index every symbols in the same
entry type (variable entry).  I may have been wrong in thinking it
wasn’t possible, but there has to be a lot of work for the docset
generator to do that.  I wouldn’t be surprized that the Elisp docset was
indexed by hand from Kapeli.  He has economic incitatives to make good
quality indexation.

Furthermore, we could support much more indexation capabilities given
the list of supported entry types[1].

> Most of the docs offered with Zeal are about languages and other
> similar systems, so docs of GNU packages among them is more like the
> odd one out than the rule.  And given the contents, which is exactly
> the Info manual, I don't see why bother.  Am I missing something?

What I meant was really the uniform search interface it provides.  For
advanced Emacs users, it’s not a problem looking what we want from Emacs
itself (or Info entries), but for a newcomers, it might look insane.
People are now used to search engines when looking at documentation.
Most new developpers I know won’t even bother buying physical books.

> This should be discussed on the Texinfo list, not here.  The current
> Texinfo translator is highly customizable, so it could be that it can
> be adapted to these needs much better.

Since we were talking about Emacs, I thought this thread was more
appropriate.  I was refferring to the difficulty of porting GNU Emacs
Lisp documentation to the Dash docset format.

[1] https://kapeli.com/docsets#supportedentrytypes

Re: docstrings and elisp reference

[Eli Zaretskii]

> From: Etienne Prud’homme <e.e.f.prudhomme@gmail.com>
> Cc: stephen_leake@stephe-leake.org,  drew.adams@oracle.com,  emacs-devel@gnu.org
> Date: Thu, 08 Jun 2017 14:10:13 -0400
> 
> > What's more, at least the docs it has for Bash and ELisp are exactly
> > the respective Texinfo manuals, with the same text and the same node
> > structure.  What is the advantage of using a different browser for the
> > exact same text?
> 
> That’s my point.  I must confess that the available documentation for
> Elisp has better support now for choosing whether we search for a macro,
> command, function, etc.  It used to index every symbols in the same
> entry type (variable entry).

Not sure I understand what is it that you allude to.  We always used
different Texinfo commands for different kinds of symbols: @defmac for
macros, @defvar for variables, @defun for functions, "@deffn Command"
for commands, @defspec for special forms, etc.

> Furthermore, we could support much more indexation capabilities given
> the list of supported entry types[1].

If Texinfo lacks some of these, they could easily be added as the
language continues to evolve and being developed.  I see no
fundamental problems here, just something to code.

> What I meant was really the uniform search interface it provides.  For
> advanced Emacs users, it’s not a problem looking what we want from Emacs
> itself (or Info entries), but for a newcomers, it might look insane.

We _are_ in an Emacs forum, where the Emacs way of doing things is the
"normal" way.  And I definitely wouldn't say that the Emacs help
system is "insane" by any measure.  E.g., it offers hyperlinks that
everyone is accustomed to.

> People are now used to search engines when looking at documentation.

Did you try giving a list of keywords to the Emacs apropos commands?
That feature was added in Emacs 23, I think.

IOW, Emacs uses search-engine techniques as well, always did.  You
cannot find your way through hundreds of pages of manuals without some
reasonably smart indexing and searching facilities.

> > This should be discussed on the Texinfo list, not here.  The current
> > Texinfo translator is highly customizable, so it could be that it can
> > be adapted to these needs much better.
> 
> Since we were talking about Emacs, I thought this thread was more
> appropriate.  I was refferring to the difficulty of porting GNU Emacs
> Lisp documentation to the Dash docset format.

I still feel I don't understand the nature of that difficulty.

RE: docstrings and elisp reference

[Drew Adams]

> Did you try giving a list of keywords to the Emacs apropos commands?
> That feature was added in Emacs 23, I think.

(Emacs 22.)

Re: docstrings and elisp reference

[Etienne Prud’homme]

I want to make it clear I’m not questionning the usefulness of Texinfo,
nor I am criticizing the Emacs documentation system.

> Not sure I understand what is it that you allude to.  We always used
> different Texinfo commands for different kinds of symbols: @defmac for
> macros, @defvar for variables, @defun for functions, "@deffn Command"
> for commands, @defspec for special forms, etc.

That’s great and I hope we continue to use this system.

> We _are_ in an Emacs forum, where the Emacs way of doing things is the
> "normal" way.  And I definitely wouldn't say that the Emacs help
> system is "insane" by any measure.  E.g., it offers hyperlinks that
> everyone is accustomed to.

Far from me to say that the Emacs way is insane!  Forgive me father Eli.
I’m only pointing out that Emacs normal way of doing things is pretty
different from editors like Atom or many new (hackable) editors used.
I’m only talking about documentation here.  Whether Emacs way is better,
I’m probably too biased to tell, but people coming from those editors
are not used to _some_ of the documentation interface we provide.

That being said, I can bodly say that the Elisp printed manual is much
better than any other text-editors or program API I know.

The original subject of this thread was “docstrings and elisp
reference” i.e. JavaDoc comments style.  I’m thinking about ways we
could add relevant semantic information in either source.  We already
have support for some of that in docstrings i.e. ‘\\[]’ or ‘\\<>’.

I’m sorry if it looked like I was addressing other points.

> I still feel I don't understand the nature of that difficulty.

Looking at the source of the generated HTML documents, I think I might
have confused porting other GNU documentations.  I once tried to make
the Autoconf documentation available on Zeal at almost the same time as
I tried with Elisp.

Jean-Christophe seems to be right about using more the HTML conversion
templates.  Some of the semantic tagging is lost in the HTML conversion
and that’s where the difficulty was.

--
Etienne

Re: docstrings and elisp reference

[Jean-Christophe Helary]

> On Jun 9, 2017, at 5:22, Etienne Prud’homme <e.e.f.prudhomme@gmail.com> wrote:
> 
> Jean-Christophe seems to be right about using more the HTML conversion
> templates.  Some of the semantic tagging is lost in the HTML conversion
> and that’s where the difficulty was.

As I mentioned, there are ways to maintain compatibility with old HTML tags and systematically add CSS hooks to *every* texinfo tag and more (for ex. splitting the defun argument list to separate arguments from identifiers like &optional and &rest, but there are other things too).

And if we can do the same for the docstrings and link the two together, we'll have a really powerful way to find information and learn in html too.

Jean-Christophe 

Re: docstrings and elisp reference

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > It’s worth mentioning that JavaDoc documentation style is particularly
  > useful for Object-Oriented Programming and strongly typed languages.  I
  > think it’s even mandatory with OOP to be readable.

I could be mistaken, but I think that Javadoc is comparable to Emacs
Lisp doc strings.

If that is correct, then of course Javadoc documentation is useful.
Emacs Lisp doc strings are useful too.  But concatenating them does
not make a clear tutorial/reference manual.  We need both!

  > However, I also think Jean-Christophe makes a good point about
  > documentation generation.  Not with duplication, but semantic support.

What is "semantic support"?

  > Those tools are highly effective for semantic indexation for newcomers

What is that?

  > since they offer a simple interface for hundred FLOSS libraries.

Could you explain what "a simple interface" to a library means?
Normally each library defines its own interface.

								      GNU
  > projects are almost nonexistent.

You can't mean that literally, so what do you mean?

Maybe "semantic support" should be added to Emacs Lisp,
but until I know what it means, I can't have an opinion.

  > I’ve been trying in the past to port GNU projects documentation and I
  > finally gave up.  I find Texinfo to be very limited when it comes to
  > semantic support.  It’s really hard to extract meaningful definitions
  > from texi files.

I think what you are trying to do does not make sense.  You can't
extract the full description of a single function from a good manual
because a good manual does not try to describe each function
separately.  What you want for this is something like a doc string,
and a good manual is not like a collection of doc strings.


-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: docstrings and elisp reference

[Etienne Prud’homme]

Richard Stallman <rms@gnu.org> writes:

> I could be mistaken, but I think that Javadoc is comparable to Emacs
> Lisp doc strings.

Emacs Lisp doc strings have support for some of JavaDoc features, but
cannot compare in many things.  Many things described in doc strings are
informal and are pretty difficult to extract.  For example, we have no
formal way of telling:

- the type of the function return value
- what exceptions can be thrown from the function
- does this function has side effects
- since when it was introduced
- it’s a hook variable
- if the comment includes a file system path in the example

And many other things I don’t have in mind now.

Of course, some of those things are directly addressed with options
variables.

> If that is correct, then of course Javadoc documentation is useful.
> Emacs Lisp doc strings are useful too.  But concatenating them does
> not make a clear tutorial/reference manual.  We need both!
>
>   > However, I also think Jean-Christophe makes a good point about
>   > documentation generation.  Not with duplication, but semantic support.

> What is "semantic support"?

It is more or less tagging useful information about the content
generated.  Particularly useful when doing indexation of either comments
or the documentation.  As I said another response, _I have mistaken_
some other GNU tools that I tried to port at the same time as the Emacs
Lisp manual.

I think it could apply to both the reference manual and the doc strings
in general.
								      
>   > GNU projects are almost nonexistent.
>
> You can't mean that literally, so what do you mean?

The tool I described is an offline documentation viewer.  The docset is
what is interesting.  What we usually call docset is a tarball with the
documentation and an index tagging the documentation.  The index is a
standardized SQLite table with a restricted set of identifiers and their
classes.  There are several programs allowing searching this
documentation standard.  Zeal is an example, but the _helm-dash_ MELPA
package also offers a basic interface for searching the docs.

In order to make a docset, we can use documentation generators like
JavaDoc, Doxygen, etc.  We can also use the online documentation
provided by the project.  This way, the documentation is either fetched
from the website (like Mozilla Developer Network) or the HTML version of
a project is taken.

To make indexation, we generally make small programs to record all
functions/macros/variables/... and their reference in an HTML document.
This way, the index can easily be updated when the documentation changes
without having to review each references.

Therefore, for an index to be created, we either have to use Natural
Language Processing (which we never do) or use some kind of clues in the
HTML document describing the type of information to index.

The perfect way would be to use XML directly, but few or no projects use
it.  We instead rely on the HTML tag class names from the documentation
documents.  Almost every time, they can be really helpful to know the
type of the tagged information.  Most developpers making the CSS
stylesheets use names like “function” (or an other arbitrary name) when
tagging the functions.  That’s the same for variables, macros, etc.

That is where _most_ GNU projects seem to lack the capability.  It looks
like Jean-Christophe was right in pointing out it might be because we’re
not using HTML exporting templates at its best.

GNU projects are usually harder to index because of that.  Thus we have
a minority of GNU tools listed.  I’m not saying that for criticizing
GNU, but I think it’s an important issue we could address in the near
future.

Of course, this discussion is about Emacs.  I hope I clarified some of
what I said.

--
Etienne

Re: docstrings and elisp reference

[Eli Zaretskii]

> From: Etienne Prud’homme <e.e.f.prudhomme@gmail.com>
> Date: Fri, 09 Jun 2017 01:20:26 -0400
> Cc: stephen_leake@stephe-leake.org, drew.adams@oracle.com, emacs-devel@gnu.org

I think I've said enough in this thread, so just 2 more comments:

> That is where _most_ GNU projects seem to lack the capability.  It looks
> like Jean-Christophe was right in pointing out it might be because we’re
> not using HTML exporting templates at its best.
> 
> GNU projects are usually harder to index because of that.  Thus we have
> a minority of GNU tools listed.  I’m not saying that for criticizing
> GNU, but I think it’s an important issue we could address in the near
> future.

Texinfo can produce HTML (which can be customized) and XML from the
Texinfo sources.  These facilities can be used to enhance the
"indexing" (I'd call it "tagging" instead, to discern from the indices
we have in the manuals) of the produced documentation, if that is the
feature you think is missing in the Info manuals.  Then preparing a
document in the structure you want for any GNU project should be very
easy.  It's possible that the Texinfo language might need some
additions to support some of that, but that should be no problem,
given the active development of the Texinfo package.

> Of course, this discussion is about Emacs.

And therein is my second issue: reading the Bash and the ELisp manuals
in Zeal, I'm presented with exactly the same contents as when I read
them in Emacs's Info.  IOW, what I have in Zeal is an Info browser
without the index-searching facilities which make Info so efficient
for me.  So for me, and especially in the context of Emacs, it makes
very little sense to talk about these browsers, because they seem to
give be a subset of what we already have in Info.  Perhaps I'm missing
something important.

Re: docstrings and elisp reference

[Yuri Khan]

On Fri, Jun 9, 2017 at 2:08 PM, Eli Zaretskii <eliz@gnu.org> wrote:

> And therein is my second issue: reading the Bash and the ELisp manuals
> in Zeal, I'm presented with exactly the same contents as when I read
> them in Emacs's Info.  IOW, what I have in Zeal is an Info browser
> without the index-searching facilities which make Info so efficient
> for me.  So for me, and especially in the context of Emacs, it makes
> very little sense to talk about these browsers, because they seem to
> give be a subset of what we already have in Info.  Perhaps I'm missing
> something important.

The important thing you’re missing is availability of other,
non-Texinfo documentation sets, in that same browser. Being able to
browse all necessary documentation in one unified browser is big.
Especially if that one browser can be invoked by pressing a single key
in Emacs and will search the right docset(s) depending on your major
mode.

Re: docstrings and elisp reference

[Eli Zaretskii]

> From: Yuri Khan <yuri.v.khan@gmail.com>
> Date: Fri, 9 Jun 2017 15:27:30 +0700
> Cc: Etienne Prud’homme <e.e.f.prudhomme@gmail.com>, 
> 	Stephen Leake <stephen_leake@stephe-leake.org>, "rms@gnu.org" <rms@gnu.org>, 
> 	Drew Adams <drew.adams@oracle.com>, Emacs developers <emacs-devel@gnu.org>
> 
> The important thing you’re missing is availability of other,
> non-Texinfo documentation sets, in that same browser.

But this discussion's context was GNU projects, all of which have
Texinfo manuals.  (And Richard is just finishing his work on a manual
that describes the C language, so this, too, will be available soon
enough, let's hope.)

Re: docstrings and elisp reference

[Eli Zaretskii]

> Date: Fri, 09 Jun 2017 12:38:14 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: e.e.f.prudhomme@gmail.com, stephen_leake@stephe-leake.org, rms@gnu.org,
> 	drew.adams@oracle.com, emacs-devel@gnu.org
> 
> > From: Yuri Khan <yuri.v.khan@gmail.com>
> > Date: Fri, 9 Jun 2017 15:27:30 +0700
> > Cc: Etienne Prud’homme <e.e.f.prudhomme@gmail.com>, 
> > 	Stephen Leake <stephen_leake@stephe-leake.org>, "rms@gnu.org" <rms@gnu.org>, 
> > 	Drew Adams <drew.adams@oracle.com>, Emacs developers <emacs-devel@gnu.org>
> > 
> > The important thing you’re missing is availability of other,
> > non-Texinfo documentation sets, in that same browser.
> 
> But this discussion's context was GNU projects, all of which have
> Texinfo manuals.

And btw, I very much doubt that the (apparent) vision of having all,
or even most of the documentation available in a single format is
practically attainable anyway.  Unless that format is HTML or some
variant of it, because most projects provide their documentation in
that format, and Web browsers are ubiquitous.  Why should we believe
that the Dash/Zeal docset format will win the world?

Re: docstrings and elisp reference

[Yuri Khan]

On Fri, Jun 9, 2017 at 7:21 PM, Eli Zaretskii <eliz@gnu.org> wrote:

> And btw, I very much doubt that the (apparent) vision of having all,
> or even most of the documentation available in a single format is
> practically attainable anyway.  Unless that format is HTML or some
> variant of it, because most projects provide their documentation in
> that format, and Web browsers are ubiquitous.

Dash/Zeal’s format *is* HTML, for the most part.

> Why should we believe that the Dash/Zeal docset format will win the world?

Perhaps it won’t. Perhaps it will evolve into something better. Or it
will force something else to evolve into something better.

Re: docstrings and elisp reference

[Eli Zaretskii]

> From: Yuri Khan <yuri.v.khan@gmail.com>
> Date: Fri, 9 Jun 2017 19:32:13 +0700
> Cc: Etienne Prud’homme <e.e.f.prudhomme@gmail.com>, 
> 	Stephen Leake <stephen_leake@stephe-leake.org>, "rms@gnu.org" <rms@gnu.org>, 
> 	Drew Adams <drew.adams@oracle.com>, Emacs developers <emacs-devel@gnu.org>
> 
> On Fri, Jun 9, 2017 at 7:21 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
> > And btw, I very much doubt that the (apparent) vision of having all,
> > or even most of the documentation available in a single format is
> > practically attainable anyway.  Unless that format is HTML or some
> > variant of it, because most projects provide their documentation in
> > that format, and Web browsers are ubiquitous.
> 
> Dash/Zeal’s format *is* HTML, for the most part.

I don't know what you mean by "for the most part", but since it cannot
simply display a GNU manual converted to HTML by texi2any, I guess
something else is needed that a docset needs to provide that isn't
readily available.  So packages need to _want_ to provide this format,
otherwise their docs will not be viewable in Zeal.

Re: docstrings and elisp reference

[Etienne Prud’homme]

Eli Zaretskii <eliz@gnu.org> writes:

> I don't know what you mean by "for the most part", but since it cannot
> simply display a GNU manual converted to HTML by texi2any, I guess
> something else is needed that a docset needs to provide that isn't
> readily available.  So packages need to _want_ to provide this format,
> otherwise their docs will not be viewable in Zeal.

You previously said that you had enough of this discussion and I would
understand if you don’t read it.  I explained the process in a response
to RMS:

> Therefore, for an index to be created, we either have to use Natural
> Language Processing (which we never do) or use some kind of clues in
> the HTML document describing the type of information to index.

> The perfect way would be to use XML directly, but few or no projects
> use it.  We instead rely on the HTML tag class names from the
> documentation documents.  Almost every time, they can be really
> helpful to know the type of the tagged information.  Most developpers
> making the CSS stylesheets use names like “function” (or an other
> arbitrary name) when tagging the functions.  That’s the same for
> variables, macros, etc.

But as I said also in the same post, it looks like Jean-Christophe is
right in saying we don’t use the exporting templates enough.  The texi
files seems to have a lot of meaningful “tags” (I call that semantic) we
could use when exporting the manual documentation to HTML.  Why not even
use JavaScript to enable searching the manuals and/or
expanding/collapsing setions.

A good example of what could be done is the ECMAScript 2018 Language
Specifications[1].

[1] https://tc39.github.io/ecma262/ (WARNING: It looks a JS file doesn’t
contain any license information for LibreJS users, but the blocked
script is licensed under MIT).

--
Etienne

Re: docstrings and elisp reference

[Jean-Christophe Helary]

> On Jun 10, 2017, at 4:24, Etienne Prud’homme <e.e.f.prudhomme@gmail.com> wrote:
> 
> But as I said also in the same post, it looks like Jean-Christophe is
> right in saying we don’t use the exporting templates enough.  The texi
> files seems to have a lot of meaningful “tags” (I call that semantic) we
> could use when exporting the manual documentation to HTML.  Why not even
> use JavaScript to enable searching the manuals and/or
> expanding/collapsing setions.

I'd like to point out that in the 2014 thread that I was referred to earlier, RMS asked whether it would be possible to implement some display features (display/hide nodes, etc.) with HTML and Stephen Turnbull replied that, well, yes it was possible and relatively easy.

https://lists.gnu.org/archive/html/emacs-devel/2014-12/msg01911.html

Reading the whole thread shows that there was already some kind of consensus back in 2014 about what we could do with the HTML output (ie much more than what we do now) and how we could/should improve it.

So, my personal conclusion for the current thread is that:

1) We need to improve the texinfo export templates for HTML (I have no opinion for the other templates, but it could be the same) so that *no* information is lost in the conversion process and possible some information is added.

2) the i18n infrastructure that is very slowly emerging will allow for extraction of docstrings along with messages, so that we'll eventually be able to use that to create a "javadoc" like output for the docstrings with the information that Etienne mentioned earlier:

> Emacs Lisp doc strings have support for some of JavaDoc features, but
> cannot compare in many things.  Many things described in doc strings are
> informal and are pretty difficult to extract.  For example, we have no
> formal way of telling:
> 
> - the type of the function return value
> - what exceptions can be thrown from the function
> - does this function has side effects
> - since when it was introduced
> - it’s a hook variable
> - if the comment includes a file system path in the example
> 
> And many other things I don’t have in mind now.

3) When we have 1) and 2) we can have a fully interlinked HTML documentation set that could be used as a standard format for Emacs (as was discussed in 2014) *and* as a base format for all sorts of documentation display applications that use HTML as the standard, so that Emacs documentation system features are not limited to the Emacs application but are literally all over the place.

Jean-Christophe

Re: docstrings and elisp reference

[Nikolay Kudryavtsev]

What's important to understand here is that docstring documentation is 
of very limited use. This applies to both Emacs docstrings and 
everything created by javadoc and its cousins.

Let's say I want to do something with an API that I have no prior 
knowledge of and I have only the docstring documentation. I start by 
searching docstrings for something that looks vaguely useful for solving 
the problem at hand. Would the first thing(be it a class, or a function) 
I found be the optimal solution? Maybe there's actually a more 
specialized version of this function that better suits my case. Or maybe 
it's the opposite - I stumbled into a specialized function, where the 
broader one is more suited for me. There's no way to know until you read 
everything that looks even slightly important.

Because of this, the documentation needs some glue, that puts separate 
elements into a proper context. Docstrings are incapable of solving this 
problem yet. You can't just have docstrings that go "if you find this 
function useful, maybe you actually need this-other-function or maybe 
this-totally-other-function".

The key word here is _discoverability_. Because of this, you can't just 
dispose for example of Elisp manual, since it's exactly that rug that 
ties the room together.

I'm not saying that it's impossible to build a better documentation 
system than we currently have in info. It's just a massive investment 
that would at best be a very slight improvement.

As a practical example I installed Zeal and downloaded MySQL manual for 
it. It has different indexes, but there is no table of contents(1). 
Let's say I want to calculate a logarithm. I go by function index and 
find one of the log functions. Luckily that's the full MySQL Reference 
Manual and not some javadoc abomination and it provides all log 
functions within the same context, and that within a broader context of 
math functions. Also I can't search the whole manual at once, only 
indexes or a single chapter(2). And it's worth noting that the search is 
just a dumb text search(just as in info) and not some natural language 
processing one as proposed by Etienne Prud’homme in this thread. So, due 
to 1 and 2 Zeal is not superior to info in usability, while 
discoverability is a feature of the material and not presentation.

-- 
Best Regards,
Nikolay Kudryavtsev

RE: docstrings and elisp reference

[Drew Adams]

> What's important to understand here is that docstring documentation
> is of very limited use. This applies to both Emacs docstrings and
> everything created by javadoc and its cousins.
> 
> Let's say I want to do something with an API that I have no prior
> knowledge of and I have only the docstring documentation. I start by
> searching docstrings for something that looks vaguely useful for solving
> the problem at hand. Would the first thing(be it a class, or a function)
> I found be the optimal solution?
>
> Maybe there's actually a more
> specialized version of this function that better suits my case. Or maybe
> it's the opposite - I stumbled into a specialized function, where the
> broader one is more suited for me. There's no way to know until you read
> everything that looks even slightly important.
> 
> Because of this, the documentation needs some glue, that puts separate
> elements into a proper context. Docstrings are incapable of solving this
> problem yet. You can't just have docstrings that go "if you find this
> function useful, maybe you actually need this-other-function or maybe
> this-totally-other-function".
> 
> The key word here is _discoverability_. Because of this, you can't just
> dispose for example of Elisp manual, since it's exactly that rug that
> ties the room together.

+1.  Plus: the source code - in particular, Lisp.
That's the glue behind the glue and the single source of
information and truth.  (Well, not quite single - see C.)

A typical such quest might lead from one or more doc strings
to the Elisp manual, and then to source code - to find the
most useful functions and variables to use, or to find parts
of the code that are enlightening or useful for constructing
new code for the job.

Or it might lead from the Elisp manual to one or both of
the other sources of such info.  Or it might lead from
a single doc string to the source code.  Or back & forth.

All three are useful: doc strings, Elisp manual, source
code.  They overlap, but they also complement each other,
sometimes in terms of information content but mainly in
terms of presentation, audience, and means of access.
It's all in the source code, but the other, help, features
often facilitate discoverability and improve understanding.

Similarly, other help commands and functions: apropos etc.

Yes, Java source code is also consultable.  But I think
that the boost that you get from combining the various
information sources that Emacs offers is something more.

I don't think that JavaDoc (especially typical JavaDoc!)
plus source code gives you what you get with Emacs.

Partly that is due to Lisp being what Lisp is, perhaps,
but it is also probably due to the ubiquitous and
fine-scale attachment of doc/help to code and even to
Lisp symbols.

The comment above about "typical JavaDoc" is really
meant to emphasize this other aspect of Emacs help:

It is good not just because the constructs of doc
strings and manual _exist_ but because key people
in Emacs development have long insisted on their
importance.  Just being able to attach doc strings
or create JavaDoc is not enough.  You also need a
commitment to using them to create good help.

Good help does not fall from the sky.  It does not
come for free, generated from the source code.

Why?  Because part of the job of coming up with
good help involves addressing the possible
audiences, contexts and use cases.  Unless and
until info about that is encoded in the source
code, it needs to be added.  (And encoding it in
the source code would involve the same "user
documentation" work.) 

IMHO, that commitment is the most important factor -
the glue behind the glue.  Let's hope it continues.

There is no substitute for careful human time and
clear eyes devoted to help/doc.  No degree of
"reuse" or generation of deliverables can substitute
for that.  Tools are fine, but they don't replace
paying attention.

Re: docstrings and elisp reference

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Emacs Lisp doc strings have support for some of JavaDoc features, but
  > cannot compare in many things.  Many things described in doc strings are
  > informal and are pretty difficult to extract.  For example, we have no
  > formal way of telling:

  > - the type of the function return value
  > - what exceptions can be thrown from the function
  > - does this function has side effects
  > - since when it was introduced
  > - it’s a hook variable
  > - if the comment includes a file system path in the example

What this amounts to is that they are basically similar
but JavaDoc has additional features.

Those that make sense to add, we can add to the doc strings.

  > - the type of the function return value

That is not crucial in Lisp  the way it is in Java.
Unlike Java, Lisp does not declare or limit the type of any values.

In some cases we document what the type will actually be,
using English text.

  > - what exceptions can be thrown from the function

In general, there is no limit to what exceptions can be thrown.
In Lisp, that is dynamic.

  > - does this function has side effects

In general, any Lisp function can have side effects.

We could define a more systematic way to indicate in doc strings that a
certain function has no side effects, that you can assume it preserves
the current buffer, or that it preserves the search results.
And whatever else could be useful.

  > - since when it was introduced

That does not seem important to me.

  > - it’s a hook variable

We could define a systematic way to indicate this in doc strings.

  > - if the comment includes a file system path in the example

I don't understand what that means.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: docstrings and elisp reference

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Date: Fri, 09 Jun 2017 23:19:48 -0400
> Cc: stephen_leake@stephe-leake.org, drew.adams@oracle.com, emacs-devel@gnu.org
> 
>   > - the type of the function return value
> 
> That is not crucial in Lisp  the way it is in Java.
> Unlike Java, Lisp does not declare or limit the type of any values.

It could be important for Texinfo in general, though, because it is
used to describe other languages and facilities where this does
matter.

Likewise for other attributes you mention as unimportant for Lisp.

>   > - since when it was introduced
> 
> That does not seem important to me.

But we already do have this information for customizable variables at
least.

>   > - it’s a hook variable
> 
> We could define a systematic way to indicate this in doc strings.

I think the names of hook variables always match a few specific
regular expressions, so this information is already there.

Re: docstrings and elisp reference

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > >   > - the type of the function return value
  > > 
  > > That is not crucial in Lisp  the way it is in Java.
  > > Unlike Java, Lisp does not declare or limit the type of any values.

  > It could be important for Texinfo in general,

I was responding to a proposed feature for Lisp doc strings (imitating
Javadoc).  Texinfo is a different issue.

Texinfo already has a standard way to indicate the return type of a
function, used in the C Library manual.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: docstrings and elisp reference

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > >   > - since when it was introduced
  > > 
  > > That does not seem important to me.

  > But we already do have this information for customizable variables at
  > least.

There are special reasons for wanting that for customization features.

  > > We could define a systematic way to indicate this in doc strings.

  > I think the names of hook variables always match a few specific
  > regular expressions, so this information is already there.

There are a few old hook variables that we did not rename to follow
the standard naming convention.  We could add those specific names
as alternatives in the regexp; then it would be 100% accurate.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: docstrings and elisp reference

[Stefan Monnier]

> There are a few old hook variables that we did not rename to follow
> the standard naming convention.

IIRC I tried to rename them all (tho still supporting the old names,
marked as obsolete).  If you find some that are not marked as obsolete,
please report them,


        Stefan

Re: docstrings and elisp reference

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > IIRC I tried to rename them all (tho still supporting the old names,
  > marked as obsolete).  If you find some that are not marked as obsolete,
  > please report them,

The current discussion is about documenting reliably whether each
variable is a hook or not.  The old name is probably obsolete, but a
user might enter it in C-h v.  How will we say whether it is
a hook?

If C-h v can see it is an alias and look at the new name, that would
be a way to handle this.
-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: docstrings and elisp reference

[Stefan Monnier]

> The current discussion is about documenting reliably whether each
> variable is a hook or not.  The old name is probably obsolete, but a
> user might enter it in C-h v.  How will we say whether it is
> a hook?

C-h v will say that it's obsolete and point to the new name which
indicates clearly it's a hook.


        Stefan

Re: docstrings and elisp reference

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > C-h v will say that it's obsolete and point to the new name which
  > indicates clearly it's a hook.

Maybe that is good enough.  I am not sure.

Someone praised Javadoc for indicating that sort of thing in a
mechanical way.  If we need that, we could implement it based on the
data that C-h v looks at, but C-h v itself won't do the job.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: docstrings and elisp reference

[Yuri Khan]

On Sat, Jun 10, 2017 at 10:19 AM, Richard Stallman <rms@gnu.org> wrote:

>   > - the type of the function return value
>
> That is not crucial in Lisp  the way it is in Java.
> Unlike Java, Lisp does not declare or limit the type of any values.
>
> In some cases we document what the type will actually be,
> using English text.

And that documentation is immensely helpful.

Moreover, I will hazard a guess that any attempt to shoehorn that
documentation into a formal type description language will limit its
usefulness to readers.

As anecdotal evidence, I will offer an example. XML is a hierarchical
dynamically-typed data structure, very similar to Lisp S-expressions
in terms of capabilities. In order to be able to document and validate
permissible constructs, people devised not one but two formal
languages: DTD and XML Schema. Of these, the latter is itself
XML-based, and thus very appealing to implementors.

The resulting schema documents are very verbose and hard to read, even
for a trained professional. The programming equivalent of legalese.

Re: docstrings and elisp reference

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > What is "semantic support"?

  > It is more or less tagging useful information about the content
  > generated.  Particularly useful when doing indexation of either comments
  > or the documentation.

Sorry, I still don't understand what the idea is.

  > The tool I described is an offline documentation viewer.  The docset is
  > what is interesting.  What we usually call docset is a tarball with the
  > documentation and an index tagging the documentation.

I don't really understand this, and why isn't Emacs a good way to view
Emacs doc strings interactively?

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: docstrings and elisp reference

[Joost Kremers]

On Tue, Jun 06 2017, Stephen Leake wrote:
> Doc strings are for short reference information for single 
> functions/variables.
>
> texinfo docs are for more indepth explanations of several 
> related functions/features.
>
> They are complementary; they should not duplicate each other.

And yet, the Elisp manual documents a large number of functions 
and variables / options in a format that is very reminiscent of 
doc strings and which provides basically the same information. I 
kinda see the point in making the doc string and that description 
identical. Obviously, the manual gives a lot more background info 
and groups the functions and variables together in a meaningful 
way, but wouldn't it make sense / be possible to extract the 
descriptions in the manual directly from the source?


-- 
Joost Kremers
Life has its moments

Re: docstrings and elisp reference

[Stefan Monnier]

> And yet, the Elisp manual documents a large number of functions and
> variables / options in a format that is very reminiscent of doc strings and
> which provides basically the same information.

Agreed.  And I also think this is a problem.
From where I stand, the way to fix it is:
- Better integration between the manual browser and the docstring
  browser, so you can click on a function/variable name in the manual to
  get to its docstring and you can easily jump from a docstring to the
  relevant section of the manual.
- Remove the redundant var/fun documentation from the manual (this is
  a tedious job: there is some redundance but it's not systematic, so
  we can't just do this removal systematically).
- Come up with a way to re-add those var/fun documentation into the
  printed manual (e.g. by adding to the Texinfo source external
  references to docstrings, which are then processed by now ad-hoc
  script).


        Stefan

Re: docstrings and elisp reference

[Yuri Khan]

On Wed, Jun 7, 2017 at 8:03 PM, Stefan Monnier <monnier@iro.umontreal.ca> wrote:

> - Better integration between the manual browser and the docstring
>   browser, so you can click on a function/variable name in the manual to
>   get to its docstring and you can easily jump from a docstring to the
>   relevant section of the manual.

In principle, you already can. Being able to press just RET or click
instead of <f1> o RET would be nice, though.

Re: docstrings and elisp reference

[Stefan Monnier]

>> - Better integration between the manual browser and the docstring
>> browser, so you can click on a function/variable name in the manual to
>> get to its docstring and you can easily jump from a docstring to the
>> relevant section of the manual.
> In principle, you already can. Being able to press just RET or click
> instead of <f1> o RET would be nice, though.

Of course `C-h o RET` works OK most of the time, but it's not the same
as having the var/fun name highlighted as a link that you can click.
We could even have a way to dynamically bring the docstring into the
Info manual (the same external-refs that allows an ad-hoc script to add
the docstrings to the printed version could be used to dynamically
insert the docstring into the *Info* buffer on-the-fly).


        Stefan

RE: docstrings and elisp reference

[Drew Adams]

> - Better integration between the manual browser and the docstring
>   browser, so you can click on a function/variable name in the manual to
>   get to its docstring and you can easily jump from a docstring to the
>   relevant section of the manual.

Yes, 100%.  I've long argued that.  Cross-reference, not text reuse.

And Emacs has already come a fair distance in that direction.  It
could still be improved.

> - Remove the redundant var/fun documentation from the manual (this is
>   a tedious job: there is some redundance but it's not systematic, so
>   we can't just do this removal systematically).

We should not remove redundant _information_, if there is a
reason that it will help readers.  And that's the case for 99.9%
of the var/fun presentations in the manual.  There is often a
discussion about whether a given var/fun should be explicitly
doc'd in the manual - it's a judgment call.  It's not because
we _can_ add the doc for a var/fun to the manual that we should.

Think redundancy NOT for maintainers (I'm not saying you did that)
but for readers.

A reader should be _able_ to click to go to additional info
or to a different presentation/orientation of the same material.

But it is important, IMO, for the manual to have the information
that it has, even when that means that the same information is
provided also elsewhere (e.g. in a doc string).  We lose, do not
gain, if we decide to remove the doc for a given function etc.
from the manual just because it has a doc string - or vice versa.

The question of redundancy _for the user_ means that we also do
not, in general, want to have exactly the same text in doc string
and manual.  A user does not want to click a function name in one
or the other, only to end up seeing exactly the same text in the
other location.

"Reuse" for readers is generally a negative - it means reading
the same thing again, generally without being forewarned.
Little is more frustrating for a reader.  ("Why did you send me
here?  This is the same text!")

That is not to say that repetition in doc is always negative.
Far from it.  It's about judgment: repeat when there is a good
(e.g. pedagogical) reason to repeat.

> - Come up with a way to re-add those var/fun documentation into the
>   printed manual (e.g. by adding to the Texinfo source external
>   references to docstrings, which are then processed by now ad-hoc
>   script).

Not a good idea, IMO.  See above.

Re: docstrings and elisp reference

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Wed, 07 Jun 2017 09:03:04 -0400
> 
> >From where I stand, the way to fix it is:
> - Better integration between the manual browser and the docstring
>   browser, so you can click on a function/variable name in the manual to
>   get to its docstring and you can easily jump from a docstring to the
>   relevant section of the manual.
> - Remove the redundant var/fun documentation from the manual (this is
>   a tedious job: there is some redundance but it's not systematic, so
>   we can't just do this removal systematically).
> - Come up with a way to re-add those var/fun documentation into the
>   printed manual (e.g. by adding to the Texinfo source external
>   references to docstrings, which are then processed by now ad-hoc
>   script).

FWIW, I disagree with this direction.  I think we will end up with a
much lower-level of documentation if we go that way.  I also very much
doubt we will find a volunteer to perform the 2nd item of that list:
it's a very large and ungrateful job, and some of the text to be
removed is IMNSHO excellent prose which it would be pity to lose.

Re: docstrings and elisp reference

[Dmitry Gutov]

On 6/6/17 11:25 PM, Stephen Leake wrote:

> they should not duplicate each other.

Then you have something else in mind than the current situation with 
documentation in Emacs.

RE: docstrings and elisp reference

[Drew Adams]

> > they should not duplicate each other.
> 
> Then you have something else in mind than the current
> situation with documentation in Emacs.

No, you do, if you think the intention now is that
they duplicate each other.  It never has been.

Some of the information is often the same; that's all.

The presentation and context of use are different.
The level of detail is often different.  (Sometimes
there is more detail in the manual; sometimes there
is more in a doc string.)

In particular, doc strings are written as user help.
The interactive use, if any, is typically described
first, and from the point of an interactive user.
Not so, the Elisp manual.

The doc string for a command tells a user what it
does and how a prefix arg changes the behavior.
In many cases the user need have no little or no
knowledge of Emacs Lisp.

The Elisp doc for a command tells a Lisp programmer
what the function and its parameters are, and
describes their behavior.  It might or might not
mention that one of the parameters corresponds to
the prefix arg when called interactively.

Different contexts; different purposes; not
necessarily the same readers or readers with the
same immediate goal.

Re: docstrings and elisp reference

[Dmitry Gutov]

On 6/7/17 12:21 AM, Drew Adams wrote:
>>> they should not duplicate each other.
>>
>> Then you have something else in mind than the current
>> situation with documentation in Emacs.
> 
> No, you do, if you think the intention now is that
> they duplicate each other.  It never has been.

Please look up the difference between "intention" and "current situation".

> Some of the information is often the same; that's all.

That's called "duplication".

> In particular, doc strings are written as user help.
> The interactive use, if any, is typically described
> first, and from the point of an interactive user.
> Not so, the Elisp manual.

We have different manuals, with different goals.

The fact that docstrings often describe the interactive case (when it 
exists) doesn't make them necessarily targeted at the end user. Just 
exhaustive.

Re: docstrings and elisp reference

[Eli Zaretskii]

> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Wed, 7 Jun 2017 00:50:23 +0300
> 
> On 6/7/17 12:21 AM, Drew Adams wrote:
> >>> they should not duplicate each other.
> >>
> >> Then you have something else in mind than the current
> >> situation with documentation in Emacs.
> > 
> > No, you do, if you think the intention now is that
> > they duplicate each other.  It never has been.
> 
> Please look up the difference between "intention" and "current situation".

There is a difference, that's true.  But the fact that reality is
different from the ideal doesn't mean we should give up the ideal, at
least not lightly.  And we certainly should consider whether the
alternative proposal will produce a far worse situation than what we
have today.  Once again, I suggest that you take a good look at
manuals of GnuTLS and Guile, they are produced using the methodology
that you propose.  That is our future if we go that way.  Do you
really like the result?

> > Some of the information is often the same; that's all.
> 
> That's called "duplication".

You are overloading the meaning of "duplication".  The original
proposal was for a literal copying of the same text.  Drew was talking
about duplicating the information, but expressed in a different,
sometimes very different, form.

> > In particular, doc strings are written as user help.
> > The interactive use, if any, is typically described
> > first, and from the point of an interactive user.
> > Not so, the Elisp manual.
> 
> We have different manuals, with different goals.
> 
> The fact that docstrings often describe the interactive case (when it 
> exists) doesn't make them necessarily targeted at the end user. Just 
> exhaustive.

This doesn't really resolve the issue pointed out by Drew.  And that
is only one of the issues, there are others, also valid ones.

Re: docstrings and elisp reference

[Dmitry Gutov]

On 6/7/17 8:28 AM, Eli Zaretskii wrote:
> There is a difference, that's true.  But the fact that reality is
> different from the ideal doesn't mean we should give up the ideal, at
> least not lightly.  And we certainly should consider whether the
> alternative proposal will produce a far worse situation than what we
> have today.

Of course a carefully hand-crafted manual is best for the users. But by 
how much?

What we have now is the situation where we don't have a lot of manpower, 
and more people are interested in contributing code (and docstrings, at 
most) than there are those whole also contribute to the manual. I'm sure 
you know it all yourself.

Having some features absent from the manual can be bad enough, but the 
cases where the docstring was updated but the manual wasn't (e.g. the 
contributor didn't know where to look there for the things to update) 
are even more problematic. And yes, that possibility is created by 
duplication of information, not text (although I've seen both).

My personal "good enough" ideal is a high-level overview of a package in 
Commentary, and a set of docstrings. Sometimes that's not enough, and 
then we produce a manual that describes things in more detail and ties 
things together. I think the "more detail" part is a good indicator of 
whether we actually need a manual, aside from manuals like Elisp Into 
and Emacs Intro, of course.

> Once again, I suggest that you take a good look at
> manuals of GnuTLS and Guile, they are produced using the methodology
> that you propose.  That is our future if we go that way.  Do you
> really like the result?

Do you mean e.g. 
https://www.gnu.org/software/guile/manual/html_node/index.html?

At a brief glance, it looks well-structured and fairly readable. But I'm 
not sure I see the part where it's auto-generated.

>> That's called "duplication".
> 
> You are overloading the meaning of "duplication".

It's still one of the common uses of the word.

> The original
> proposal was for a literal copying of the same text.  Drew was talking
> about duplicating the information, but expressed in a different,
> sometimes very different, form.

And my problem is that it's very often expressed in a very similar form, 
if not copied verbatim.

> This doesn't really resolve the issue pointed out by Drew.  And that
> is only one of the issues, there are others, also valid ones.

Drew said docstrings are mostly for the interactive case. That's wildly 
inaccurate: as an Elisp programmer, I almost always look at the 
docstrings and comments, but very rarely at the manuals. Many others do 
the same.

Re: docstrings and elisp reference

[Alan Mackenzie]

Hello, Dmitry.

On Sat, Jun 17, 2017 at 15:46:58 +0300, Dmitry Gutov wrote:
> On 6/7/17 8:28 AM, Eli Zaretskii wrote:
> > There is a difference, that's true.  But the fact that reality is
> > different from the ideal doesn't mean we should give up the ideal, at
> > least not lightly.  And we certainly should consider whether the
> > alternative proposal will produce a far worse situation than what we
> > have today.

> Of course a carefully hand-crafted manual is best for the users. But by 
> how much?

> What we have now is the situation where we don't have a lot of manpower, 
> and more people are interested in contributing code (and docstrings, at 
> most) than there are those whole also contribute to the manual. I'm sure 
> you know it all yourself.

Are we really all that short of manpower?  Compared with the Emacs
project in times past?  My impression is that most contributors here do
their best to update the manuals wrt their contributions.

[ .... ]

> Drew said docstrings are mostly for the interactive case. That's wildly 
> inaccurate: as an Elisp programmer, I almost always look at the 
> docstrings and comments, but very rarely at the manuals. Many others do 
> the same.

For what it's worth, I read the Elisp manual frequently.  I value its
style, its content and its readability.  I also attempt to update it and
the Emacs manual each time I make pertinent changes.  It was the quality
of these manuals which lead me greatly to prefer Emacs over XEmacs
around 15 years ago.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: docstrings and elisp reference

[Dmitry Gutov]

Hi Alan,

On 6/17/17 4:04 PM, Alan Mackenzie wrote:

> Are we really all that short of manpower?

That has been my impression. Not for me to decide, though.

> For what it's worth, I read the Elisp manual frequently.

Okay. But do you refer to it more often than to docstrings?

Re: docstrings and elisp reference

[Alan Mackenzie]

Hello, Dmitry.

On Sat, Jun 17, 2017 at 16:55:27 +0300, Dmitry Gutov wrote:
> On 6/17/17 4:04 PM, Alan Mackenzie wrote:

> > Are we really all that short of manpower?

> That has been my impression. Not for me to decide, though.

My impression is that we have been relatively stable over the last
few/several/many years.  Some old time regulars have dropped out, some
new people (including yourself) have replaced them.

> > For what it's worth, I read the Elisp manual frequently.

> Okay. But do you refer to it more often than to docstrings?

No, I refer to doc strings more often, but I probably spend about the
same amount of time reading doc strings and the manuals.  With Emacs
being so extensive, I can't possibly hold the details of even what I use
every day in my head.  Heck, I even consult the CC Mode manual (which I
largely wrote) to help me adjust my CC Mode configuration.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: docstrings and elisp reference

[Eli Zaretskii]

> Date: Sat, 17 Jun 2017 20:14:23 +0000
> Cc: Eli Zaretskii <eliz@gnu.org>, stephen_leake@stephe-leake.org,
>   drew.adams@oracle.com, emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> > > For what it's worth, I read the Elisp manual frequently.
> 
> > Okay. But do you refer to it more often than to docstrings?
> 
> No, I refer to doc strings more often, but I probably spend about the
> same amount of time reading doc strings and the manuals.  With Emacs
> being so extensive, I can't possibly hold the details of even what I use
> every day in my head.  Heck, I even consult the CC Mode manual (which I
> largely wrote) to help me adjust my CC Mode configuration.

That's also my experience, FWIW.

Re: docstrings and elisp reference

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

When you want to refer to documentation on a function whose context
you already basically know, you probably want the doc string, because
doc strings are made for that.  The manual is made for when you want
to learn about a certain feature, including various functions and
variables.

They are both important, and each should be written in the way that is
best for its job.  Which means, quite differently.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: docstrings and elisp reference

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

Anyone that wants a concatenation of doc strings is welcome to make
one.  Why worry about it?

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: docstrings and elisp reference

[John Wiegley]

>>>>> "AM" == Alan Mackenzie <acm@muc.de> writes:

AM> Are we really all that short of manpower? Compared with the Emacs project
AM> in times past? My impression is that most contributors here do their best
AM> to update the manuals wrt their contributions.

I agree. I hear it often repeated that we are small and lack manpower, which I
think only ends up discouraging newcomers.

Maybe if we stop telling ourselves that we lack manpower, we'd realize that we
do in fact have quite a bit of it (just not all on emacs-devel), and that a
more optimistic outlook might encourage others to spend their time here.

Just a meta-comment,
-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: docstrings and elisp reference

[Eli Zaretskii]

> Cc: drew.adams@oracle.com, stephen_leake@stephe-leake.org, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sat, 17 Jun 2017 15:46:58 +0300
> 
> On 6/7/17 8:28 AM, Eli Zaretskii wrote:
> > There is a difference, that's true.  But the fact that reality is
> > different from the ideal doesn't mean we should give up the ideal, at
> > least not lightly.  And we certainly should consider whether the
> > alternative proposal will produce a far worse situation than what we
> > have today.
> 
> Of course a carefully hand-crafted manual is best for the users. But by 
> how much?

IME, by a lot.

> What we have now is the situation where we don't have a lot of manpower, 
> and more people are interested in contributing code (and docstrings, at 
> most) than there are those whole also contribute to the manual. I'm sure 
> you know it all yourself.

Actually, I'm quite satisfied by our documentation efforts, manpower
issues notwithstanding.  Lately, all changes that need to be reflected
in the manuals come with documentation patches included.  It's clear
that some of us need to invest more efforts in writing documentation,
and their contributions need to be reviewed more thoroughly, but I
think the result is much better than it used to be, when each release
needed a lot of catching up in the manuals.  I guess we will see
whether my impression is accurate when we release the next version.

> Having some features absent from the manual can be bad enough, but the 
> cases where the docstring was updated but the manual wasn't (e.g. the 
> contributor didn't know where to look there for the things to update) 
> are even more problematic.

I don't think we have this lately.

> > Once again, I suggest that you take a good look at
> > manuals of GnuTLS and Guile, they are produced using the methodology
> > that you propose.  That is our future if we go that way.  Do you
> > really like the result?
> 
> Do you mean e.g. 
> https://www.gnu.org/software/guile/manual/html_node/index.html?
> 
> At a brief glance, it looks well-structured and fairly readable.

Yes, because the good parts are actually written by hand.

> But I'm not sure I see the part where it's auto-generated.

Those are the *.doc files generated during the build.

> > The original
> > proposal was for a literal copying of the same text.  Drew was talking
> > about duplicating the information, but expressed in a different,
> > sometimes very different, form.
> 
> And my problem is that it's very often expressed in a very similar form, 
> if not copied verbatim.

Sometimes that is appropriate.  Other times it isn't.

Re: docstrings and elisp reference

[Stefan Monnier]

>> > The original proposal was for a literal copying of the same text.
>> > Drew was talking about duplicating the information, but expressed
>> > in a different, sometimes very different, form.
>> And my problem is that it's very often expressed in a very similar form, 
>> if not copied verbatim.
> Sometimes that is appropriate.  Other times it isn't.

My proposal was to enhance our tooling so that instead of manually
copying the text, the copying is done dynamically, based on special
references.  It wouldn't prevent us from having manually written docs
for those cases where "copied verbatim" is not appropriate.

Of course, it would encourage the "verbatim copy", and hence discourage
the "specially hand crafted doc".


        Stefan

Re: docstrings and elisp reference

[Paul Eggert]

Eli Zaretskii wrote:
> Lately, all changes that need to be reflected
> in the manuals come with documentation patches included.

This is a good trend, and we should continue to encourage it. I am looking 
forward to the time when we need not decorate the NEWS file with "---" and "+++" 
markers, because changes are invariably documented when they are committed.

RE: docstrings and elisp reference

[Drew Adams]

> > The original proposal was for a literal copying of the same  
> > text.  Drew was talking about duplicating the information,
> > but expressed in a different, sometimes very different, form.
> 
> And my problem is that it's very often expressed in a very
> similar form, if not copied verbatim.

What is your problem with such similarity?  That it is only
sometimes similar and not always identical?  That it is at
all similar and not totally different?

Sometimes there is no need for the presentation to be very
different; sometimes a difference makes sense.

And nothing says that every existing doc string is perfect.
Those that are apparently problematic to you, because
similar, might well be candidates for improvement.  Their
being "very similar" might be wholly appropriate, or it
might just represent inattention or laziness.

You seem to be missing the point that manual and doc string
can have different uses and purposes.  You seem to think
that when they differ that's a waste, and when they are the
same that's a waste because the text could have been reused
automatically.

Look not at the similarity or difference of this or that
function description.  Think about the difference in how
the two can be used.  It is _that_ difference that informs
possible differences or similarities of text.

A written news article is not a video clip.  Both might
tell the same story, in part or in whole, but they might
well tell it differently.  Same or similar message; but
often different delivery/presentation.

> > This doesn't really resolve the issue pointed out by Drew.
> > And that is only one of the issues, there are others, also
> > valid ones.
> 
> Drew said docstrings are mostly for the interactive case.

No.  I never said any such thing.  You are either
misreading or misrepresenting.

I said:

  The presentation and context of use are different.
  The level of detail is often different.  (Sometimes
  there is more detail in the manual; sometimes there
  is more in a doc string.)

  In particular, doc strings are written as user help.
  The interactive use, if any, is typically described
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  first, and from the point of an interactive user.
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  Not so, the Elisp manual.

That says nothing about doc strings being "mostly for
the interactive case."  It says that a doc string about
a _command_ presents the command _first_ in its use as
a command, that is, interactively.

> That's wildly inaccurate:

It's wildly inaccurate for you to say that "Drew said
docstrings are mostly for the interactive case."

> as an Elisp programmer, I almost always look at the
> docstrings and comments, but very rarely at the manuals.
> Many others do the same.

We all look at doc strings.  And many of us look at
comments - and the code that is commented.  Doc strings
are typically the most immediate help for users - and
that includes Lisp programmers.

Doc strings are not only, or even primarily, "for the
interactive case".  Doc strings are not manuals, and
manuals are not doc strings - that's all.  They have
purposes that can be different but can overlap.

Doc strings are user help, including but not limited
to help for non-Lisp users.  Interactive use of a
command is described first in a doc string because the
main purpose of a command is to be used interactively.

Hard to believe this is not obvious to us all, by now.
Video is not newsprint.  A song is not a painting.
A garden is not an essay.

But perhaps it is not obvious to you because, as you
say, you "very rarely" consult the manuals?  If doc
strings and comments suffice for you, then why worry
about this question at all?

Is it just because _you_ don't want to be bothered
maintaining the manual?  If so, and if you feel that
strongly, then don't bother.  Someone else will do
it - or it won't get done.  Either way, you need not
get bothered by it, if that's your problem.

> my problem is that it's very often expressed in
> a very similar form, if not copied verbatim.

Don't worry.

Emacs has always put importance on its manuals, as
well as its doc strings.  I hope it will long
continue to do so, whoever participates in developing
and maintaining it.

But sure, there have always been those participants
who are more, and others who are less, concerned about
doc/help, including comments, doc strings, and manuals.
Nothing says that each contributor needs to be equally
interested in every possible way of contributing.

What's wrong is to mistake one's own interests, which
are typically limited (parochial), for the broader
interests of Emacs as a whole.  Just because you might
rarely read the manuals, that's not a reason for Emacs
to drop development of manuals or reduce their quality.

Re: docstrings and elisp reference

[Thien-Thi Nguyen]

[-- Attachment #1: Type: text/plain, Size: 1049 bytes --]


() Drew Adams <drew.adams@oracle.com>
() Sat, 17 Jun 2017 07:52:47 -0700 (PDT)

   What's wrong is to mistake one's own interests, which
   are typically limited (parochial), for the broader
   interests of Emacs as a whole.  Just because you might
   rarely read the manuals, that's not a reason for Emacs
   to drop development of manuals or reduce their quality.

It's wrong inasmuch as it's incomplete.  That's natural, though.
We are all born ignorant, and begin as solipsists.  Emacs, OTOH,
knows its place: the vehicle to unfix files and ferry its users'
focus, their (point) of view, forward, backward, through various
expressions and forms.  It will never arrive, never be complete.

Still, people help it, and it helps people -- that's fine, too.

-- 
Thien-Thi Nguyen -----------------------------------------------
 (defun responsep (query)
   (pcase (context query)
     (`(technical ,ml) (correctp ml))
     ...))                              748E A0E8 1CB8 A748 9BFA
--------------------------------------- 6CE4 6703 2224 4C80 7502


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

RE: docstrings and elisp reference

[Drew Adams]

> >>> they should not duplicate each other.
> >>
> >> Then you have something else in mind than the current
> >> situation with documentation in Emacs.
> >
> > No, you do, if you think the intention now is that
> > they duplicate each other.  It never has been.
> 
> Please look up the difference between "intention" and "current situation".

There may be some examples in the "current situation"
where the doc string text and the text in the Elisp manual
are identical.  That is hardly how I would characterize
"the current situation".  Laziness and sloppiness happens.
And there might even be a few cases where having the exact
same text is not inappropriate.

But in general the "current situation" jibes well, I think,
with the "intention": doc string and manual are generally
_not_ exactly the same - their presentations, and to some
extent their content, are different.

> > Some of the information is often the same; that's all.
> 
> That's called "duplication".

INFORMATION.  Yes, much of the _information_ is duplicated.
Of course.  Almost all of the information contained in doc
strings is in the Elisp manual.

The text/presentation is different.  The contexts are
different.  The readers are often different.  A reader's
intention in consulting them is often different.

> > In particular, doc strings are written as user help.
> > The interactive use, if any, is typically described
> > first, and from the point of an interactive user.
> > Not so, the Elisp manual.
> >
> > The Elisp doc for a command tells a Lisp programmer
> > what the function and its parameters are, and
> > describes their behavior.  It might or might not
> > mention that one of the parameters corresponds to
> > the prefix arg when called interactively.
> 
> We have different manuals, with different goals.
> 
> The fact that docstrings often describe the interactive case (when it
> exists) doesn't make them necessarily targeted at the end user. Just
> exhaustive.

The question about interactive use is a one of emphasis
and audience.  Interactive use is presented _first_ in
doc strings.  The aim in presenting it is not merely to
be exhaustive.  It's about whom the different docs are
presented to and what the aims typically are in reading
them.  And yes, that involves judgment calls.

We are not (and should not be) aiming only at reducing
emacs-dev maintenance time.  We are (and should be)
aiming at the best possible help for Emacs users,
including Lisp users.

That involves trade-offs, of course.  What it does not
involve is throwing up our hands and saying "Too hard,
bothersome, and error prone.  Let's just write it once -
one size fits all - and reuse the same thing for both
Elisp manual and doc string."  (As if that were a new
idea...)

Doc/help has been extremely important to the Emacs
project since Day One.  It's part of the mantra:
self-documenting editor.

And a lot of care and time has been spent, including
by some of the most qualified and knowledgable Emacs
developers, getting it right, where "right" means
paying attention to the differences among the roles
of doc string, Emacs manual, and Elisp manual.

I'm pretty sure that those who have long held a high
priority for the quality of Emacs doc, including in
particular RMS and Eli, do not share a reductive,
JavaDoc-like view of it.  I can't speak for them, of
course.  Perhaps they will care to speak up; perhaps not.

In any case, I've given my opinion about this: using
only a JavaDoc-like approach to Emacs doc would be a
step backward, not forward.  That does not mean that
there can never be any reuse (we do that with `apropos'
functions, for example).  It means that human judgment
is called for.

RE: docstrings and elisp reference

[Drew Adams]

> And a lot of care and time has been spent, including
> by some of the most qualified and knowledgable Emacs
> developers, getting it right, where "right" means
> paying attention to the differences among the roles
> of doc string, Emacs manual, and Elisp manual.
> 
> I'm pretty sure that those who have long held a high
> priority for the quality of Emacs doc, including in
> particular RMS and Eli, do not share a reductive,
> JavaDoc-like view of it.  I can't speak for them, of
> course.  Perhaps they will care to speak up; perhaps not.

I see they've both done that now.  Those who have not
yet spent a long time concerned about doc would do
well to listen.

Doc, like helping users generally, and like lots of
other things, is not so trivial.  There is no silver
bullet or royal road to helpful doc.

Emacs deserves the best doc possible.  You might say
it's been lucky that it's always benefitted from
strong developers who are also very concerned about
doc.  But it's not luck.  Quality doc is in part what
Emacs is about.  If you're interested in Emacs then
you pretty much need to be also interested in doc/help.

UI development is, in principle, similar, especially
free-software dev.

Re: docstrings and elisp reference

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

The GNU Project has a policy about what constitutes a good manual.
A series of doc strings is not acceptable.

See Doc Strings and Manuals in the GNU Coding Standards.

Please don't take up time arguing about this -- it won't change anything.


-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: docstrings and elisp reference

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I was thinking (naively?) that having the docstrings on one side
  > and the reference on the other side was not a very efficient way
  > to maintain documentation. Is everything in the documentation
  > actually written by hand based on the docstrings? Wouldn't it be
  > nicer to have good docstrings and use them directly in the
  > documentation to avoid duplication of work?

Absolutely not!  That's a recipe for a bad manual.  The text of a good
manual section is totally different from a series of doc strings.

Doc strings are read one by one.  Each one describes one function or
variable in isolation.  They are written specifically for this kind of
use.

By contrast, a section in a manual describes several related functions
and variables, and it should be written in an integrated way that
flows smoothly.  Meanwhile, common material should be stated only
once.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: docstrings and elisp reference

[Dmitry Gutov]

On 6/7/17 1:42 AM, Richard Stallman wrote:

> Meanwhile, common material should be stated only
> once.

Stated where?

Re: docstrings and elisp reference

[Eli Zaretskii]

> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Wed, 7 Jun 2017 01:49:18 +0300
> Cc: emacs-devel@gnu.org
> 
> On 6/7/17 1:42 AM, Richard Stallman wrote:
> 
> > Meanwhile, common material should be stated only
> > once.
> 
> Stated where?

Somewhere in the vicinity, often in the same node, but sometimes in a
sibling or parent node which discuss the general attributes and
features of a family of related functions/features.  (If the common
material is far away from the place where it's relevant, there should
be a cross-reference to it, for those who read only about a specific
feature.)