~Make emacs friendlier: package documentation [POC CODE INCLUDED]

~Make emacs friendlier: package documentation [POC CODE INCLUDED]

[Boruch Baum]

[-- Attachment #1: Type: text/plain, Size: 766 bytes --]

A programmer's idea of a friendly environment might be the opposite of a
user's in that what programmer enjoys writing documentation, and having
to repeat the process with almost identical text in more than place, and
in more than one format. There are docstrings, and then there's the
commentary at the top of the elisp file, and then any info file, and
what else.

Attached is working proof-of-concept code to allow a user to easily get
all a package's in-line documentation, in the format of an org-mode
buffer, to cover cases where there is no info file, or to supplement the
info file.

Evaluate the code, Perform M-x pack-doc, and select an emacs-lisp
package file.

--
hkp://keys.gnupg.net
CA45 09B5 5351 7C11 A9D1  7286 0036 9E45 1595 8BC0

[-- Attachment #2: pack-doc.el --]
[-- Type: text/plain, Size: 2025 bytes --]

;;; pack-doc.el --- Make org documetation from package source file -*- lexical-binding: t -*-

(defun pack-doc (file)
  "Create documentation in org-mode format from FILE.
FILE is an elisp source file formatted according to the emacs
style. Result is an org mode buffer containing the file's
doumentary comments and docstrings."
  (interactive "f")
  ;; TODO: handle prompt for file if NIL
  (switch-to-buffer (get-buffer-create (concat (file-name-nondirectory file) ".org")))
  (insert-file-contents-literally file nil nil nil 'replace)
  (goto-char (point-min))
  ;; Comment-out docstrings
  (let (p0 p1 p2)
    (while (setq p0 (re-search-forward "^(def" nil t))
      (when (not (re-search-forward "^ +\"" nil t))
        (error "badly formatted file, near %d" p0))
      (setq p1 (match-beginning 0))
      (replace-match "")
      (when (not (re-search-forward "\")?$" nil t))
        (error "badly formatted file, near %d" p0))
      (setq p2 (match-beginning 0))
      (replace-match "")
      (goto-char p1)
      (narrow-to-region p1 p2) ; because p2 moves with every replacement
      (while (re-search-forward "^" nil t)
        (replace-match ";;"))
      (widen)))
  ;; Comment-out def* and adjust pre-existing comments
  (dolist (repl '(("^;;; " ";;;; ")
                  ("^$"    ";;")
                  ("^(def" ";;; (def")))
    (goto-char (point-min))
    (while (re-search-forward (car repl) nil t)
      (replace-match (cadr repl))))
  ;; Remove everything else
  (goto-char (point-min))
  (delete-non-matching-lines  "^;" (point-min) (point-max))
  ;; Create org headings and remove extra blank lines
  (dolist (repl '(("^;;;;" "**")
                  ("^;;; (def[^ ]+" "***")
                  ("^;;;" "***")
                  ("^;;" " ")
                  ("^ +$" "")
                  ("\n\n+" "\n\n")))
    (goto-char (point-min))
    (while (re-search-forward (car repl) nil t)
      (replace-match (cadr repl))))
  ;; Create top heading
  (goto-char 1)
  (delete-char 1)
  ;; Ta-da!
  (org-mode))

Re: ~Make emacs friendlier: package documentation [POC CODE INCLUDED]

[Eli Zaretskii]

> Date: Thu, 15 Oct 2020 15:09:29 -0400
> From: Boruch Baum <boruch_baum@gmx.com>
> 
> A programmer's idea of a friendly environment might be the opposite of a
> user's in that what programmer enjoys writing documentation, and having
> to repeat the process with almost identical text in more than place, and
> in more than one format. There are docstrings, and then there's the
> commentary at the top of the elisp file, and then any info file, and
> what else.
> 
> Attached is working proof-of-concept code to allow a user to easily get
> all a package's in-line documentation, in the format of an org-mode
> buffer, to cover cases where there is no info file, or to supplement the
> info file.

Thanks.  Does this do more/different things than "C-h P"?  AFAIK,
"C-h P" also uses the commentary part of a package's source file.  If
something is missing there, perhaps we could add or extend what
"C-h P" does?

Re: ~Make emacs friendlier: package documentation [POC CODE INCLUDED]

[Boruch Baum]

On 2020-10-15 22:24, Eli Zaretskii wrote:
> Thanks.  Does this do more/different things than "C-h P"?  AFAIK,
> "C-h P" also uses the commentary part of a package's source file.  If
> something is missing there, perhaps we could add or extend what
> "C-h P" does?

I just now double-checked, and my "C-h P" binds to a function
`describe-package'. Is that what you mean? That function produces very
different content in a very different format. What I submitted is
proof-of-concept code, which *could* be integrated into
`describe-package'; I'm not picky. Here are some of the differences:


+ describe-package doesn't operate on all emacs-lisp files
  + try performing it on `dired' or dired.el

+ describe-package does in fact extract the *first* section of header
  documentation commentary.
  + What it omits, that is in the proposal is all the rest of the header
    documentation, which in many cases includes information on
    installation, configuration, usage, examples, known bugs, etc.

+ describe package provides information on package 'status' (eg.
  built-in, external) which isn't in my POC.

+ My POC function additionally formats and presents *all* the
  documentation for all the package's symbols.
  + and it's done in an org-mode hierarchy for your navigating pleasure.

--
hkp://keys.gnupg.net
CA45 09B5 5351 7C11 A9D1  7286 0036 9E45 1595 8BC0

Re: ~Make emacs friendlier: package documentation [POC CODE INCLUDED]

[Eli Zaretskii]

> Date: Thu, 15 Oct 2020 15:41:32 -0400
> From: Boruch Baum <boruch_baum@gmx.com>
> Cc: emacs-devel@gnu.org
> 
> I just now double-checked, and my "C-h P" binds to a function
> `describe-package'. Is that what you mean? That function produces very
> different content in a very different format. What I submitted is
> proof-of-concept code, which *could* be integrated into
> `describe-package'; I'm not picky. Here are some of the differences:
> 
> 
> + describe-package doesn't operate on all emacs-lisp files
>   + try performing it on `dired' or dired.el
> 
> + describe-package does in fact extract the *first* section of header
>   documentation commentary.
>   + What it omits, that is in the proposal is all the rest of the header
>     documentation, which in many cases includes information on
>     installation, configuration, usage, examples, known bugs, etc.
> 
> + describe package provides information on package 'status' (eg.
>   built-in, external) which isn't in my POC.
> 
> + My POC function additionally formats and presents *all* the
>   documentation for all the package's symbols.
>   + and it's done in an org-mode hierarchy for your navigating pleasure.

Thanks.  To me this sounds like your code could be used to extend "C-h P"
in useful ways.  Perhaps the result of "C-h P" should include a button
"More information" or something that could produce an Org buffer with
the additional stuff.  WDYT?

The main point, I think, is not to create another command whose
functionality overlaps with an existing one, but instead extend the
existing command to do more when requested.

Re: ~Make emacs friendlier: package documentation [POC CODE INCLUDED]

[Boruch Baum]

On 2020-10-16 09:39, Eli Zaretskii wrote:
> Thanks.  To me this sounds like your code could be used to extend "C-h P"
> in useful ways.  Perhaps the result of "C-h P" should include a button
> "More information" or something that could produce an Org buffer with
> the additional stuff.  WDYT?
>
> The main point, I think, is not to create another command whose
> functionality overlaps with an existing one, but instead extend the
> existing command to do more when requested.

1) I share the sentiment about consolidating the help/describe commands,
   and have no objection to the feature being accessed via a link in
   describe-package (ie. no direct dedicated keybinding).

   1.1) The output of the proof-of-concept code (for now I'll call it
        pack-doc, though I don't like the name) can kind of act as a
        surrogate for a package's info manual, so it might make sense to
        also have an entry point somehow from that angle, but I don't
        have an idea for how to do that.

   1.2) The output is also meant to be useful for emacs developers and
        bug-fixers, in that the org-mode format allows them get a
        birds-eye view of the symbol landscape, and easily navigate
        about, swooping in and out to view docstrings.

        1.2.1) For this target audience, it might be desirable to
               include the actual lisp code in sub-sub-headings or in
               collapsed code blocks, but that might confuse other users
               / audiences.

2) I do have a personal agenda with this proposal, and I want to be
   explicit about it.

   2.1) Make it easier for emacs code contributors to document their
        work to best user effect and with least programmer effort /
        duplication.

   2.2) Encourage literate programming in elisp files beyond simply
        docstrings. I see many package files that are conscientiously
        documented, yet also see many core-emacs packages whose
        documentation amount to just stubs.

3) pack-doc can operate on any elisp file that complies with emacs
   coding standards, even if it isn't on the package list and doesn't
   show up as a completion option for describe-package. In testing, I
   discovered dired.el as an example of a file that would be of interest
   but couldn't be accessed with the current method used by
   describe-package. Some options for such cases:

   2.1) Change the completion/input code for describe-package (con: messy).

   2.2) Allow direct interactive access to the feature, but without a
        keybinding (con: adds another obscure feature).

4) Have you (or anyone on the list, don't be shy) tried it on an el file?

--
hkp://keys.gnupg.net
CA45 09B5 5351 7C11 A9D1  7286 0036 9E45 1595 8BC0

Re: ~Make emacs friendlier: package documentation [POC CODE INCLUDED]

[Eli Zaretskii]

> Date: Fri, 16 Oct 2020 03:34:32 -0400
> From: Boruch Baum <boruch_baum@gmx.com>
> Cc: emacs-devel@gnu.org
> 
> 4) Have you (or anyone on the list, don't be shy) tried it on an el file?

I didn't, not yet.  I'd like first to see it in its final form.

Thanks.

Re: ~Make emacs friendlier: package documentation [POC CODE INCLUDED]

[Boruch Baum]

On 2020-10-16 13:20, Eli Zaretskii wrote:
> > Date: Fri, 16 Oct 2020 03:34:32 -0400
> > From: Boruch Baum <boruch_baum@gmx.com>
> > Cc: emacs-devel@gnu.org
> >
> > 4) Have you (or anyone on the list, don't be shy) tried it on an el file?
>
> I didn't, not yet.  I'd like first to see it in its final form.

It's functionality seems pretty much in its final form as-is. What's
left to be done seems to be interface related. I don't want more
situations like [1], where I do all the work, and then abruptly get the
silent treatment.

[1] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=43709#43

--
hkp://keys.gnupg.net
CA45 09B5 5351 7C11 A9D1  7286 0036 9E45 1595 8BC0

Re: ~Make emacs friendlier: package documentation [POC CODE INCLUDED]

[Stefan Monnier]

> 1) I share the sentiment about consolidating the help/describe commands,
>    and have no objection to the feature being accessed via a link in
>    describe-package (ie. no direct dedicated keybinding).

IIUC your pack-doc provides a superset of what `C-h P` provides.
So what would be the downside(s) of replacing `C-h P` with your packdoc?

IOW what would be the advantage of adding a link in `C-h P` to your
system, instead of just replacing `C-h P` with your system?


        Stefan

Re: ~Make emacs friendlier: package documentation [POC CODE INCLUDED]

[Boruch Baum]

On 2020-10-18 09:11, Stefan Monnier wrote:
> > 1) I share the sentiment about consolidating the help/describe commands,
> >    and have no objection to the feature being accessed via a link in
> >    describe-package (ie. no direct dedicated keybinding).
>
> IIUC your pack-doc provides a superset of what `C-h P` provides.

No, the only intersection is the 'commentary' text taken from the elisp
file. describe-package presents a unique heading summary; for example:

  #+BEGIN_QUOTE
  yasnippet is a dependency package.

       Status: Installed in ‘yasnippet-20200524.2215/’ (unsigned).
      Version: 20200524.2215
       Commit: d3d6d70b1cd4818d271752468e0fdb0788db750d
      Summary: Yet another snippet extension for Emacs
     Requires: cl-lib-0.5
     Homepage: http://github.com/joaotavora/yasnippet
     Keywords: [convenience] [emulation]
  Other versions: 20200604.246 (melpa), 0.14.0 (gnu), 0.6.1 (marmalade).
  #+END_QUOTE

> So what would be the downside(s) of replacing `C-h P` with your packdoc?

I have no objection, but I do like the heading summary that
describe-package presents, and think it would be a shame to lose it.

> IOW what would be the advantage of adding a link in `C-h P` to your
> system, instead of just replacing `C-h P` with your system?

IMO: Just losing the heading summary, but that code can be integrated so
that nothing would be lost.

Playing devil's advocate: I'm not any kind of UX psychologist, but if
there is someone like that on the list, they could speak to the notion
of it maybe being a situation of presenting 'too much information' for a
user expecting the prior behavior or for a user just wanting the most
basic package description or for a user unfamiliar with org-mode.

--
hkp://keys.gnupg.net
CA45 09B5 5351 7C11 A9D1  7286 0036 9E45 1595 8BC0

Re: ~Make emacs friendlier: package documentation [POC CODE INCLUDED]

[Eli Zaretskii]

> Date: Sun, 18 Oct 2020 01:58:15 -0400
> From: Boruch Baum <boruch_baum@gmx.com>
> Cc: emacs-devel@gnu.org
> 
> On 2020-10-16 13:20, Eli Zaretskii wrote:
> > > Date: Fri, 16 Oct 2020 03:34:32 -0400
> > > From: Boruch Baum <boruch_baum@gmx.com>
> > > Cc: emacs-devel@gnu.org
> > >
> > > 4) Have you (or anyone on the list, don't be shy) tried it on an el file?
> >
> > I didn't, not yet.  I'd like first to see it in its final form.
> 
> It's functionality seems pretty much in its final form as-is.

I thought you were okay with extending "C-h P" with a variant of your
code?

Re: ~Make emacs friendlier: package documentation [POC CODE INCLUDED]

[Boruch Baum]

On 2020-10-18 17:53, Eli Zaretskii wrote:
> > Date: Sun, 18 Oct 2020 01:58:15 -0400
> > From: Boruch Baum <boruch_baum@gmx.com>
> > Cc: emacs-devel@gnu.org
> >
> > On 2020-10-16 13:20, Eli Zaretskii wrote:
> > > > Date: Fri, 16 Oct 2020 03:34:32 -0400
> > > > From: Boruch Baum <boruch_baum@gmx.com>
> > > > Cc: emacs-devel@gnu.org
> > > >
> > > > 4) Have you (or anyone on the list, don't be shy) tried it on an el file?
> > >
> > > I didn't, not yet.  I'd like first to see it in its final form.
> >
> > It's functionality seems pretty much in its final form as-is.
>
> I thought you were okay with extending "C-h P" with a variant of your
> code?

I am.

--
hkp://keys.gnupg.net
CA45 09B5 5351 7C11 A9D1  7286 0036 9E45 1595 8BC0

Re: ~Make emacs friendlier: package documentation [POC CODE INCLUDED]

[Eli Zaretskii]

> Date: Sun, 18 Oct 2020 11:05:40 -0400
> From: Boruch Baum <boruch_baum@gmx.com>
> Cc: emacs-devel@gnu.org
> 
> > > > > 4) Have you (or anyone on the list, don't be shy) tried it on an el file?
> > > >
> > > > I didn't, not yet.  I'd like first to see it in its final form.
> > >
> > > It's functionality seems pretty much in its final form as-is.
> >
> > I thought you were okay with extending "C-h P" with a variant of your
> > code?
> 
> I am.

Well, would you like then to modify your code along these lines, and
submit a modified patch?  TIA.

Re: ~Make emacs friendlier: package documentation [POC CODE INCLUDED]

[Boruch Baum]

On 2020-10-18 18:12, Eli Zaretskii wrote:
> > Date: Sun, 18 Oct 2020 11:05:40 -0400
> > From: Boruch Baum <boruch_baum@gmx.com>
> > Cc: emacs-devel@gnu.org
> >
> > > > > > 4) Have you (or anyone on the list, don't be shy) tried it on an el file?
> > > > >
> > > > > I didn't, not yet.  I'd like first to see it in its final form.
> > > >
> > > > It's functionality seems pretty much in its final form as-is.
> > >
> > > I thought you were okay with extending "C-h P" with a variant of your
> > > code?
> >
> > I am.
>
> Well, would you like then to modify your code along these lines, and
> submit a modified patch?  TIA.

At this point, not at all, because no one seems to have made even the
slightest effort to evaluate the current submission and provide real
feedback. I get the feeling that I'm being asked to invest time and
effort only to have it all ignored.

--
hkp://keys.gnupg.net
CA45 09B5 5351 7C11 A9D1  7286 0036 9E45 1595 8BC0

Re: ~Make emacs friendlier: package documentation [POC CODE INCLUDED]

[Stefan Kangas]

Boruch Baum <boruch_baum@gmx.com> writes:

>> IOW what would be the advantage of adding a link in `C-h P` to your
>> system, instead of just replacing `C-h P` with your system?
>
> IMO: Just losing the heading summary, but that code can be integrated so
> that nothing would be lost.

Having tested your code, I agree that we should keep `C-h P' as-is, but
extend it with everything from the "** Code:" header and down in your
code.  (I guess that part could be collapsed by default in order to not
overwhelm a user with details?)  It would be great if one could also
collapse the "Commentary" section.  I would also give it a headline
"Description:" using the same font as "Status:".

We would also need to check that this works also for packages that are
not installed (they don't need to show the full documentation, I don't
think, but they should at least not be broken ;-).

Bonus points if we can also add a link to the relevant Info manual,
where there is one.

BTW, does this all really need to be based on Org-mode?  Perhaps one
could make it look more polished by rolling a custom solution here, and
I'm not exactly sure what using Org-mode buys us in this case.

Three more nits:
- I wouldn't show obsolete aliases.
- The autoload cookies should probably get a better representation than
  just being copied in.
- The doc strings should be passed to `substitute-command-keys' before
  display.

Re: ~Make emacs friendlier: package documentation [POC CODE INCLUDED]

[Boruch Baum]

I just did some extra due-diligence, and need to correct something I
said in my prior email...

On 2020-10-18 10:43, Boruch Baum wrote:
> On 2020-10-18 09:11, Stefan Monnier wrote:
> > > 1) I share the sentiment about consolidating the help/describe commands,
> > >    and have no objection to the feature being accessed via a link in
> > >    describe-package (ie. no direct dedicated keybinding).
> >
> > IIUC your pack-doc provides a superset of what `C-h P` provides.
>
> No, the only intersection is the 'commentary' text taken from the elisp
> file.

That statement of mine turns out to be false. For example,
describe-package for yasnippet does not supply any commentary or package
description text at all, while my code submission is chock-full of
useful information.

--
hkp://keys.gnupg.net
CA45 09B5 5351 7C11 A9D1  7286 0036 9E45 1595 8BC0

Re: ~Make emacs friendlier: package documentation [POC CODE INCLUDED]

[Eli Zaretskii]

> Date: Sun, 18 Oct 2020 11:28:30 -0400
> From: Boruch Baum <boruch_baum@gmx.com>
> Cc: emacs-devel@gnu.org
> 
> > Well, would you like then to modify your code along these lines, and
> > submit a modified patch?  TIA.
> 
> At this point, not at all, because no one seems to have made even the
> slightest effort to evaluate the current submission and provide real
> feedback. I get the feeling that I'm being asked to invest time and
> effort only to have it all ignored.

It will not be ignored, because it adds useful features.  It's just
the that the current form is so different from the final one in many
aspects that having to review it now would mean we have to review it
twice, which I at least would like to avoid.

So please don't feel discouraged.

Re: ~Make emacs friendlier: package documentation [POC CODE INCLUDED]

[Boruch Baum]

On 2020-10-18 08:50, Stefan Kangas wrote:
> Boruch Baum <boruch_baum@gmx.com> writes:
>
> >> IOW what would be the advantage of adding a link in `C-h P` to your
> >> system, instead of just replacing `C-h P` with your system?
> >
> > IMO: Just losing the heading summary, but that code can be integrated so
> > that nothing would be lost.
>
> Having tested your code,

Thanks. For what packages? It's important to know in order to understand
the content of your feedback.

> I agree that we should keep `C-h P' as-is, but extend it with
> everything from the "** Code:" header and down in your code.

That would lose a lot of information. There could be many
sections/headings above 'code', and it turns out to seem that even the
'commentary' section that describe-package presents isn't directly from
the 'commentary' section in the elisp file (see, for example, yasnippet).

> (I guess that part could be collapsed by default in order to not
> overwhelm a user with details?)

Yes. Good idea.

> It would be great if one could also collapse the "Commentary" section.

Do-able.

> I would also give it a headline "Description:" using the same font as
> "Status:".

Don't know if that's possible for org-mode headings...

> We would also need to check that this works also for packages that are
> not installed (they don't need to show the full documentation, I don't
> think, but they should at least not be broken ;-).

My code submission should work as-is on any elisp file that conforms to
the emacs coding standard format. I've tested it for my other recent
code submission - key-assist.el

OTOH, the converse is not true. describe-package doesn't expose itself
to any elisp file, only recognized packages. Even some features that we
all think of as packages aren't exposed (eg. dired).

> Bonus points if we can also add a link to the relevant Info manual,
> where there is one.

That a good idea, but it's independent of whatever ends up happening
with my code submission, so it should be an independent bug report or
feature request against package 'describe-package'.

> BTW, does this all really need to be based on Org-mode?

Technically: of course not.

Practically: the entire point of the code submission is exploit the
             great features of org-mode, and the pre-existing format
             conventions of the emacs coding standard.

> Perhaps one could make it look more polished by rolling a custom
> solution here,

Ugh. Re-invent yet another wheel?

> and I'm not exactly sure what using Org-mode buys us in this case.

For starters: Navigation, expansion and collapse of sections.

> Three more nits:

For me to make sense of most of what you write below, I need to know at
the very least on what elisp file you encountered the problem, and
helpfully a pointer of where to find it.

> - I wouldn't show obsolete aliases.

???

> - The autoload cookies should probably get a better representation than
>   just being copied in.

???

> - The doc strings should be passed to `substitute-command-keys' before
>   display.

This one I understand, and is an excellent idea.

--
hkp://keys.gnupg.net
CA45 09B5 5351 7C11 A9D1  7286 0036 9E45 1595 8BC0

Re: ~Make emacs friendlier: package documentation [POC CODE INCLUDED]

[Boruch Baum]

On 2020-10-18 19:00, Eli Zaretskii wrote:
> > Date: Sun, 18 Oct 2020 11:28:30 -0400
> > From: Boruch Baum <boruch_baum@gmx.com>
> > Cc: emacs-devel@gnu.org
> >
> > > Well, would you like then to modify your code along these lines, and
> > > submit a modified patch?  TIA.
> >
> > At this point, not at all, because no one seems to have made even the
> > slightest effort to evaluate the current submission and provide real
> > feedback. I get the feeling that I'm being asked to invest time and
> > effort only to have it all ignored.
>
> It will not be ignored, because it adds useful features.

You couldn't possibly have an informed opinion about that because you
haven't tried it yourself.

> It's just the that the current form is so different from the final one
> in many aspects

"many aspects"? The integration that you proposed is just adding a link.

> that having to review it now would mean we have to review it twice,
> which I at least would like to avoid.

grumble.

> So please don't feel discouraged.

Not helpful.

--
hkp://keys.gnupg.net
CA45 09B5 5351 7C11 A9D1  7286 0036 9E45 1595 8BC0

Re: ~Make emacs friendlier: package documentation [POC CODE INCLUDED]

[Eli Zaretskii]

> Date: Sun, 18 Oct 2020 12:29:17 -0400
> From: Boruch Baum <boruch_baum@gmx.com>
> Cc: emacs-devel@gnu.org
> 
> > It will not be ignored, because it adds useful features.
> 
> You couldn't possibly have an informed opinion about that because you
> haven't tried it yourself.

I could certainly have an informed opinion that I will not ignore your
submission?

> > So please don't feel discouraged.
> 
> Not helpful.

Sorry.

Re: ~Make emacs friendlier: package documentation [POC CODE INCLUDED]

[Stefan Kangas]

Boruch Baum <boruch_baum@gmx.com> writes:

>> Having tested your code,
>
> Thanks. For what packages? It's important to know in order to understand
> the content of your feedback.

I tested it on lisp/time.el.

>> I agree that we should keep `C-h P' as-is, but extend it with
>> everything from the "** Code:" header and down in your code.
>
> That would lose a lot of information. There could be many
> sections/headings above 'code', and it turns out to seem that even the

OK, fair enough, so please modify the above to say "everything after the
'Commentary' section".

You could just move the sections around.  For example, the license could
be moved to the end (if it is even necessary in the GPL case).

> 'commentary' section that describe-package presents isn't directly from
> the 'commentary' section in the elisp file (see, for example, yasnippet).

That's because it comes from the package README file instead.

>> We would also need to check that this works also for packages that are
>> not installed (they don't need to show the full documentation, I don't
>> think, but they should at least not be broken ;-).
>
> My code submission should work as-is on any elisp file that conforms to
> the emacs coding standard format. I've tested it for my other recent
> code submission - key-assist.el
>
> OTOH, the converse is not true. describe-package doesn't expose itself
> to any elisp file, only recognized packages. Even some features that we
> all think of as packages aren't exposed (eg. dired).

Yes, so that's the part that would need looking into.

>> Perhaps one could make it look more polished by rolling a custom
>> solution here,
>
> Ugh. Re-invent yet another wheel?

Not if we can avoid it, but Org-mode is also not a hammer looking for a
nail.

>> Three more nits:
>
> For me to make sense of most of what you write below, I need to know at
> the very least on what elisp file you encountered the problem, and
> helpfully a pointer of where to find it.
>
>> - I wouldn't show obsolete aliases.
>
> ???

Run your code on time.el and see that obsolete aliases are there.

>> - The autoload cookies should probably get a better representation than
>>   just being copied in.
>
> ???

Run your code on time.el and see that the autoload cookies are copied
verbatim above the headings.  I think they should better be marked in a
different way.

Re: ~Make emacs friendlier: package documentation [POC CODE INCLUDED]

[Stefan Monnier]

> IOW what would be the advantage of adding a link in `C-h P` to your
> system, instead of just replacing `C-h P` with your system?

Oh, having just tried it, I see one other advantage: the current `C-h p`
displays much more quickly because it doesn't need to load `org-mode`,
which tends to take a while (not insanely long either, but long enough
to be noticeable).


        Stefan

Re: ~Make emacs friendlier: package documentation [POC CODE INCLUDED]

[Kévin Le Gouguec]

Boruch Baum <boruch_baum@gmx.com> writes:

>> and I'm not exactly sure what using Org-mode buys us in this case.
>
> For starters: Navigation, expansion and collapse of sections.

FWIW, outline-mode now features section-cycling commands (TAB and S-TAB)
that are very similar to org-mode's (cf. bug#41130).

Org has many other interesting features of course; still, maybe
outline-mode would be enough for this use-case?

Re: ~Make emacs friendlier: package documentation [POC CODE INCLUDED]

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Eli Zaretskii <eliz@gnu.org>,  emacs-devel@gnu.org
> Date: Sun, 18 Oct 2020 16:18:34 -0400
> 
> > IOW what would be the advantage of adding a link in `C-h P` to your
> > system, instead of just replacing `C-h P` with your system?
> 
> Oh, having just tried it, I see one other advantage: the current `C-h p`
> displays much more quickly because it doesn't need to load `org-mode`,
> which tends to take a while (not insanely long either, but long enough
> to be noticeable).

That is a general issue with using Org for other features: it takes
eons to load.

Re: ~Make emacs friendlier: package documentation [POC CODE INCLUDED]

[Boruch Baum]

[-- Attachment #1: Type: text/plain, Size: 2492 bytes --]



On 2020-10-18 22:40, Kévin Le Gouguec wrote:
> Boruch Baum <boruch_baum@gmx.com> writes:
>
> >> and I'm not exactly sure what using Org-mode buys us in this case.
> >
> > For starters: Navigation, expansion and collapse of sections.
>
> FWIW, outline-mode now features section-cycling commands (TAB and S-TAB)
> that are very similar to org-mode's (cf. bug#41130).

I'm OK with trying to use outline-minor-mode, but didn't see the
features yet in my version of emacs-snapshot. When will the patches for
41130 be committed and merged?

In the meantime, I've responded to the Stefan's (Kangas) feedback with
modified code, and started looking to a better alternative to posting
entire versions to the list. I see that I have an old old account on
savannah that I don't ever remember using, so I set up an ssh key and
have now been toying unsuccessfully to create a git repository there for
this feature (I'm now calling it org-el-file.el; not a great name yet,
but better than pack-doc).

Do I not have permissions to create a repo on my savannah account? If
not, how can one be created? I'd like three, called: 1) org-el-file
(this thread); 2) key-assist (bug #43709); 3) diredc (not yet shared,
but ready).

The attached version:

  + handles compressed source files

  + marks autoloaded functiions

  + identifies symbol definition

  + aligns symbols names (mostly)

  + labels colophon section

  + starts with commentary section open

  + improves definition regexp

  + attempts to perform substitute-command-keys

    + this turned out to be non-trivial, because the keybindings might
      not yet be defined and the keymap for the related mode might not
      yet be loaded.

    + the current attempt is an improvement, but I'm stuck on removing
      the leading back-slash.

It was very useful getting feedback because Stefan chose an el file with
what I'll call 'badly' (ie. unexpectedly) organized symbols, in the
sense that I was expecting all `defcustom's to be listed together in a
section beginning "^;;; Customization variables:", and so on for symbol
categories ";;; Global variables:", ";;; Buffer-local variables", ";;;
Internal functions:", ";;; Hook functions:", ";;; Interactive
functions:", etc. The new code no longer supposes such orderliness and
consequently retains function names that perform the symbol definitions.

--
hkp://keys.gnupg.net
CA45 09B5 5351 7C11 A9D1  7286 0036 9E45 1595 8BC0

[-- Attachment #2: org-el-file.el --]
[-- Type: text/plain, Size: 2946 bytes --]

;;; org-el-file.el --- Make org documetation from elisp source file -*- lexical-binding: t -*-

(defun org-el-file (file)
  "Create documentation in org-mode format from FILE.
FILE is an elisp source file formatted according to the emacs
style. Result is an org mode buffer containing the file's
doumentary comments and docstrings."
  (interactive "f")
  (switch-to-buffer (get-buffer-create (concat (file-name-nondirectory file) ".org")))
  (insert-file-contents file nil nil nil 'replace)
  (goto-char (point-min))
  ;; Comment-out docstrings
  (let (p0 p1 p2)
    (while (setq p0 (re-search-forward "^(def" nil t))
      (when (not (re-search-forward "^ +\"" nil t))
        (error "badly formatted file, near %d" p0))
      (setq p1 (match-beginning 0))
      (replace-match "")
      (when (not (re-search-forward "\")?$" nil t))
        (error "badly formatted file, near %d" p0))
      (setq p2 (match-beginning 0))
      (replace-match "")
      (goto-char p1)
      (narrow-to-region p1 p2) ; because p2 moves with every replacement
      (while (re-search-forward "^" nil t)
        (replace-match ";;"))
      (widen)))
  ;; Comment-out def* and adjust pre-existing comments
  (dolist (repl '(("^;;; " ";;;; ")
                  ("^$"    ";;")
                  ("^(def" ";;; (def")))
    (goto-char (point-min))
    (while (re-search-forward (car repl) nil t)
      (replace-match (cadr repl))))
  ;; Remove everything else
  (goto-char (point-min))
  (delete-non-matching-lines  "^;" (point-min) (point-max))
  ;; Move autoload declarations within their target's definition
  (goto-char (point-min))
  (while (re-search-forward "^;;;###autoload\n" nil t)
    (replace-match "")
    (re-search-forward "\n")
    (replace-match " [autoloaded]\n"))
  ;; substitute command keys
  (goto-char (point-min))
  (while (re-search-forward "\\\\\\[[^]]+]" nil t)
    (replace-match (substitute-command-keys (match-string 0))))
  ;; Create org headings and remove extra blank lines
  (dolist
      (repl '(("^;;;;" "**")
              ("^;;; (def\\([^ ]+\\) \\([^ \n]+\\)\\( ([^)]*)\\)?[^\n]*" "*** def\\1\t\\2\\3")
              ("^;;;" "***")
              ("^;;" " ")
              ("^ +$" "")
              ("\n\n+" "\n\n")))
    (goto-char (point-min))
    (while (re-search-forward (car repl) nil t)
      (replace-match (cadr repl))))
  ;; Create top heading
  (goto-char (point-min))
  (delete-char 1)
  ;; Create colophon heading
  (forward-line 1)
  (insert "** Colophon:\n")
  ;; Ta-da!
  (goto-char (point-min))
  (org-mode)
  (org-cycle) ; open up first-level headings
  (when (re-search-forward "^\*\* Commentary:" nil t)
    (goto-char (match-beginning 0))
    ;; open up content of anny commentary text
    (org-cycle)))

;; TODO:
;;
;; + The single \t inserted into "(def[^ ]+" headings in insufficient
;;   to vertically align symbol names when the "(def[^ ]+" is
;;   `define-derived-mode' or `define-obsolete-function-alias'

Re: ~Make emacs friendlier: package documentation [POC CODE INCLUDED]

[Boruch Baum]

On 2020-10-19 05:24, Eli Zaretskii wrote:
> > From: Stefan Monnier <monnier@iro.umontreal.ca>
> > Cc: Eli Zaretskii <eliz@gnu.org>,  emacs-devel@gnu.org
> > Date: Sun, 18 Oct 2020 16:18:34 -0400
> >
> > > IOW what would be the advantage of adding a link in `C-h P` to your
> > > system, instead of just replacing `C-h P` with your system?
> >
> > Oh, having just tried it, I see one other advantage: the current `C-h p`
> > displays much more quickly because it doesn't need to load `org-mode`,
> > which tends to take a while (not insanely long either, but long enough
> > to be noticeable).
>
> That is a general issue with using Org for other features: it takes
> eons to load.

Yeah, I noticed it when checking emacs-snapshot for outline-cycle.
I never noticed it before probably because my emacs loads it at start up
to create the emacs init file. I'm ok with trying outline-minor-mode
instead of org-mode once the cycle / navigation features are committed / merged.

--
hkp://keys.gnupg.net
CA45 09B5 5351 7C11 A9D1  7286 0036 9E45 1595 8BC0

Re: ~Make emacs friendlier: package documentation [POC CODE INCLUDED]

[Stefan Monnier]

> Yeah, I noticed it when checking emacs-snapshot for outline-cycle.
> I never noticed it before probably because my emacs loads it at start up
> to create the emacs init file. I'm ok with trying outline-minor-mode
> instead of org-mode once the cycle / navigation features are committed / merged.

BTW, this time-to-load problem of Org is something that would be good to
fix (basically by being more careful to load things more lazily, so
a simple Org file can open quickly and only those files which use more
functionality take longer).

It's the second time now already that we end up discussing using
outline-(minor-)mode instead of org-mode just because of org-mode's
"excessive" time to load.


        Stefan

Re: ~Make emacs friendlier: package documentation [POC CODE INCLUDED]

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > BTW, this time-to-load problem of Org is something that would be good to
  > fix (basically by being more careful to load things more lazily, so
  > a simple Org file can open quickly and only those files which use more
  > functionality take longer).

Separating some of the specific Org applications
from Org mode itself, which we should do to make the design clearer,
would help with this as well.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: ~Make emacs friendlier: package documentation [POC CODE INCLUDED]

[Boruch Baum]

On 2020-10-20 01:11, Richard Stallman wrote:
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > BTW, this time-to-load problem of Org is something that would be good to
>   > fix (basically by being more careful to load things more lazily, so
>   > a simple Org file can open quickly and only those files which use more
>   > functionality take longer).
>
> Separating some of the specific Org applications
> from Org mode itself, which we should do to make the design clearer,
> would help with this as well.

The org-mode people have their own list, so this may only be reaching
the eyes of some sub-set of them. Maybe cross-post this suggestion?

--
hkp://keys.gnupg.net
CA45 09B5 5351 7C11 A9D1  7286 0036 9E45 1595 8BC0

Re: ~Make emacs friendlier: package documentation [POC CODE INCLUDED]

[Kévin Le Gouguec]

Boruch Baum <boruch_baum@gmx.com> writes:

> On 2020-10-18 22:40, Kévin Le Gouguec wrote:
>> Boruch Baum <boruch_baum@gmx.com> writes:
>>
>> >> and I'm not exactly sure what using Org-mode buys us in this case.
>> >
>> > For starters: Navigation, expansion and collapse of sections.
>>
>> FWIW, outline-mode now features section-cycling commands (TAB and S-TAB)
>> that are very similar to org-mode's (cf. bug#41130).
>
> I'm OK with trying to use outline-minor-mode, but didn't see the
> features yet in my version of emacs-snapshot.

Right, those commands are only bound for the major mode (outline-mode).
If section cycling and navigation commands are the main thing drawing
you to org-mode for this feature, maybe outline-mode is enough?

Re: ~Make emacs friendlier: package documentation [POC CODE INCLUDED]

[Boruch Baum]

On 2020-10-21 07:52, Kévin Le Gouguec wrote:
> Boruch Baum <boruch_baum@gmx.com> writes:
>
> > On 2020-10-18 22:40, Kévin Le Gouguec wrote:
> >> Boruch Baum <boruch_baum@gmx.com> writes:
> >>
> >> >> and I'm not exactly sure what using Org-mode buys us in this case.
> >> >
> >> > For starters: Navigation, expansion and collapse of sections.
> >>
> >> FWIW, outline-mode now features section-cycling commands (TAB and S-TAB)
> >> that are very similar to org-mode's (cf. bug#41130).
> >
> > I'm OK with trying to use outline-minor-mode, but didn't see the
> > features yet in my version of emacs-snapshot.
>
> Right, those commands are only bound for the major mode (outline-mode).
> If section cycling and navigation commands are the main thing drawing
> you to org-mode for this feature, maybe outline-mode is enough?

OK. I see the reason I didn't see the feature was that the commit wasn't
pushed until 2020-10-19.

--
hkp://keys.gnupg.net
CA45 09B5 5351 7C11 A9D1  7286 0036 9E45 1595 8BC0

Re: ~Make emacs friendlier: package documentation [POC CODE INCLUDED]

[Stefan Monnier]

Hi, here are my comments written on the fly as I read your code.

While I'm here: have you looked into what it would take to add the magic
link to the `C-h P` page?


        Stefan


> ;;; org-el-file.el --- Make org documetation from elisp source file -*- lexical-binding: t -*-

[ Of course, this file would need the usual copyright/license blurb,
  here, as well as the `;;; Commentary:` and `;;; Code:` sections so it
  can be nicely viewed with itself.  ]

> (defun org-el-file (file)
>   "Create documentation in org-mode format from FILE.
> FILE is an elisp source file formatted according to the emacs
                                                          ^^^^^
I think our illustrious software's name deserves to be capitalized ;-)

> style. Result is an org mode buffer containing the file's
       ^^^
By convention, we put two spaces between sentences (this is used by
sentence navigation and filling according to `sentence-end-double-space`).

> doumentary comments and docstrings."
>   (interactive "f")

I think this should use an interactive spec similar to that of `find-library`.

>   (switch-to-buffer (get-buffer-create (concat (file-name-nondirectory file) ".org")))

Please never ever use `switch-to-buffer` in your ELisp code, since its
meaning is unclear (OT1H it says "change content of selected window" and
OTOH it says "display this buffer", making it unclear what should happen
when this buffer can't be displayed in the selected window).
E.g. Use `display-buffer`, `pop-to-buffer`, `pop-to-buffer-same-window`, ...

The buffer name you chose looks like a file-buffer's name, whereas this
is a "special buffer" not linked to any file, for which we traditionally
use names surrounded by stars.  Also, I don't think ".org" is
needed there.  One more thing, using just (file-name-nondirectory file)
isn't good enough for files like `lisp/cedet/semantic/bovine/debug.el`
(since that conflicts with `lisp/emacs-lisp/debug.el`; the package's
name is `semantic/bovine/debug.el`).  If you use my recommendation above
for the interactive spec, then the user will have to write something
like "semantic/bovine/debug" and you can reuse that name directly
without having to strip anything from it.

Of course, you could also opt to go in the direction of making your
buffer into a "real" file-buffer (or even a real file) in which case
you could use `create-file-buffer` (which will choose the right buffer
name based on the usual rules such as uniquify).

>   (insert-file-contents file nil nil nil 'replace)
>   (goto-char (point-min))
>   ;; Comment-out docstrings
>   (let (p0 p1 p2)
>     (while (setq p0 (re-search-forward "^(def" nil t))
>       (when (not (re-search-forward "^ +\"" nil t))
>         (error "badly formatted file, near %d" p0))

[ You can rewrite `when (not` to `unless`. ]
[ You don't need `p0` here, you can use (match-beginning 0) instead.  ]

I don't think we should treat this as an error.  Also, I think it's
important to bound the search to the sexp that's started by the
open-paren.

Many sexps that start with `(def` don't have docstrings (e.g. all those
(defvar FOO) which are just there to tell the byte-compiler that we're
expecting those vars to be defined elsewhere), so we should be careful
not to incorrectly match a docstring with some unrelated previous
`(defvar`.

>       (setq p1 (match-beginning 0))

I recommend you use `let` for it here (so you never need to `setq` the
variable, which saves you from having to think about which value of that
variable you're getting when you refer to it).  Same for `p2`: use `let`
at the point where you can give it its real value rather than having
a "dummy" let followed by a `setq` later.

>       (replace-match "")
>       (when (not (re-search-forward "\")?$" nil t))
>         (error "badly formatted file, near %d" p0))

A " char can legitimately appear at the end of a line *within*
a docstring .  Better use `forward-sexp` to jump over the docstring
while obeying the usual backslash escaping rules (but make sure you set
the buffer in `emacs-lisp-mode`, first).

>       (goto-char p1)
>       (narrow-to-region p1 p2) ; because p2 moves with every replacement
>       (while (re-search-forward "^" nil t)
>         (replace-match ";;"))
>       (widen)))

The better option, IMO is to do

    (goto-char p2)
    (while (> (point) p1) ... (forward-line -1) ...)

>   ;; Comment-out def* and adjust pre-existing comments
>   (dolist (repl '(("^;;; " ";;;; ")
>                   ("^$"    ";;")
>                   ("^(def" ";;; (def")))

Traditionally the outline convention used in Elisp is that ";;; " is
a top-level heading, ";;;; " is a subheading, ";;;;; " is
a subsubheading (and ";;" is not a heading at all), whereas you seem to
use a convention that's inverted in this respect, which will not play
well with files which use ";;;; " and friends (which are often the
better structured ones, IME).

>   (dolist
>       (repl '(("^;;;;" "**")
>               ("^;;; (def\\([^ ]+\\) \\([^ \n]+\\)\\( ([^)]*)\\)?[^\n]*" "*** def\\1\t\\2\\3")
>               ("^;;;" "***")
>               ("^;;" " ")
>               ("^ +$" "")
>               ("\n\n+" "\n\n")))

It doesn't really matter, admittedly, but it does seem like we could
avoid the intermediate step of adding the semi-colons at the beginning
of "all" the lines only to remove them right after.

>     (goto-char (point-min))
>     (while (re-search-forward (car repl) nil t)
>       (replace-match (cadr repl))))

If we keep the two-step approach, then you'll probably want to define an
auxiliary function which takes a list of regexp+replacement and does the
searches+replacements.

>   ;; Create top heading
>   (goto-char (point-min))
>   (delete-char 1)

Which char do we intend to delete here and why?

>   ;; Create colophon heading
>   (forward-line 1)
>   (insert "** Colophon:\n")

I see you don't use the top-level "*" headers at all.  Any specific
reason for that?

>   ;; Ta-da!
>   (goto-char (point-min))
>   (org-mode)
>   (org-cycle) ; open up first-level headings

I'd do the `goto-char` after setting up org-mode, just in case org-mode
moves point: I know it arguably shouldn't/won't, but it doesn't cost
anything to switch the two and it saves us from having to worry about
it.  Also, if `org-cycle` may open, but it may do other things as well
(it's a DWIM command meant for interactive use), so in ELisp code we're
better off using a lower-level function which only does "open up
first-level headings", which should also save us from having to write
a comment explaining what we're intending to do.

>   (when (re-search-forward "^\*\* Commentary:" nil t)

These backslashes don't do any good here (as evidenced by the red
warning faced applied to the by font-lock ;-).  You probably intended for
them to be doubled.

>    ;; open up content of anny commentary text
                           ^^^^
                      short for anniversary?