Use emacs docstrings in sphinx-docs

Use emacs docstrings in sphinx-docs

[David Bremner]

This series obsoletes the WIP series at

     id:20180527185530.6795-1-david@tethera.net

I think it's ready for discussion and potential merging.

This is a simple tool that loads an elisp file, and extracts all the
variables/functions it defines and the corresponding
docstrings.

[PATCH 01/11] emacs: initial version of rstdoc.el

These are technically bugfixes for the elisp, and could be applied independent
of the rest of the series.

[PATCH 02/11] emacs: require notmuch-lib from notmuch-wash.el
[PATCH 03/11] emacs: escape quote in docstring

There are 3 patches worth of build system glue. In particular, we now
only build the emacs-manual if we have emacs.

[PATCH 04/11] emacs: build docstring (rsti) files
[PATCH 05/11] doc: only build notmuch-emacs docs if Emacs is present
[PATCH 06/11] doc/emacs: require extracted docstrings for sphinx or

The remaining patches in the series use the extracted docstrings to
start filling in the notmuch-emacs manual.

[PATCH 01/11] emacs: initial version of rstdoc.el

[David Bremner]

This small library is intended to support batch extraction of Emacs
Lisp docstrings from source files. Clients will need to include (or
replace) rstdoc.rsti.
---
 emacs/rstdoc.el   | 85 +++++++++++++++++++++++++++++++++++++++++++++++
 emacs/rstdoc.rsti | 22 ++++++++++++
 2 files changed, 107 insertions(+)
 create mode 100644 emacs/rstdoc.el
 create mode 100644 emacs/rstdoc.rsti

diff --git a/emacs/rstdoc.el b/emacs/rstdoc.el
new file mode 100644
index 00000000..2225aefc
--- /dev/null
+++ b/emacs/rstdoc.el
@@ -0,0 +1,85 @@
+;;; rstdoc.el --- help generate documentation from docstrings -*-lexical-binding: t-*-
+
+;; Copyright (C) 2018 David Bremner
+
+;; Author: David Bremner <david@tethera.net>
+;; Created: 26 May 2018
+;; Keywords: emacs lisp, documentation
+;; Homepage: https://notmuchmail.org
+
+;; This file is not part of GNU Emacs.
+
+;; rstdoc.el is free software: you can redistribute it and/or modify it
+;; under the terms of the GNU General Public License as published by
+;; the Free Software Foundation, either version 3 of the License, or
+;; (at your option) any later version.
+;;
+;; rstdoc.el is distributed in the hope that it will be useful, but
+;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+;; General Public License for more details.
+;;
+;; You should have received a copy of the GNU General Public License
+;; along with rstdoc.el.  If not, see <https://www.gnu.org/licenses/>.
+;;
+
+;;; Commentary:
+;;
+
+;; Rstdoc provides a facility to extract all of the docstrings defined in
+;; an elisp source file. Usage:
+;;
+;; emacs -Q --batch -L . -l rstdoc -f rstdoc-batch-extract foo.el foo.rsti
+
+;;; Code:
+
+(provide 'rstdoc)
+
+(defun rstdoc-batch-extract ()
+  "Extract docstrings to and from the files on the command line"
+  (apply #'rstdoc-extract command-line-args-left))
+
+(defun rstdoc-extract (in-file out-file)
+  "Write docstrings from IN-FILE to OUT-FILE"
+  (load-file in-file)
+  (let* ((definitions (cdr (assoc (expand-file-name in-file) load-history)))
+	 (doc-hash (make-hash-table :test 'eq)))
+    (mapc
+     (lambda (elt)
+       (let ((pair
+	      (pcase elt
+		(`(defun . ,name) (cons name (documentation name)))
+		(`(,_ . ,_)  nil)
+		(sym (cons sym (get sym 'variable-documentation))))))
+	 (when (and pair (cdr pair))
+	   (puthash (car pair) (cdr pair) doc-hash))))
+     definitions)
+    (with-temp-buffer
+      (maphash
+       (lambda (key val)
+	 (rstdoc--insert-docstring key val))
+       doc-hash)
+      (write-region (point-min) (point-max) out-file))))
+
+(defun rstdoc--insert-docstring (symbol docstring)
+  (insert (format "\n.. |docstring::%s| replace::\n" symbol))
+  (insert (replace-regexp-in-string "^" "    " (rstdoc--rst-quote-string docstring)))
+  (insert "\n"))
+
+(defvar rst--escape-alist
+  '( ("\\\\='" . "\\\\'")
+     ("\\([^\\]\\)'" . "\\1`")
+     ("^[[:space:]\t]*$" . "|br|")
+     ("^[[:space:]\t]" . "|indent| "))
+    "list of (regex . replacement) pairs")
+
+(defun rstdoc--rst-quote-string (str)
+  (with-temp-buffer
+    (insert str)
+    (dolist (pair rst--escape-alist)
+      (goto-char (point-min))
+      (while (re-search-forward (car pair) nil t)
+	(replace-match (cdr pair))))
+    (buffer-substring (point-min) (point-max))))
+
+;;; rstdoc.el ends here
diff --git a/emacs/rstdoc.rsti b/emacs/rstdoc.rsti
new file mode 100644
index 00000000..f138b7bf
--- /dev/null
+++ b/emacs/rstdoc.rsti
@@ -0,0 +1,22 @@
+.. -*- rst -*-
+
+.. |br| replace:: |br-texinfo| |br-html|
+       
+.. |br-texinfo| raw:: texinfo
+
+   @* @*
+
+.. |br-html| raw:: html
+
+   <br /><br />
+
+.. |indent| replace:: |indent-texinfo| |indent-html|
+
+.. |indent-texinfo| raw:: texinfo
+
+   @* @ @ @ @
+
+.. |indent-html| raw:: html
+
+   <br />&nbsp;&nbsp;&nbsp;&nbsp;
+
-- 
2.17.1

[PATCH 02/11] emacs: require notmuch-lib from notmuch-wash.el

[David Bremner]

This is needed so that notmuch-wash.el is loadable by itself; in
particular for the docstring processing.
---
 emacs/notmuch-wash.el | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/emacs/notmuch-wash.el b/emacs/notmuch-wash.el
index 5f8b9267..54108d93 100644
--- a/emacs/notmuch-wash.el
+++ b/emacs/notmuch-wash.el
@@ -24,7 +24,7 @@
 ;;; Code:
 
 (require 'coolj)
-
+(require 'notmuch-lib)
 (declare-function notmuch-show-insert-bodypart "notmuch-show" (msg part depth &optional hide))
 (defvar notmuch-show-indent-messages-width)
 
-- 
2.17.1

[PATCH 03/11] emacs: escape quote in docstring

[David Bremner]

This prevents emacs from turning it into a fancy, non-evaluable
docstring (and also makes it easier to deal with during extraction to
sphinx).
---
 emacs/notmuch.el | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/emacs/notmuch.el b/emacs/notmuch.el
index 44402f8a..1f453357 100644
--- a/emacs/notmuch.el
+++ b/emacs/notmuch.el
@@ -711,7 +711,7 @@ A thread with TAG will have FACE applied.
 Here is an example of how to color search results based on tags.
  (the following text would be placed in your ~/.emacs file):
 
- (setq notmuch-search-line-faces '((\"unread\" . (:foreground \"green\"))
+ (setq notmuch-search-line-faces \\='((\"unread\" . (:foreground \"green\"))
                                    (\"deleted\" . (:foreground \"red\"
 						  :background \"blue\"))))
 
-- 
2.17.1

[PATCH 04/11] emacs: build docstring (rsti) files

[David Bremner]

These are intended to included in the sphinx manual for notmuch-emacs.
The stamp file makes it easier to depend on the docstrings from other
parts of the build
---
 emacs/.gitignore     |  1 +
 emacs/Makefile.local | 11 +++++++++--
 2 files changed, 10 insertions(+), 2 deletions(-)

diff --git a/emacs/.gitignore b/emacs/.gitignore
index fbf8dde6..b9873b0a 100644
--- a/emacs/.gitignore
+++ b/emacs/.gitignore
@@ -1,4 +1,5 @@
 /.eldeps*
 /*.elc
+/*.rsti
 /notmuch-version.el
 /notmuch-pkg.el
diff --git a/emacs/Makefile.local b/emacs/Makefile.local
index 1b3ef584..38d1f5c5 100644
--- a/emacs/Makefile.local
+++ b/emacs/Makefile.local
@@ -45,6 +45,10 @@ emacs_images := \
 	$(srcdir)/$(dir)/notmuch-logo.png
 
 emacs_bytecode = $(emacs_sources:.el=.elc)
+emacs_docstrings = $(emacs_sources:.el=.rsti)
+
+docstring.stamp: ${emacs_docstrings}
+	touch $@
 
 # Because of defmacro's and defsubst's, we have to account for load
 # dependencies between Elisp files when byte compiling.  Otherwise,
@@ -76,6 +80,8 @@ CLEAN+=$(dir)/.eldeps $(dir)/.eldeps.tmp $(dir)/.eldeps.x
 ifeq ($(HAVE_EMACS),1)
 %.elc: %.el $(global_deps)
 	$(call quiet,EMACS) --directory emacs -batch -f batch-byte-compile $<
+%.rsti: %.el
+	$(call quiet,EMACS) -batch -L emacs -l rstdoc -f rstdoc-batch-extract $< $@
 endif
 
 elpa: $(ELPA_FILE)
@@ -93,7 +99,7 @@ endif
 
 ifeq ($(WITH_EMACS),1)
 ifeq ($(HAVE_EMACS),1)
-all: $(emacs_bytecode)
+all: $(emacs_bytecode) $(emacs_docstrings)
 install-emacs: $(emacs_bytecode)
 endif
 
@@ -117,4 +123,5 @@ ifeq ($(WITH_DESKTOP),1)
 	-update-desktop-database "$(DESTDIR)$(desktop_dir)"
 endif
 
-CLEAN := $(CLEAN) $(emacs_bytecode) $(dir)/notmuch-version.el $(dir)/notmuch-pkg.el
+CLEAN := $(CLEAN) $(emacs_bytecode) $(dir)/notmuch-version.el $(dir)/notmuch-pkg.el \
+	$(emacs_docstrings) docstring.stamp
-- 
2.17.1

[PATCH 05/11] doc: only build notmuch-emacs docs if Emacs is present

[David Bremner]

For notmuch-emacs-mua, this is just guessing what the user wants, but
for notmuch-emacs.rst, emacs will now be required to extract the
docstrings for the manual.
---
 doc/Makefile.local |  4 ++++
 doc/conf.py        | 24 +++++++++++++++++-------
 2 files changed, 21 insertions(+), 7 deletions(-)

diff --git a/doc/Makefile.local b/doc/Makefile.local
index 16459e35..69853454 100644
--- a/doc/Makefile.local
+++ b/doc/Makefile.local
@@ -7,6 +7,10 @@ SPHINXOPTS    := -q
 SPHINXBUILD   = sphinx-build
 DOCBUILDDIR      := $(dir)/_build
 
+ifeq ($(WITH_EMACS),1)
+SPHINXOPTS += -t with_emacs
+endif
+
 # Internal variables.
 ALLSPHINXOPTS   := -d $(DOCBUILDDIR)/doctrees $(SPHINXOPTS) $(srcdir)/$(dir)
 APIMAN		:= $(DOCBUILDDIR)/man/man3/notmuch.3
diff --git a/doc/conf.py b/doc/conf.py
index efbafc93..25f8146c 100644
--- a/doc/conf.py
+++ b/doc/conf.py
@@ -29,6 +29,9 @@ release = version
 # directories to ignore when looking for source files.
 exclude_patterns = ['_build']
 
+if not tags.has('with_emacs'):
+    exclude_patterns += ['notmuch-emacs*','man1/notmuch-emacs*']
+
 # The name of the Pygments (syntax highlighting) style to use.
 pygments_style = 'sphinx'
 
@@ -81,12 +84,15 @@ man_pages = [
 
     ('man1/notmuch-dump', 'notmuch-dump',
      u'creates a plain-text dump of the tags of each message',
-     [notmuch_authors], 1),
+     [notmuch_authors], 1) ]
 
+if tags.has('with_emacs'):
+    man_pages += [
     ('man1/notmuch-emacs-mua', 'notmuch-emacs-mua',
      u'send mail with notmuch and emacs',
-     [notmuch_authors], 1),
+     [notmuch_authors], 1) ]
 
+man_pages += [
     ('man5/notmuch-hooks', 'notmuch-hooks',
      u'hooks for notmuch',
      [notmuch_authors], 5),
@@ -143,11 +149,15 @@ man_pages = [
 # If true, do not generate a @detailmenu in the "Top" node's menu.
 texinfo_no_detailmenu = True
 
-texinfo_documents = [
-    ('notmuch-emacs', 'notmuch-emacs', u'notmuch-emacs documentation',
-     notmuch_authors, 'notmuch-emacs',
-     'emacs based front-end for notmuch', 'Miscellaneous'),
-]
+if tags.has('with_emacs'):
+    texinfo_documents = [
+        ('notmuch-emacs', 'notmuch-emacs', u'notmuch-emacs documentation',
+         notmuch_authors, 'notmuch-emacs',
+         'emacs based front-end for notmuch', 'Miscellaneous'),
+    ]
+else:
+    texinfo_documents = []
+
 
 # generate texinfo list from man page list
 texinfo_documents += [
-- 
2.17.1

[PATCH 06/11] doc/emacs: require extracted docstrings for sphinx or info manual

[David Bremner]

We need to use the stamp file here in order not to depend on the order
the submakefiles are included.
---
 doc/Makefile.local | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/doc/Makefile.local b/doc/Makefile.local
index 69853454..604de767 100644
--- a/doc/Makefile.local
+++ b/doc/Makefile.local
@@ -41,6 +41,10 @@ INFO_INFO_FILES := $(INFO_TEXI_FILES:.texi=.info)
 %.gz: %
 	rm -f $@ && gzip --stdout $^ > $@
 
+ifeq ($(WITH_EMACS),1)
+sphinx-html sphinx-texinfo: docstring.stamp
+endif
+
 sphinx-html:
 	$(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(DOCBUILDDIR)/html
 
-- 
2.17.1

[PATCH 07/11] doc/emacs: add documentation for stashing 'c X' bindings

[David Bremner]

This is the first of a series of changes requiring the extracted docstrings.
---
 doc/notmuch-emacs.rst | 60 +++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 60 insertions(+)

diff --git a/doc/notmuch-emacs.rst b/doc/notmuch-emacs.rst
index ce2e358e..82bb35a0 100644
--- a/doc/notmuch-emacs.rst
+++ b/doc/notmuch-emacs.rst
@@ -190,6 +190,9 @@ pressing RET after positioning the cursor on a hidden part.
     advance to the next message, or advance to the next thread (if
     already on the last message of a thread).
 
+``c``
+    :ref:`show-copy`
+
 ``N``
     Move to next message
 
@@ -208,6 +211,55 @@ pressing RET after positioning the cursor on a hidden part.
 ``?``
     Display full set of key bindings
 
+.. _show-copy:
+
+Copy to kill-ring
+-----------------
+
+You can use the usually Emacs ways of copying text to the kill-ring,
+but notmuch also provides some shortcuts. These keys are available in
+:ref:`notmuch-show`, and :ref:`notmuch-tree`. A subset are available
+in :ref:`notmuch-search`.
+
+``c F``	``notmuch-show-stash-filename``
+   |docstring::notmuch-show-stash-filename|
+
+``c G`` ``notmuch-show-stash-git-send-email``
+   |docstring::notmuch-show-stash-git-send-email|
+
+``c I`` ``notmuch-show-stash-message-id-stripped``
+   |docstring::notmuch-show-stash-message-id-stripped|
+
+``c L`` ``notmuch-show-stash-mlarchive-link-and-go``
+   |docstring::notmuch-show-stash-mlarchive-link-and-go|
+
+``c T`` ``notmuch-show-stash-tags``
+   |docstring::notmuch-show-stash-tags|
+
+``c c`` ``notmuch-show-stash-cc``
+   |docstring::notmuch-show-stash-cc|
+
+``c d`` ``notmuch-show-stash-date``
+   |docstring::notmuch-show-stash-date|
+
+``c f`` ``notmuch-show-stash-from``
+   |docstring::notmuch-show-stash-from|
+
+``c i`` ``notmuch-show-stash-message-id``
+   |docstring::notmuch-show-stash-message-id|
+
+``c l`` ``notmuch-show-stash-mlarchive-link``
+   |docstring::notmuch-show-stash-mlarchive-link|
+
+``c s`` ``notmuch-show-stash-subject``
+   |docstring::notmuch-show-stash-subject|
+
+``c t`` ``notmuch-show-stash-to``
+   |docstring::notmuch-show-stash-to|
+
+``c ?``
+    Show all available copying commands
+
 .. _notmuch-tree:
 
 notmuch-tree
@@ -218,6 +270,9 @@ email archives. Each line in the buffer represents a single
 message giving the relative date, the author, subject, and any
 tags.
 
+``c``
+    :ref:`show-copy`
+
 ``<return>``
    Displays that message.
 
@@ -300,3 +355,8 @@ suffix exist it will be read instead (just one of these, chosen in this
 order). Most often users create ``~/.emacs.d/notmuch-config.el`` and just
 work with it. If Emacs was invoked with the ``-q`` or ``--no-init-file``
 options, ``notmuch-init-file`` is not read.
+
+.. include:: ../emacs/rstdoc.rsti
+
+
+.. include:: ../emacs/notmuch-show.rsti
-- 
2.17.1

[PATCH 08/11] doc/emacs: document notmuch-message-headers*

[David Bremner]

More precisely, copy the docstrings into notmuch-emacs documentation pages.
---
 doc/notmuch-emacs.rst | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/doc/notmuch-emacs.rst b/doc/notmuch-emacs.rst
index 82bb35a0..7026c609 100644
--- a/doc/notmuch-emacs.rst
+++ b/doc/notmuch-emacs.rst
@@ -211,6 +211,14 @@ pressing RET after positioning the cursor on a hidden part.
 ``?``
     Display full set of key bindings
 
+Display of messages can be controlled by the following variables
+
+:index:`notmuch-message-headers`
+       |docstring::notmuch-message-headers|
+
+:index:`notmuch-message-headers-visible`
+       |docstring::notmuch-message-headers-visible|
+
 .. _show-copy:
 
 Copy to kill-ring
-- 
2.17.1

[PATCH 09/11] doc/emacs: document notmuch-tagging-keys

[David Bremner]

Calling these "Global keys" is arguably a bit of a stretch, but they
do work in all notmuch modes except notmuch-hello.
---
 doc/notmuch-emacs.rst | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/doc/notmuch-emacs.rst b/doc/notmuch-emacs.rst
index 7026c609..2b3da904 100644
--- a/doc/notmuch-emacs.rst
+++ b/doc/notmuch-emacs.rst
@@ -338,7 +338,7 @@ operations specified in ``notmuch-tagging-keys``; i.e. each
 
 :index:`notmuch-tagging-keys`
 
-   A list of keys and corresponding tagging operations.
+  |docstring::notmuch-tagging-keys|
 
 Configuration
 =============
@@ -368,3 +368,5 @@ options, ``notmuch-init-file`` is not read.
 
 
 .. include:: ../emacs/notmuch-show.rsti
+
+.. include:: ../emacs/notmuch-tag.rsti
-- 
2.17.1

[PATCH 10/11] doc/emacs: document notmuch-poll*

[David Bremner]

The current "documentation" for these variables consists of only the
variable names.
---
 doc/notmuch-emacs.rst | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/doc/notmuch-emacs.rst b/doc/notmuch-emacs.rst
index 2b3da904..b9eeffee 100644
--- a/doc/notmuch-emacs.rst
+++ b/doc/notmuch-emacs.rst
@@ -349,8 +349,10 @@ Importing Mail
 --------------
 
 :index:`notmuch-poll`
+   |docstring::notmuch-poll|
 
 :index:`notmuch-poll-script`
+   |docstring::notmuch-poll-script|
 
 Init File
 ---------
@@ -366,6 +368,7 @@ options, ``notmuch-init-file`` is not read.
 
 .. include:: ../emacs/rstdoc.rsti
 
+.. include:: ../emacs/notmuch-lib.rsti
 
 .. include:: ../emacs/notmuch-show.rsti
 
-- 
2.17.1

[PATCH 11/11] doc/emacs: document notmuch-cycle-notmuch-buffers

[David Bremner]

For some reason I couldn't find this when I searched, so add it to the
manual.
---
 doc/notmuch-emacs.rst | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/doc/notmuch-emacs.rst b/doc/notmuch-emacs.rst
index b9eeffee..2e85b8a5 100644
--- a/doc/notmuch-emacs.rst
+++ b/doc/notmuch-emacs.rst
@@ -340,6 +340,12 @@ operations specified in ``notmuch-tagging-keys``; i.e. each
 
   |docstring::notmuch-tagging-keys|
 
+Buffer navigation
+=================
+
+:index:`notmuch-cycle-notmuch-buffers`
+   |docstring::notmuch-cycle-notmuch-buffers|
+
 Configuration
 =============
 
@@ -368,6 +374,8 @@ options, ``notmuch-init-file`` is not read.
 
 .. include:: ../emacs/rstdoc.rsti
 
+.. include:: ../emacs/notmuch.rsti
+
 .. include:: ../emacs/notmuch-lib.rsti
 
 .. include:: ../emacs/notmuch-show.rsti
-- 
2.17.1

Re: [PATCH 02/11] emacs: require notmuch-lib from notmuch-wash.el

[David Bremner]

David Bremner <david@tethera.net> writes:

> This is needed so that notmuch-wash.el is loadable by itself; in
> particular for the docstring processing.

pushed patches 2 and 3

d

Re: [PATCH 01/11] emacs: initial version of rstdoc.el

[David Bremner]

David Bremner <david@tethera.net> writes:

> This small library is intended to support batch extraction of Emacs
> Lisp docstrings from source files. Clients will need to include (or
> replace) rstdoc.rsti.

Heads up:

I've marked this series as ready to merge. Any objections should come
ASAP.

d

Re: [PATCH 01/11] emacs: initial version of rstdoc.el

[David Bremner]

David Bremner <david@tethera.net> writes:

> David Bremner <david@tethera.net> writes:
>
>> This small library is intended to support batch extraction of Emacs
>> Lisp docstrings from source files. Clients will need to include (or
>> replace) rstdoc.rsti.
>
> Heads up:
>
> I've marked this series as ready to merge. Any objections should come
> ASAP.
>

As promised/threatened I have merged this series to master

d