[RFC PATCH 0/4] doc: use Hawkmoth for API documentation

[RFC PATCH 0/4] doc: use Hawkmoth for API documentation

[Jani Nikula]

Hawkmoth [1] is a minimalistic Sphinx C Domain autodoc directive
extension to incorporate formatted C source code comments into Sphinx
based documentation, by yours truly. See commit message of patch 2 for
details.

This series is probably more a real world test case for Hawkmoth than a
serious attempt at getting it merged to Notmuch. Nonetheless, I decided
to post it as food for thought. With hopes that one day it'll be a
serious and compelling alternative for Sphinx based C API documentation.

'make sphinx-html' is probably more interesting than 'make build-man'.

BR,
Jani.


[1] https://github.com/jnikula/hawkmoth


Jani Nikula (4):
  doc: add man section 3 handling
  doc: use hawkmoth for generating libnotmuch api documentation
  lib: fix the biggest hawkmoth offenders in notmuch.h
  lib: add reStructuredText bling for API documentation in notmuch.h

 doc/Makefile.local      |  10 ++--
 doc/conf.py             |  14 +++++
 doc/index.rst           |   1 +
 doc/man3/libnotmuch.rst |  20 +++++++
 lib/notmuch.h           | 139 ++++++++++++++++++++++++++----------------------
 5 files changed, 117 insertions(+), 67 deletions(-)
 create mode 100644 doc/man3/libnotmuch.rst

-- 
2.11.0

[RFC PATCH 1/4] doc: add man section 3 handling

[Jani Nikula]

We don't actually add any section 3 man pages, but be prepared.
---
 doc/Makefile.local | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)

diff --git a/doc/Makefile.local b/doc/Makefile.local
index c6f05ca879c0..04ff49b7a7fb 100644
--- a/doc/Makefile.local
+++ b/doc/Makefile.local
@@ -13,14 +13,16 @@ APIMAN		:= $(DOCBUILDDIR)/man/man3/notmuch.3
 DOXYFILE	:= $(srcdir)/$(dir)/doxygen.cfg
 
 MAN1_RST := $(wildcard $(srcdir)/doc/man1/*.rst)
+MAN3_RST := $(wildcard $(srcdir)/doc/man3/*.rst)
 MAN5_RST := $(wildcard $(srcdir)/doc/man5/*.rst)
 MAN7_RST := $(wildcard $(srcdir)/doc/man7/*.rst)
-MAN_RST_FILES := $(MAN1_RST) $(MAN5_RST) $(MAN7_RST)
+MAN_RST_FILES := $(MAN1_RST) $(MAN3_RST) $(MAN5_RST) $(MAN7_RST)
 
 MAN1_ROFF := $(patsubst $(srcdir)/doc/%,$(DOCBUILDDIR)/man/%,$(MAN1_RST:.rst=.1))
+MAN3_ROFF := $(patsubst $(srcdir)/doc/%,$(DOCBUILDDIR)/man/%,$(MAN3_RST:.rst=.3))
 MAN5_ROFF := $(patsubst $(srcdir)/doc/%,$(DOCBUILDDIR)/man/%,$(MAN5_RST:.rst=.5))
 MAN7_ROFF := $(patsubst $(srcdir)/doc/%,$(DOCBUILDDIR)/man/%,$(MAN7_RST:.rst=.7))
-MAN_ROFF_FILES := $(MAN1_ROFF) $(MAN5_ROFF) $(MAN7_ROFF)
+MAN_ROFF_FILES := $(MAN1_ROFF) $(MAN3_ROFF) $(MAN5_ROFF) $(MAN7_ROFF)
 
 MAN_GZIP_FILES := $(addsuffix .gz,${MAN_ROFF_FILES})
 
@@ -51,7 +53,7 @@ ${MAN_ROFF_FILES}: $(DOCBUILDDIR)/.roff.stamp
 $(DOCBUILDDIR)/.roff.stamp: ${MAN_RST_FILES}
 ifeq ($(HAVE_SPHINX),1)
 	$(SPHINXBUILD) -b man $(ALLSPHINXOPTS) $(DOCBUILDDIR)/man
-	for section in 1 5 7; do \
+	for section in 1 3 5 7; do \
 	    mkdir -p $(DOCBUILDDIR)/man/man$${section}; \
 	    mv $(DOCBUILDDIR)/man/*.$${section} $(DOCBUILDDIR)/man/man$${section}; \
 	done
@@ -90,9 +92,11 @@ else
 build-man: ${MAN_GZIP_FILES}
 install-man: ${MAN_GZIP_FILES}
 	mkdir -p "$(DESTDIR)$(mandir)/man1"
+	mkdir -p "$(DESTDIR)$(mandir)/man3"
 	mkdir -p "$(DESTDIR)$(mandir)/man5"
 	mkdir -p "$(DESTDIR)$(mandir)/man7"
 	install -m0644 $(filter %.1.gz,$(MAN_GZIP_FILES)) $(DESTDIR)/$(mandir)/man1
+	install -m0644 $(filter %.3.gz,$(MAN_GZIP_FILES)) $(DESTDIR)/$(mandir)/man3
 	install -m0644 $(filter %.5.gz,$(MAN_GZIP_FILES)) $(DESTDIR)/$(mandir)/man5
 	install -m0644 $(filter %.7.gz,$(MAN_GZIP_FILES)) $(DESTDIR)/$(mandir)/man7
 	cd $(DESTDIR)/$(mandir)/man1 && ln -sf notmuch.1.gz notmuch-setup.1.gz
-- 
2.11.0

[RFC PATCH 2/4] doc: use hawkmoth for generating libnotmuch api documentation

[Jani Nikula]

Hawkmoth [1] is a minimalistic Sphinx C Domain autodoc directive
extension to incorporate formatted C source code comments into Sphinx
based documentation, by yours truly.

This adds libnotmuch(3) man page to incorporate API documentation
using the extension. The naming is intentional to not conflict with
the Doxygen based man page, for now.

Hawkmoth has a few advantages over Doxygen. It directly integrates
with Sphinx, and allows reStructuredText syntax in the documentation
comments. It's simple and minimal, without the overlap of Doxygen and
Sphinx.

Of course, there are some disadvantages as well. Hawkmoth is a project
very much in the larval stage. It's not even packaged yet. It
undoubtedly has plenty of unexpected issues and bugs. It only has one
contributor to fix them. It only targets C documentation.

Hawkmoth C parsing is based on Clang Python Bindings, but then Doxygen
also depends on Clang.

On Debian, the minimum dependencies are:

 * python-clang-3.8
 * libclang-3.8-dev
 * libclang1-3.8

Unfortunately python-clang seems to be for Python 2 only, so you'll
need to use Python 2 based Sphinx as well. (To be investigated.)

As can be seen from the size of this patch, the simplicity of Hawkmoth
makes it a compelling alternative for Sphinx based C API
documentation.

[1] https://github.com/jnikula/hawkmoth
---
 doc/conf.py             | 14 ++++++++++++++
 doc/index.rst           |  1 +
 doc/man3/libnotmuch.rst | 20 ++++++++++++++++++++
 3 files changed, 35 insertions(+)
 create mode 100644 doc/man3/libnotmuch.rst

diff --git a/doc/conf.py b/doc/conf.py
index 0e65413d917a..4b25cfd4dbf2 100644
--- a/doc/conf.py
+++ b/doc/conf.py
@@ -4,6 +4,9 @@
 import sys
 import os
 
+# FIXME: Use PYTHONPATH=$PYTHONPATH:/path/to/hawkmoth or this:
+#sys.path.insert(0, os.path.abspath('/path/to/hawkmoth'))
+
 # The suffix of source filenames.
 source_suffix = '.rst'
 
@@ -32,6 +35,9 @@ exclude_patterns = ['_build']
 # The name of the Pygments (syntax highlighting) style to use.
 pygments_style = 'sphinx'
 
+# Extensions
+extensions = ['cautodoc']
+
 # -- Options for HTML output ----------------------------------------------
 
 # The theme to use for HTML and HTML Help pages.  See the documentation for
@@ -126,6 +132,10 @@ man_pages = [
     ('man1/notmuch-tag', 'notmuch-tag',
      u'add/remove tags for all messages matching the search terms',
      [notmuch_authors], 1),
+
+    ('man3/libnotmuch', 'libnotmuch',
+     u'notmuch library API',
+     [notmuch_authors], 3),
 ]
 
 # If true, show URL addresses after external links.
@@ -156,3 +166,7 @@ texinfo_documents += [
         x[2],				# description
         'Miscellaneous'			# category
     ) for x in man_pages]
+
+# cautodoc options
+cautodoc_root = os.path.abspath('..')
+cautodoc_compat = 'javadoc-liberal'
diff --git a/doc/index.rst b/doc/index.rst
index aa6c9f40462c..bbf0c6873b76 100644
--- a/doc/index.rst
+++ b/doc/index.rst
@@ -25,6 +25,7 @@ Contents:
    man7/notmuch-search-terms
    man1/notmuch-show
    man1/notmuch-tag
+   man3/libnotmuch
 
 Indices and tables
 ==================
diff --git a/doc/man3/libnotmuch.rst b/doc/man3/libnotmuch.rst
new file mode 100644
index 000000000000..0c583ccd00b9
--- /dev/null
+++ b/doc/man3/libnotmuch.rst
@@ -0,0 +1,20 @@
+==========
+libnotmuch
+==========
+
+SYNOPSIS
+========
+
+::
+
+   #include <notmuch.h>
+
+NOTMUCH API
+===========
+
+.. c:autodoc:: lib/notmuch.h
+
+SEE ALSO
+========
+
+**notmuch(1)**
-- 
2.11.0

[RFC PATCH 3/4] lib: fix the biggest hawkmoth offenders in notmuch.h

[Jani Nikula]

The documentation was written for Doxygen, some updates for Hawkmoth.
---
 lib/notmuch.h | 94 +++++++++++++++++++++++++++++++----------------------------
 1 file changed, 50 insertions(+), 44 deletions(-)

diff --git a/lib/notmuch.h b/lib/notmuch.h
index 02586a9126e3..ab01944cb2b0 100644
--- a/lib/notmuch.h
+++ b/lib/notmuch.h
@@ -18,12 +18,10 @@
  * Author: Carl Worth <cworth@cworth.org>
  */
 
-/**
- * @defgroup notmuch The notmuch API
+/*
+ * The notmuch API
  *
  * Not much of an email library, (just index and search)
- *
- * @{
  */
 
 #ifndef NOTMUCH_H
@@ -78,22 +76,18 @@ NOTMUCH_BEGIN_DECLS
  * Check the version of the notmuch library being compiled against.
  *
  * Return true if the library being compiled against is of the
- * specified version or above. For example:
+ * specified version or above. For example::
  *
- * @code
- * #if LIBNOTMUCH_CHECK_VERSION(3, 1, 0)
- *     (code requiring libnotmuch 3.1.0 or above)
- * #endif
- * @endcode
+ *   #if LIBNOTMUCH_CHECK_VERSION(3, 1, 0)
+ *       (code requiring libnotmuch 3.1.0 or above)
+ *   #endif
  *
  * LIBNOTMUCH_CHECK_VERSION has been defined since version 3.1.0; to
- * check for versions prior to that, use:
+ * check for versions prior to that, use::
  *
- * @code
- * #if !defined(NOTMUCH_CHECK_VERSION)
- *     (code requiring libnotmuch prior to 3.1.0)
- * #endif
- * @endcode
+ *   #if !defined(NOTMUCH_CHECK_VERSION)
+ *       (code requiring libnotmuch prior to 3.1.0)
+ *   #endif
  */
 #define LIBNOTMUCH_CHECK_VERSION(major, minor, micro)			\
     (LIBNOTMUCH_MAJOR_VERSION > (major) ||					\
@@ -112,7 +106,7 @@ typedef int notmuch_bool_t;
  * A zero value (NOTMUCH_STATUS_SUCCESS) indicates that the function
  * completed without error. Any other value indicates an error.
  */
-typedef enum _notmuch_status {
+typedef enum {
     /**
      * No error occurred.
      */
@@ -130,8 +124,8 @@ typedef enum _notmuch_status {
      * A Xapian exception occurred.
      *
      * @todo We don't really want to expose this lame XAPIAN_EXCEPTION
-     * value. Instead we should map to things like DATABASE_LOCKED or
-     * whatever.
+     *       value. Instead we should map to things like
+     *       DATABASE_LOCKED or whatever.
      */
     NOTMUCH_STATUS_XAPIAN_EXCEPTION,
     /**
@@ -209,16 +203,28 @@ notmuch_status_to_string (notmuch_status_t status);
 /* Various opaque data types. For each notmuch_<foo>_t see the various
  * notmuch_<foo> functions below. */
 #ifndef __DOXYGEN__
+
+/** Notmuch database. */
 typedef struct _notmuch_database notmuch_database_t;
+/** Notmuch query. */
 typedef struct _notmuch_query notmuch_query_t;
+/** Notmuch thread list. */
 typedef struct _notmuch_threads notmuch_threads_t;
+/** Notmuch thread. */
 typedef struct _notmuch_thread notmuch_thread_t;
+/** Notmuch message list. */
 typedef struct _notmuch_messages notmuch_messages_t;
+/** Notmuch message. */
 typedef struct _notmuch_message notmuch_message_t;
+/** Notmuch tag list. */
 typedef struct _notmuch_tags notmuch_tags_t;
+/** Notmuch directory. */
 typedef struct _notmuch_directory notmuch_directory_t;
+/** Notmuch filename list. */
 typedef struct _notmuch_filenames notmuch_filenames_t;
+/** Notmuch config list. */
 typedef struct _notmuch_config_list notmuch_config_list_t;
+/** Notmuch param. */
 typedef struct _notmuch_param notmuch_param_t;
 #endif /* __DOXYGEN__ */
 
@@ -845,7 +851,7 @@ notmuch_query_add_tag_exclude (notmuch_query_t *query, const char *tag);
  * object is owned by the query and as such, will only be valid until
  * notmuch_query_destroy.
  *
- * Typical usage might be:
+ * Typical usage might be::
  *
  *     notmuch_query_t *query;
  *     notmuch_threads_t *threads;
@@ -886,7 +892,7 @@ notmuch_query_search_threads (notmuch_query_t *query,
  * Deprecated alias for notmuch_query_search_threads.
  *
  * @deprecated Deprecated as of libnotmuch 5 (notmuch 0.25). Please
- * use notmuch_query_search_threads instead.
+ *   use notmuch_query_search_threads instead.
  *
  */
 NOTMUCH_DEPRECATED(5,0)
@@ -899,7 +905,7 @@ notmuch_query_search_threads_st (notmuch_query_t *query, notmuch_threads_t **out
  * messages object is owned by the query and as such, will only be
  * valid until notmuch_query_destroy.
  *
- * Typical usage might be:
+ * Typical usage might be::
  *
  *     notmuch_query_t *query;
  *     notmuch_messages_t *messages;
@@ -941,7 +947,7 @@ notmuch_query_search_messages (notmuch_query_t *query,
  * Deprecated alias for notmuch_query_search_messages
  *
  * @deprecated Deprecated as of libnotmuch 5 (notmuch 0.25). Please use
- * notmuch_query_search_messages instead.
+ *   notmuch_query_search_messages instead.
  *
  */
 
@@ -1039,7 +1045,7 @@ notmuch_query_count_messages (notmuch_query_t *query, unsigned int *count);
  *
  *
  * @deprecated Deprecated since libnotmuch 5.0 (notmuch 0.25). Please
- * use notmuch_query_count_messages instead.
+ *   use notmuch_query_count_messages instead.
  */
 NOTMUCH_DEPRECATED(5,0)
 notmuch_status_t
@@ -1074,7 +1080,7 @@ notmuch_query_count_threads (notmuch_query_t *query, unsigned *count);
  * Deprecated alias for notmuch_query_count_threads
  *
  * @deprecated Deprecated as of libnotmuch 5.0 (notmuch 0.25). Please
- * use notmuch_query_count_threads_st instead.
+ *   use notmuch_query_count_threads_st instead.
  */
 NOTMUCH_DEPRECATED(5,0)
 notmuch_status_t
@@ -1206,7 +1212,7 @@ notmuch_thread_get_newest_date (notmuch_thread_t *thread);
  * notmuch_thread_destroy or until the query from which it derived is
  * destroyed).
  *
- * Typical usage might be:
+ * Typical usage might be::
  *
  *     notmuch_thread_t *thread;
  *     notmuch_tags_t *tags;
@@ -1414,7 +1420,7 @@ notmuch_message_reindex (notmuch_message_t *message,
 /**
  * Message flags.
  */
-typedef enum _notmuch_message_flag {
+typedef enum {
     NOTMUCH_MESSAGE_FLAG_MATCH,
     NOTMUCH_MESSAGE_FLAG_EXCLUDED,
 
@@ -1477,7 +1483,7 @@ notmuch_message_get_header (notmuch_message_t *message, const char *header);
  * valid for as long as the message is valid, (which is until the
  * query from which it derived is destroyed).
  *
- * Typical usage might be:
+ * Typical usage might be::
  *
  *     notmuch_message_t *message;
  *     notmuch_tags_t *tags;
@@ -1563,13 +1569,15 @@ notmuch_message_remove_all_tags (notmuch_message_t *message);
  * flags, and adds or removes tags on 'message' as follows when these
  * flags are present:
  *
- *	Flag	Action if present
- *	----	-----------------
- *	'D'	Adds the "draft" tag to the message
- *	'F'	Adds the "flagged" tag to the message
- *	'P'	Adds the "passed" tag to the message
- *	'R'	Adds the "replied" tag to the message
- *	'S'	Removes the "unread" tag from the message
+ * ====  =========================================
+ * Flag	 Action if present
+ * ====  =========================================
+ * D     Adds the "draft" tag to the message
+ * F     Adds the "flagged" tag to the message
+ * P     Adds the "passed" tag to the message
+ * R     Adds the "replied" tag to the message
+ * S     Removes the "unread" tag from the message
+ * ====  =========================================
  *
  * For each flag that is not present, the opposite action (add/remove)
  * is performed for the corresponding tags.
@@ -1642,7 +1650,7 @@ notmuch_message_tags_to_maildir_flags (notmuch_message_t *message);
  *
  * The ability to do freeze/thaw allows for safe transactions to
  * change tag values. For example, explicitly setting a message to
- * have a given set of tags might look like this:
+ * have a given set of tags might look like this::
  *
  *    notmuch_message_freeze (message);
  *
@@ -1709,7 +1717,7 @@ void
 notmuch_message_destroy (notmuch_message_t *message);
 
 /**
- * @name Message Properties
+ * Message Properties
  *
  * This interface provides the ability to attach arbitrary (key,value)
  * string pairs to a message, to remove such pairs, and to iterate
@@ -1718,7 +1726,7 @@ notmuch_message_destroy (notmuch_message_t *message);
  * depend on these properties.
  *
  */
-/**@{*/
+
 /**
  * Retrieve the value for a single property key
  *
@@ -1795,7 +1803,7 @@ typedef struct _notmuch_string_map_iterator notmuch_message_properties_t;
  * @param[in] exact    if TRUE, require exact match with key. Otherwise
  *		       treat as prefix.
  *
- * Typical usage might be:
+ * Typical usage might be::
  *
  *     notmuch_message_properties_t *list;
  *
@@ -1883,8 +1891,6 @@ notmuch_message_properties_value (notmuch_message_properties_t *properties);
 void
 notmuch_message_properties_destroy (notmuch_message_properties_t *properties);
 
-/**@}*/
-
 /**
  * Is the given 'tags' iterator pointing at a valid tag.
  *
@@ -1944,11 +1950,11 @@ notmuch_tags_destroy (notmuch_tags_t *tags);
  * identification of new messages to be added to the database. The
  * recommended usage is as follows:
  *
- *   o Read the mtime of a directory from the filesystem
+ *   - Read the mtime of a directory from the filesystem
  *
- *   o Call add_message for all mail files in the directory
+ *   - Call add_message for all mail files in the directory
  *
- *   o Call notmuch_directory_set_mtime with the mtime read from the
+ *   - Call notmuch_directory_set_mtime with the mtime read from the
  *     filesystem.
  *
  * Then, when wanting to check for updates to the directory in the
-- 
2.11.0

[RFC PATCH 4/4] lib: add reStructuredText bling for API documentation in notmuch.h

[Jani Nikula]

Document notmuch_database_create() using as much bling as
possible. This is pretty nice in the output, but probably too
excessive in source code. Nonetheless, it's an example of what could
be done.
---
 lib/notmuch.h | 45 +++++++++++++++++++++++++--------------------
 1 file changed, 25 insertions(+), 20 deletions(-)

diff --git a/lib/notmuch.h b/lib/notmuch.h
index ab01944cb2b0..c3307d5835dd 100644
--- a/lib/notmuch.h
+++ b/lib/notmuch.h
@@ -229,38 +229,43 @@ typedef struct _notmuch_param notmuch_param_t;
 #endif /* __DOXYGEN__ */
 
 /**
- * Create a new, empty notmuch database located at 'path'.
+ * Create a new, empty notmuch database located at ``path``.
  *
- * The path should be a top-level directory to a collection of
- * plain-text email messages (one message per file). This call will
- * create a new ".notmuch" directory within 'path' where notmuch will
- * store its data.
+ * This call will create a new ``.notmuch`` directory within ``path``
+ * where notmuch will store its data.
  *
- * After a successful call to notmuch_database_create, the returned
- * database will be open so the caller should call
- * notmuch_database_destroy when finished with it.
+ * After a successful call to :c:func:`notmuch_database_create()`, the
+ * returned database will be open so the caller should call
+ * :c:func:`notmuch_database_destroy()` when finished with it.
  *
  * The database will not yet have any data in it
- * (notmuch_database_create itself is a very cheap function). Messages
- * contained within 'path' can be added to the database by calling
- * notmuch_database_add_message.
+ * (:c:func:`notmuch_database_create()` itself is a very cheap
+ * function). Messages contained within ``path`` can be added to the
+ * database by calling :c:func:`notmuch_database_add_message()`.
  *
  * In case of any failure, this function returns an error status and
- * sets *database to NULL (after printing an error message on stderr).
+ * sets ``*database`` to ``NULL`` (after printing an error message on
+ * ``stderr``).
  *
- * Return value:
+ * @param path The ``path`` should be a top-level directory to a
+ *   collection of plain-text email messages (one message per file).
  *
- * NOTMUCH_STATUS_SUCCESS: Successfully created the database.
+ * @param database Pointer to database.
  *
- * NOTMUCH_STATUS_NULL_POINTER: The given 'path' argument is NULL.
+ * @return 
+ *   :c:macro:`NOTMUCH_STATUS_SUCCESS`: Successfully created the database.
  *
- * NOTMUCH_STATUS_OUT_OF_MEMORY: Out of memory.
+ *   :c:macro:`NOTMUCH_STATUS_NULL_POINTER`: The given ``path``
+ *   argument is ``NULL``.
  *
- * NOTMUCH_STATUS_FILE_ERROR: An error occurred trying to create the
- *	database file (such as permission denied, or file not found,
- *	etc.), or the database already exists.
+ *   :c:macro:`NOTMUCH_STATUS_OUT_OF_MEMORY`: Out of memory.
  *
- * NOTMUCH_STATUS_XAPIAN_EXCEPTION: A Xapian exception occurred.
+ *   :c:macro:`NOTMUCH_STATUS_FILE_ERROR`: An error occurred trying to
+ *   create the database file (such as permission denied, or file not
+ *   found, etc.), or the database already exists.
+ *
+ *   :c:macro:`NOTMUCH_STATUS_XAPIAN_EXCEPTION`: A Xapian exception
+ *   occurred.
  */
 notmuch_status_t
 notmuch_database_create (const char *path, notmuch_database_t **database);
-- 
2.11.0