doc build warnings when building out-of-tree

doc build warnings when building out-of-tree

[Daniel Kahn Gillmor]

[-- Attachment #1.1: Type: text/plain, Size: 7367 bytes --]

When building out-of-tree, the documentation (both manpages and info
files) are incomplete, but they do not explicitly fail.  build logs
follow from doing "mkdir build && ../configure && make".

If there's a way to make these warnings into hard failures, i think that
would be good -- we don't want to accidentally ship the output they
create.  And of course actually fixing it would be even better.

I'm afraid i'm not proficient enough in sphinx to know how to do either,
so i'm just reporting a bug for now.

    --dkg

WITH_EMACS=1 sphinx-build -b man -d doc/_build/man_doctrees -q ../doc doc/_build/man
/home/dkg/src/notmuch/notmuch/doc/notmuch-emacs.rst:383: WARNING: Problems with "include" directive path:
InputError: [Errno 2] No such file or directory: '../emacs/notmuch.rsti'.
/home/dkg/src/notmuch/notmuch/doc/notmuch-emacs.rst:385: WARNING: Problems with "include" directive path:
InputError: [Errno 2] No such file or directory: '../emacs/notmuch-lib.rsti'.
/home/dkg/src/notmuch/notmuch/doc/notmuch-emacs.rst:387: WARNING: Problems with "include" directive path:
InputError: [Errno 2] No such file or directory: '../emacs/notmuch-show.rsti'.
/home/dkg/src/notmuch/notmuch/doc/notmuch-emacs.rst:389: WARNING: Problems with "include" directive path:
InputError: [Errno 2] No such file or directory: '../emacs/notmuch-tag.rsti'.
/home/dkg/src/notmuch/notmuch/doc/notmuch-emacs.rst:220: WARNING: Undefined substitution referenced: "docstring::notmuch-message-headers".
/home/dkg/src/notmuch/notmuch/doc/notmuch-emacs.rst:223: WARNING: Undefined substitution referenced: "docstring::notmuch-message-headers-visible".
/home/dkg/src/notmuch/notmuch/doc/notmuch-emacs.rst:236: WARNING: Undefined substitution referenced: "docstring::notmuch-show-stash-filename".
/home/dkg/src/notmuch/notmuch/doc/notmuch-emacs.rst:239: WARNING: Undefined substitution referenced: "docstring::notmuch-show-stash-git-send-email".
/home/dkg/src/notmuch/notmuch/doc/notmuch-emacs.rst:242: WARNING: Undefined substitution referenced: "docstring::notmuch-show-stash-message-id-stripped".
/home/dkg/src/notmuch/notmuch/doc/notmuch-emacs.rst:245: WARNING: Undefined substitution referenced: "docstring::notmuch-show-stash-mlarchive-link-and-go".
/home/dkg/src/notmuch/notmuch/doc/notmuch-emacs.rst:248: WARNING: Undefined substitution referenced: "docstring::notmuch-show-stash-tags".
/home/dkg/src/notmuch/notmuch/doc/notmuch-emacs.rst:251: WARNING: Undefined substitution referenced: "docstring::notmuch-show-stash-cc".
/home/dkg/src/notmuch/notmuch/doc/notmuch-emacs.rst:254: WARNING: Undefined substitution referenced: "docstring::notmuch-show-stash-date".
/home/dkg/src/notmuch/notmuch/doc/notmuch-emacs.rst:257: WARNING: Undefined substitution referenced: "docstring::notmuch-show-stash-from".
/home/dkg/src/notmuch/notmuch/doc/notmuch-emacs.rst:260: WARNING: Undefined substitution referenced: "docstring::notmuch-show-stash-message-id".
/home/dkg/src/notmuch/notmuch/doc/notmuch-emacs.rst:263: WARNING: Undefined substitution referenced: "docstring::notmuch-show-stash-mlarchive-link".
/home/dkg/src/notmuch/notmuch/doc/notmuch-emacs.rst:266: WARNING: Undefined substitution referenced: "docstring::notmuch-show-stash-subject".
/home/dkg/src/notmuch/notmuch/doc/notmuch-emacs.rst:269: WARNING: Undefined substitution referenced: "docstring::notmuch-show-stash-to".
/home/dkg/src/notmuch/notmuch/doc/notmuch-emacs.rst:347: WARNING: Undefined substitution referenced: "docstring::notmuch-tagging-keys".
/home/dkg/src/notmuch/notmuch/doc/notmuch-emacs.rst:353: WARNING: Undefined substitution referenced: "docstring::notmuch-cycle-notmuch-buffers".
/home/dkg/src/notmuch/notmuch/doc/notmuch-emacs.rst:364: WARNING: Undefined substitution referenced: "docstring::notmuch-poll".
/home/dkg/src/notmuch/notmuch/doc/notmuch-emacs.rst:367: WARNING: Undefined substitution referenced: "docstring::notmuch-poll-script".
[…]


WITH_EMACS=1 sphinx-build -b texinfo -d doc/_build/texinfo_doctrees -q ../doc doc/_build/texinfo
/home/dkg/src/notmuch/notmuch/doc/notmuch-emacs.rst:383: WARNING: Problems with "include" directive path:
InputError: [Errno 2] No such file or directory: '../emacs/notmuch.rsti'.
/home/dkg/src/notmuch/notmuch/doc/notmuch-emacs.rst:385: WARNING: Problems with "include" directive path:
InputError: [Errno 2] No such file or directory: '../emacs/notmuch-lib.rsti'.
/home/dkg/src/notmuch/notmuch/doc/notmuch-emacs.rst:387: WARNING: Problems with "include" directive path:
InputError: [Errno 2] No such file or directory: '../emacs/notmuch-show.rsti'.
/home/dkg/src/notmuch/notmuch/doc/notmuch-emacs.rst:389: WARNING: Problems with "include" directive path:
InputError: [Errno 2] No such file or directory: '../emacs/notmuch-tag.rsti'.
/home/dkg/src/notmuch/notmuch/doc/notmuch-emacs.rst:220: WARNING: Undefined substitution referenced: "docstring::notmuch-message-headers".
/home/dkg/src/notmuch/notmuch/doc/notmuch-emacs.rst:223: WARNING: Undefined substitution referenced: "docstring::notmuch-message-headers-visible".
/home/dkg/src/notmuch/notmuch/doc/notmuch-emacs.rst:236: WARNING: Undefined substitution referenced: "docstring::notmuch-show-stash-filename".
/home/dkg/src/notmuch/notmuch/doc/notmuch-emacs.rst:239: WARNING: Undefined substitution referenced: "docstring::notmuch-show-stash-git-send-email".
/home/dkg/src/notmuch/notmuch/doc/notmuch-emacs.rst:242: WARNING: Undefined substitution referenced: "docstring::notmuch-show-stash-message-id-stripped".
/home/dkg/src/notmuch/notmuch/doc/notmuch-emacs.rst:245: WARNING: Undefined substitution referenced: "docstring::notmuch-show-stash-mlarchive-link-and-go".
/home/dkg/src/notmuch/notmuch/doc/notmuch-emacs.rst:248: WARNING: Undefined substitution referenced: "docstring::notmuch-show-stash-tags".
/home/dkg/src/notmuch/notmuch/doc/notmuch-emacs.rst:251: WARNING: Undefined substitution referenced: "docstring::notmuch-show-stash-cc".
/home/dkg/src/notmuch/notmuch/doc/notmuch-emacs.rst:254: WARNING: Undefined substitution referenced: "docstring::notmuch-show-stash-date".
/home/dkg/src/notmuch/notmuch/doc/notmuch-emacs.rst:257: WARNING: Undefined substitution referenced: "docstring::notmuch-show-stash-from".
/home/dkg/src/notmuch/notmuch/doc/notmuch-emacs.rst:260: WARNING: Undefined substitution referenced: "docstring::notmuch-show-stash-message-id".
/home/dkg/src/notmuch/notmuch/doc/notmuch-emacs.rst:263: WARNING: Undefined substitution referenced: "docstring::notmuch-show-stash-mlarchive-link".
/home/dkg/src/notmuch/notmuch/doc/notmuch-emacs.rst:266: WARNING: Undefined substitution referenced: "docstring::notmuch-show-stash-subject".
/home/dkg/src/notmuch/notmuch/doc/notmuch-emacs.rst:269: WARNING: Undefined substitution referenced: "docstring::notmuch-show-stash-to".
/home/dkg/src/notmuch/notmuch/doc/notmuch-emacs.rst:347: WARNING: Undefined substitution referenced: "docstring::notmuch-tagging-keys".
/home/dkg/src/notmuch/notmuch/doc/notmuch-emacs.rst:353: WARNING: Undefined substitution referenced: "docstring::notmuch-cycle-notmuch-buffers".
/home/dkg/src/notmuch/notmuch/doc/notmuch-emacs.rst:364: WARNING: Undefined substitution referenced: "docstring::notmuch-poll".
/home/dkg/src/notmuch/notmuch/doc/notmuch-emacs.rst:367: WARNING: Undefined substitution referenced: "docstring::notmuch-poll-script".

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: doc build warnings when building out-of-tree

[David Bremner]

Daniel Kahn Gillmor <dkg@fifthhorseman.net> writes:

> When building out-of-tree, the documentation (both manpages and info
> files) are incomplete, but they do not explicitly fail.  build logs
> follow from doing "mkdir build && ../configure && make".
>
> If there's a way to make these warnings into hard failures, i think that
> would be good -- we don't want to accidentally ship the output they
> create.  And of course actually fixing it would be even better.
>
> I'm afraid i'm not proficient enough in sphinx to know how to do either,
> so i'm just reporting a bug for now.
>

I can't duplicate these errors with sphinx-doc 2.4.3-3 on Debian.

Are the rsti files actually being built before calling sphinx-build?

d

Re: doc build warnings when building out-of-tree

[David Bremner]

David Bremner <david@tethera.net> writes:

> Daniel Kahn Gillmor <dkg@fifthhorseman.net> writes:
>
> I can't duplicate these errors with sphinx-doc 2.4.3-3 on Debian.
>
> Are the rsti files actually being built before calling sphinx-build?
>
> d

OK, I understand now. The case I thought was working was actually using
the rsti files from a previous in-tree build.

I think the quick and dirty fix is to add '-W' to the sphinx-build
line. I'll see if I can find a better fix. It looks like the issue is
using relative paths in the rst. In the out of tree build those are
still resolving relative to the source file (reasonably enough). Maybe
we can use the environment variable NOTMUCH_BUILDDIR in the config
file.

d

[PATCH] doc: fix for out-of-tree builds of notmuch-emacs docs

[David Bremner]

The sphinx-doc include directive does not have the ability to include
files from the build tree, so we replace the include with reading the
files in conf.py. The non-trivial downside of this is that the emacs
docstrings are now defined for every rst source file. They are
namespaced with docstring::, so hopefully there will not be any
surprises. One thing that is noticable is a small (absolute) time
penalty in running sphinx-doc.
---
 doc/Makefile.local    |  2 +-
 doc/conf.py           | 19 +++++++++++++++----
 doc/notmuch-emacs.rst | 10 ----------
 3 files changed, 16 insertions(+), 15 deletions(-)

diff --git a/doc/Makefile.local b/doc/Makefile.local
index b4e0c955..769438ed 100644
--- a/doc/Makefile.local
+++ b/doc/Makefile.local
@@ -4,7 +4,7 @@ dir := doc
 
 # You can set these variables from the command line.
 SPHINXOPTS    := -q
-SPHINXBUILD   = WITH_EMACS=${WITH_EMACS} sphinx-build
+SPHINXBUILD   = WITH_EMACS=${WITH_EMACS} RSTI_DIR=$(realpath emacs) sphinx-build
 DOCBUILDDIR      := $(dir)/_build
 
 # Internal variables.
diff --git a/doc/conf.py b/doc/conf.py
index fc9738ff..d8841d99 100644
--- a/doc/conf.py
+++ b/doc/conf.py
@@ -29,10 +29,21 @@ release = version
 # directories to ignore when looking for source files.
 exclude_patterns = ['_build']
 
-# If we don't have emacs (or the user configured --without-emacs),
-# don't build the notmuch-emacs docs, as they need emacs to generate
-# the docstring include files
-if os.environ.get('WITH_EMACS') != '1':
+if os.environ.get('WITH_EMACS') == '1':
+    # Hacky reimplementation of include to workaround limitations of
+    # sphinx-doc
+    accumulator='.. include:: /../emacs/rstdoc.rsti\n\n' # in the source tree
+    rsti_dir=os.environ.get('RSTI_DIR')
+    # the other files are from the build tree
+    for file in ['notmuch.rsti', 'notmuch-lib.rsti', 'notmuch-show.rsti', 'notmuch-tag.rsti']:
+        with open(rsti_dir+'/'+file) as f:
+            for line in f:
+                accumulator += line
+    rst_epilog=accumulator
+else:
+    # If we don't have emacs (or the user configured --without-emacs),
+    # don't build the notmuch-emacs docs, as they need emacs to generate
+    # the docstring include files
     exclude_patterns.append('notmuch-emacs.rst')
 
 # The name of the Pygments (syntax highlighting) style to use.
diff --git a/doc/notmuch-emacs.rst b/doc/notmuch-emacs.rst
index 1655e2f0..de47b726 100644
--- a/doc/notmuch-emacs.rst
+++ b/doc/notmuch-emacs.rst
@@ -377,13 +377,3 @@ suffix exist it will be read instead (just one of these, chosen in this
 order). Most often users create ``~/.emacs.d/notmuch-config.el`` and just
 work with it. If Emacs was invoked with the ``-q`` or ``--no-init-file``
 options, ``notmuch-init-file`` is not read.
-
-.. include:: ../emacs/rstdoc.rsti
-
-.. include:: ../emacs/notmuch.rsti
-
-.. include:: ../emacs/notmuch-lib.rsti
-
-.. include:: ../emacs/notmuch-show.rsti
-
-.. include:: ../emacs/notmuch-tag.rsti
-- 
2.26.2

Re: [PATCH] doc: fix for out-of-tree builds of notmuch-emacs docs

[Tomi Ollila]

On Sun, May 31 2020, David Bremner wrote:

> The sphinx-doc include directive does not have the ability to include
> files from the build tree, so we replace the include with reading the
> files in conf.py. The non-trivial downside of this is that the emacs
> docstrings are now defined for every rst source file. They are
> namespaced with docstring::, so hopefully there will not be any
> surprises. One thing that is noticable is a small (absolute) time
> penalty in running sphinx-doc.

IMO good enough tolerable hack to get this done for no (read: I don't
know alternatives).

Some comments below...

> ---
>  doc/Makefile.local    |  2 +-
>  doc/conf.py           | 19 +++++++++++++++----
>  doc/notmuch-emacs.rst | 10 ----------
>  3 files changed, 16 insertions(+), 15 deletions(-)
>
> diff --git a/doc/Makefile.local b/doc/Makefile.local
> index b4e0c955..769438ed 100644
> --- a/doc/Makefile.local
> +++ b/doc/Makefile.local
> @@ -4,7 +4,7 @@ dir := doc
>  
>  # You can set these variables from the command line.
>  SPHINXOPTS    := -q
> -SPHINXBUILD   = WITH_EMACS=${WITH_EMACS} sphinx-build
> +SPHINXBUILD   = WITH_EMACS=${WITH_EMACS} RSTI_DIR=$(realpath emacs) sphinx-build

I was about to comment the above, before realized $(realpath ...) executes
GNU Make function instead of system command :) ...

>  DOCBUILDDIR      := $(dir)/_build
>  
>  # Internal variables.
> diff --git a/doc/conf.py b/doc/conf.py
> index fc9738ff..d8841d99 100644
> --- a/doc/conf.py
> +++ b/doc/conf.py
> @@ -29,10 +29,21 @@ release = version
>  # directories to ignore when looking for source files.
>  exclude_patterns = ['_build']
>  
> -# If we don't have emacs (or the user configured --without-emacs),
> -# don't build the notmuch-emacs docs, as they need emacs to generate
> -# the docstring include files
> -if os.environ.get('WITH_EMACS') != '1':
> +if os.environ.get('WITH_EMACS') == '1':
> +    # Hacky reimplementation of include to workaround limitations of
> +    # sphinx-doc
> +    accumulator='.. include:: /../emacs/rstdoc.rsti\n\n' # in the source tree

accumulator=['.. include:: /../emacs/rstdoc.rsti\n\n' # in the source tree]

(in list, see below)

> +    rsti_dir=os.environ.get('RSTI_DIR')

spaces around '='

> +    # the other files are from the build tree
> +    for file in ['notmuch.rsti', 'notmuch-lib.rsti', 'notmuch-show.rsti', 'notmuch-tag.rsti']:

tuple ('notmuch.rsti', 'notmuch-lib.rsti', ...) could have been used

> +        with open(rsti_dir+'/'+file) as f:
> +            for line in f:
> +                accumulator += line
> +    rst_epilog=accumulator

           accumulator.append(open(rsti_dir + '/' + file).read())
       accumulator = ''.join(accumulator)
       rst_epilog = accumulator

alternative last 2 lines

       rst_epilog = ''.join(accumulator)
       del accumulator

(I personally would have left 'accumulator' away and used 'rst_epilog'
directly (and perhaps commented its use as temporary list type).)


> +else:
> +    # If we don't have emacs (or the user configured --without-emacs),
> +    # don't build the notmuch-emacs docs, as they need emacs to generate
> +    # the docstring include files
>      exclude_patterns.append('notmuch-emacs.rst')
>  
>  # The name of the Pygments (syntax highlighting) style to use.

Tomi

Re: [PATCH] doc: fix for out-of-tree builds of notmuch-emacs docs

[David Bremner]

Tomi Ollila <tomi.ollila@iki.fi> writes:

> On Sun, May 31 2020, David Bremner wrote:
>
>> The sphinx-doc include directive does not have the ability to include
>> files from the build tree, so we replace the include with reading the
>> files in conf.py. The non-trivial downside of this is that the emacs
>> docstrings are now defined for every rst source file. They are
>> namespaced with docstring::, so hopefully there will not be any
>> surprises. One thing that is noticable is a small (absolute) time
>> penalty in running sphinx-doc.
>
> IMO good enough tolerable hack to get this done for no (read: I don't
> know alternatives).
>
> Some comments below...

Pushed a version with most of Tomi's comments applied (and
s/append/extend/)

d