[PATCH] doc: use separate doctrees for distinct builders

[PATCH] doc: use separate doctrees for distinct builders

[David Bremner]

It seems our previous attempt with order-only targets was not
sufficient to avoid problems with sphinx-builds doctree cache [0].
Looking around at other people's approaches [1], using seperate
doctrees was suggested. I guess there might be a slight loss of
efficiency, but it seems more robust.

[0]: build failures were first noticed in debian experimental, but I was able to duplicate it in
     my usual build environment about 1 in 8 builds.

[1]: in particular
     https://salsa.debian.org/mpd-team/mpc/commit/9e3fc1657d043d75755993846c93f7700b97f907
---
 doc/Makefile.local | 16 ++++------------
 1 file changed, 4 insertions(+), 12 deletions(-)

diff --git a/doc/Makefile.local b/doc/Makefile.local
index 651168f4..719172fe 100644
--- a/doc/Makefile.local
+++ b/doc/Makefile.local
@@ -8,7 +8,7 @@ SPHINXBUILD   = HAVE_EMACS=${HAVE_EMACS} sphinx-build
 DOCBUILDDIR      := $(dir)/_build
 
 # Internal variables.
-ALLSPHINXOPTS   := -d $(DOCBUILDDIR)/doctrees $(SPHINXOPTS) $(srcdir)/$(dir)
+ALLSPHINXOPTS   := $(SPHINXOPTS) $(srcdir)/$(dir)
 APIMAN		:= $(DOCBUILDDIR)/man/man3/notmuch.3
 DOXYFILE	:= $(srcdir)/$(dir)/doxygen.cfg
 
@@ -42,24 +42,16 @@ ifeq ($(WITH_EMACS),1)
 $(DOCBUILDDIR)/.roff.stamp sphinx-html sphinx-texinfo: docstring.stamp
 endif
 
-# Sequentialize the calls to sphinx-build to avoid races with
-# reading/writing cached state. This uses GNU make specific
-# "order-only" prerequisites.
-
-sphinx-html: | $(DOCBUILDDIR)/.roff.stamp
-sphinx-texinfo: | sphinx-html
-sphinx-info: | sphinx-texinfo
-
 sphinx-html: $(DOCBUILDDIR)/.html.stamp
 
 $(DOCBUILDDIR)/.html.stamp: $(ALL_RST_FILES)
-	$(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(DOCBUILDDIR)/html
+	$(SPHINXBUILD) -b html -d $(DOCBUILDDIR)/html_doctrees $(ALLSPHINXOPTS) $(DOCBUILDDIR)/html
 	touch $@
 
 sphinx-texinfo: $(DOCBUILDDIR)/.texi.stamp
 
 $(DOCBUILDDIR)/.texi.stamp: $(ALL_RST_FILES)
-	$(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(DOCBUILDDIR)/texinfo
+	$(SPHINXBUILD) -b texinfo -d $(DOCBUILDDIR)/texinfo_doctrees $(ALLSPHINXOPTS) $(DOCBUILDDIR)/texinfo
 	touch $@
 
 sphinx-info: sphinx-texinfo
@@ -75,7 +67,7 @@ ${MAN_ROFF_FILES}: $(DOCBUILDDIR)/.roff.stamp
 # instance of this recipe for each roff file.
 $(DOCBUILDDIR)/.roff.stamp: ${MAN_RST_FILES}
 ifeq ($(HAVE_SPHINX),1)
-	$(SPHINXBUILD) -b man $(ALLSPHINXOPTS) $(DOCBUILDDIR)/man
+	$(SPHINXBUILD) -b man -d $(DOCBUILDDIR)/man_doctrees $(ALLSPHINXOPTS) $(DOCBUILDDIR)/man
 	for section in 1 5 7; do \
 	    mkdir -p $(DOCBUILDDIR)/man/man$${section}; \
 	    mv $(DOCBUILDDIR)/man/*.$${section} $(DOCBUILDDIR)/man/man$${section}; \
-- 
2.20.1

Re: [PATCH] doc: use separate doctrees for distinct builders

[Tomi Ollila]

On Fri, May 31 2019, David Bremner wrote:

> It seems our previous attempt with order-only targets was not
> sufficient to avoid problems with sphinx-builds doctree cache [0].
> Looking around at other people's approaches [1], using seperate

probably good, but you may want to s/seperate/separate/

Tomi

> doctrees was suggested. I guess there might be a slight loss of
> efficiency, but it seems more robust.
>
> [0]: build failures were first noticed in debian experimental, but I was able to duplicate it in
>      my usual build environment about 1 in 8 builds.
>
> [1]: in particular
>      https://salsa.debian.org/mpd-team/mpc/commit/9e3fc1657d043d75755993846c93f7700b97f907
> ---
>  doc/Makefile.local | 16 ++++------------
>  1 file changed, 4 insertions(+), 12 deletions(-)
>
> diff --git a/doc/Makefile.local b/doc/Makefile.local
> index 651168f4..719172fe 100644
> --- a/doc/Makefile.local
> +++ b/doc/Makefile.local
> @@ -8,7 +8,7 @@ SPHINXBUILD   = HAVE_EMACS=${HAVE_EMACS} sphinx-build
>  DOCBUILDDIR      := $(dir)/_build
>  
>  # Internal variables.
> -ALLSPHINXOPTS   := -d $(DOCBUILDDIR)/doctrees $(SPHINXOPTS) $(srcdir)/$(dir)
> +ALLSPHINXOPTS   := $(SPHINXOPTS) $(srcdir)/$(dir)
>  APIMAN		:= $(DOCBUILDDIR)/man/man3/notmuch.3
>  DOXYFILE	:= $(srcdir)/$(dir)/doxygen.cfg
>  
> @@ -42,24 +42,16 @@ ifeq ($(WITH_EMACS),1)
>  $(DOCBUILDDIR)/.roff.stamp sphinx-html sphinx-texinfo: docstring.stamp
>  endif
>  
> -# Sequentialize the calls to sphinx-build to avoid races with
> -# reading/writing cached state. This uses GNU make specific
> -# "order-only" prerequisites.
> -
> -sphinx-html: | $(DOCBUILDDIR)/.roff.stamp
> -sphinx-texinfo: | sphinx-html
> -sphinx-info: | sphinx-texinfo
> -
>  sphinx-html: $(DOCBUILDDIR)/.html.stamp
>  
>  $(DOCBUILDDIR)/.html.stamp: $(ALL_RST_FILES)
> -	$(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(DOCBUILDDIR)/html
> +	$(SPHINXBUILD) -b html -d $(DOCBUILDDIR)/html_doctrees $(ALLSPHINXOPTS) $(DOCBUILDDIR)/html
>  	touch $@
>  
>  sphinx-texinfo: $(DOCBUILDDIR)/.texi.stamp
>  
>  $(DOCBUILDDIR)/.texi.stamp: $(ALL_RST_FILES)
> -	$(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(DOCBUILDDIR)/texinfo
> +	$(SPHINXBUILD) -b texinfo -d $(DOCBUILDDIR)/texinfo_doctrees $(ALLSPHINXOPTS) $(DOCBUILDDIR)/texinfo
>  	touch $@
>  
>  sphinx-info: sphinx-texinfo
> @@ -75,7 +67,7 @@ ${MAN_ROFF_FILES}: $(DOCBUILDDIR)/.roff.stamp
>  # instance of this recipe for each roff file.
>  $(DOCBUILDDIR)/.roff.stamp: ${MAN_RST_FILES}
>  ifeq ($(HAVE_SPHINX),1)
> -	$(SPHINXBUILD) -b man $(ALLSPHINXOPTS) $(DOCBUILDDIR)/man
> +	$(SPHINXBUILD) -b man -d $(DOCBUILDDIR)/man_doctrees $(ALLSPHINXOPTS) $(DOCBUILDDIR)/man
>  	for section in 1 5 7; do \
>  	    mkdir -p $(DOCBUILDDIR)/man/man$${section}; \
>  	    mv $(DOCBUILDDIR)/man/*.$${section} $(DOCBUILDDIR)/man/man$${section}; \
> -- 
> 2.20.1
>
> _______________________________________________
> notmuch mailing list
> notmuch@notmuchmail.org
> https://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH] doc: use separate doctrees for distinct builders

[David Bremner]

David Bremner <david@tethera.net> writes:

> It seems our previous attempt with order-only targets was not
> sufficient to avoid problems with sphinx-builds doctree cache [0].
> Looking around at other people's approaches [1], using seperate
> doctrees was suggested. I guess there might be a slight loss of
> efficiency, but it seems more robust.

pushed to release (soon to master), with Tomi's spellink fichs