[PATCH] build/docs: move docstring prereq to file targets

[PATCH] build/docs: move docstring prereq to file targets

[David Bremner]

Under a sufficiently high level of parallelism [1] there seems to be a
a race condition that allows sphinx-build to start running before the
docstrings are extracted. This change moves the docstring stamp from
the phony targets sphinx-html and sphinx-info to the file targets that
they depend on. I'm not sure why this makes things better, but I am
fairly confident it does not make things worse, and experimentally it
seems to eliminate the race condition.

[1]: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=976934
---
 doc/Makefile.local | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/doc/Makefile.local b/doc/Makefile.local
index 60bd7184..f476d1da 100644
--- a/doc/Makefile.local
+++ b/doc/Makefile.local
@@ -43,7 +43,7 @@ INFO_INFO_FILES := $(INFO_TEXI_FILES:.texi=.info)
 	rm -f $@ && gzip --no-name --stdout $^ > $@
 
 ifeq ($(WITH_EMACS),1)
-$(DOCBUILDDIR)/.roff.stamp sphinx-html sphinx-texinfo: docstring.stamp
+$(DOCBUILDDIR)/.roff.stamp $(DOCBUILDDIR)/.html.stamp $(DOCBUILDDIR)/.texi.stamp : docstring.stamp
 endif
 
 sphinx-html: $(DOCBUILDDIR)/.html.stamp
-- 
2.29.2

Re: [PATCH] build/docs: move docstring prereq to file targets

[Tomi Ollila]

On Wed, Dec 09 2020, David Bremner wrote:

> Under a sufficiently high level of parallelism [1] there seems to be a
> a race condition that allows sphinx-build to start running before the
> docstrings are extracted. This change moves the docstring stamp from
> the phony targets sphinx-html and sphinx-info to the file targets that
> they depend on. I'm not sure why this makes things better, but I am
> fairly confident it does not make things worse, and experimentally it
> seems to eliminate the race condition.

Good enough for me if this helps. I also don't see reason why that would
make things better (and probably not things worse), just that I got
headache reading that Makefile ;) (and, for example, these particular
targets mentioned would not need to be marked .PHONY...)

I'd suggest to monitor the behaviour for a while and if things are
consistently better then merge -- such a things when we don't know
enough experience may be enough (or someone(tm) could try to parse make
debug logs ;/) -- or someone(tm) may tell us why that heleps :D

Tomi

>
> [1]: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=976934
> ---
>  doc/Makefile.local | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/doc/Makefile.local b/doc/Makefile.local
> index 60bd7184..f476d1da 100644
> --- a/doc/Makefile.local
> +++ b/doc/Makefile.local
> @@ -43,7 +43,7 @@ INFO_INFO_FILES := $(INFO_TEXI_FILES:.texi=.info)
>  	rm -f $@ && gzip --no-name --stdout $^ > $@
>  
>  ifeq ($(WITH_EMACS),1)
> -$(DOCBUILDDIR)/.roff.stamp sphinx-html sphinx-texinfo: docstring.stamp
> +$(DOCBUILDDIR)/.roff.stamp $(DOCBUILDDIR)/.html.stamp $(DOCBUILDDIR)/.texi.stamp : docstring.stamp
>  endif
>  
>  sphinx-html: $(DOCBUILDDIR)/.html.stamp
> -- 
> 2.29.2
> _______________________________________________
> notmuch mailing list -- notmuch@notmuchmail.org
> To unsubscribe send an email to notmuch-leave@notmuchmail.org

Re: [PATCH] build/docs: move docstring prereq to file targets

[David Bremner]

Tomi Ollila <tomi.ollila@iki.fi> writes:

> On Wed, Dec 09 2020, David Bremner wrote:
>
>> Under a sufficiently high level of parallelism [1] there seems to be a
>> a race condition that allows sphinx-build to start running before the
>> docstrings are extracted. This change moves the docstring stamp from
>> the phony targets sphinx-html and sphinx-info to the file targets that
>> they depend on. I'm not sure why this makes things better, but I am
>> fairly confident it does not make things worse, and experimentally it
>> seems to eliminate the race condition.
>
> Good enough for me if this helps. I also don't see reason why that would
> make things better (and probably not things worse), just that I got
> headache reading that Makefile ;) (and, for example, these particular
> targets mentioned would not need to be marked .PHONY...)
>
> I'd suggest to monitor the behaviour for a while and if things are
> consistently better then merge -- such a things when we don't know
> enough experience may be enough (or someone(tm) could try to parse make
> debug logs ;/) -- or someone(tm) may tell us why that heleps :D

I'll try a test upload to Debian and see if that fixes the problem
there. I did 2000 builds (sortof by mistake) at -j32 on a 16 thread / 8
core machine, with no crashes. Much less parallelism than the original
report, but lots of repetition.

d

Re: [PATCH] build/docs: move docstring prereq to file targets

[David Bremner]

Tomi Ollila <tomi.ollila@iki.fi> writes:

> On Wed, Dec 09 2020, David Bremner wrote:
>
>> Under a sufficiently high level of parallelism [1] there seems to be a
>> a race condition that allows sphinx-build to start running before the
>> docstrings are extracted. This change moves the docstring stamp from
>> the phony targets sphinx-html and sphinx-info to the file targets that
>> they depend on. I'm not sure why this makes things better, but I am
>> fairly confident it does not make things worse, and experimentally it
>> seems to eliminate the race condition.
>
> Good enough for me if this helps. I also don't see reason why that would
> make things better (and probably not things worse), just that I got
> headache reading that Makefile ;) (and, for example, these particular
> targets mentioned would not need to be marked .PHONY...)
>
> I'd suggest to monitor the behaviour for a while and if things are
> consistently better then merge -- such a things when we don't know
> enough experience may be enough (or someone(tm) could try to parse make
> debug logs ;/) -- or someone(tm) may tell us why that heleps :D

Lucas confirmed this also fixes the problem on his 160 thread testbed,
so I've merged to master to prevent things getting out of sync.

In principle it's not a debian specific problem, so I could do a point
release if there is demand.

d