subsequent rebuilds of notmuch always re-build sphinx and ruby

subsequent rebuilds of notmuch always re-build sphinx and ruby

[Daniel Kahn Gillmor]

[-- Attachment #1: Type: text/plain, Size: 5169 bytes --]

Hi folks--

when i run "make" from my source tree, and it succeeds, i typically
expect that running "make" again will show that nothing needs to be
done.

but that's not the case.  Both the sphinx-based documentation and the
ruby notmuch.so are always rebuilt, i think due to some sort of
dependency loop.

But i don't really understand the dependencies there.  Maybe someone who
understands either ruby or sphinx better than i do can clean it up?
Having clean dependency tracking and quick rebuilds makes a project a
lot more fun to hack on.

Here's a trace of the rebuild process:

0 dkg@alice:~/src/notmuch/notmuch$ ./configure && make
 […]
make[1]: Leaving directory '/home/dkg/src/notmuch/notmuch/bindings/ruby'
0 dkg@alice:~/src/notmuch/notmuch$ make --trace
doc/Makefile.local:53: update target 'sphinx-html' due to: docstring.stamp
sphinx-build -b html -d doc/_build/doctrees -q ./doc doc/_build/html
doc/Makefile.local:56: update target 'sphinx-texinfo' due to: docstring.stamp
sphinx-build -b texinfo -d doc/_build/doctrees -q ./doc doc/_build/texinfo
doc/Makefile.local:59: update target 'sphinx-info' due to: sphinx-texinfo
make -C doc/_build/texinfo info
make[1]: Entering directory '/home/dkg/src/notmuch/notmuch/doc/_build/texinfo'
Makefile:32: update target 'notmuch-search-terms.info' due to: notmuch-search-terms.texi
makeinfo --no-split -o 'notmuch-search-terms.info' 'notmuch-search-terms.texi'
Makefile:32: update target 'notmuch-compact.info' due to: notmuch-compact.texi
makeinfo --no-split -o 'notmuch-compact.info' 'notmuch-compact.texi'
Makefile:32: update target 'notmuch-show.info' due to: notmuch-show.texi
makeinfo --no-split -o 'notmuch-show.info' 'notmuch-show.texi'
Makefile:32: update target 'notmuch-reply.info' due to: notmuch-reply.texi
makeinfo --no-split -o 'notmuch-reply.info' 'notmuch-reply.texi'
Makefile:32: update target 'notmuch-hooks.info' due to: notmuch-hooks.texi
makeinfo --no-split -o 'notmuch-hooks.info' 'notmuch-hooks.texi'
Makefile:32: update target 'notmuch-config.info' due to: notmuch-config.texi
makeinfo --no-split -o 'notmuch-config.info' 'notmuch-config.texi'
Makefile:32: update target 'notmuch-reindex.info' due to: notmuch-reindex.texi
makeinfo --no-split -o 'notmuch-reindex.info' 'notmuch-reindex.texi'
Makefile:32: update target 'notmuch-restore.info' due to: notmuch-restore.texi
makeinfo --no-split -o 'notmuch-restore.info' 'notmuch-restore.texi'
Makefile:32: update target 'notmuch-new.info' due to: notmuch-new.texi
makeinfo --no-split -o 'notmuch-new.info' 'notmuch-new.texi'
Makefile:32: update target 'notmuch-dump.info' due to: notmuch-dump.texi
makeinfo --no-split -o 'notmuch-dump.info' 'notmuch-dump.texi'
Makefile:32: update target 'notmuch-address.info' due to: notmuch-address.texi
makeinfo --no-split -o 'notmuch-address.info' 'notmuch-address.texi'
Makefile:32: update target 'notmuch-tag.info' due to: notmuch-tag.texi
makeinfo --no-split -o 'notmuch-tag.info' 'notmuch-tag.texi'
Makefile:32: update target 'notmuch-count.info' due to: notmuch-count.texi
makeinfo --no-split -o 'notmuch-count.info' 'notmuch-count.texi'
Makefile:32: update target 'notmuch-search.info' due to: notmuch-search.texi
makeinfo --no-split -o 'notmuch-search.info' 'notmuch-search.texi'
Makefile:32: update target 'notmuch-emacs-mua.info' due to: notmuch-emacs-mua.texi
makeinfo --no-split -o 'notmuch-emacs-mua.info' 'notmuch-emacs-mua.texi'
Makefile:32: update target 'notmuch-emacs.info' due to: notmuch-emacs.texi
makeinfo --no-split -o 'notmuch-emacs.info' 'notmuch-emacs.texi'
Makefile:32: update target 'notmuch-properties.info' due to: notmuch-properties.texi
makeinfo --no-split -o 'notmuch-properties.info' 'notmuch-properties.texi'
Makefile:32: update target 'notmuch-insert.info' due to: notmuch-insert.texi
makeinfo --no-split -o 'notmuch-insert.info' 'notmuch-insert.texi'
Makefile:32: update target 'notmuch.info' due to: notmuch.texi
makeinfo --no-split -o 'notmuch.info' 'notmuch.texi'
make[1]: Leaving directory '/home/dkg/src/notmuch/notmuch/doc/_build/texinfo'
bindings/Makefile.local:8: update target 'ruby-bindings' due to: lib/libnotmuch.so
cd bindings/ruby && \
	EXTRA_LDFLAGS="-Wl,--no-undefined" \
	LIBNOTMUCH="../../lib/libnotmuch.so" \
	NOTMUCH_SRCDIR='/home/dkg/src/notmuch/notmuch' \
	ruby extconf.rb --vendor
creating Makefile
make -C bindings/ruby
make[1]: Entering directory '/home/dkg/src/notmuch/notmuch/bindings/ruby'
Makefile:258: update target 'notmuch.so' due to: Makefile
echo linking shared-object notmuch.so
linking shared-object notmuch.so
rm -f notmuch.so
gcc -shared -o notmuch.so database.o directory.o filenames.o init.o message.o messages.o query.o status.o tags.o thread.o threads.o -L. -L/usr/lib/x86_64-linux-gnu -L. -Wl,-z,relro -Wl,-z,now -fstack-protector -rdynamic -Wl,-export-dynamic -Wl,--no-undefined -Wl,-z,relro -Wl,-z,now -Wl,--compress-debug-sections=zlib   ../../lib/libnotmuch.so -lruby-2.5  -lpthread -lgmp -ldl -lcrypt -lm   -lc
make[1]: Leaving directory '/home/dkg/src/notmuch/notmuch/bindings/ruby'
0 dkg@alice:~/src/notmuch/notmuch$


      --dkg

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: subsequent rebuilds of notmuch always re-build sphinx and ruby

[David Bremner]

Daniel Kahn Gillmor <dkg@fifthhorseman.net> writes:

> Hi folks--
>
> when i run "make" from my source tree, and it succeeds, i typically
> expect that running "make" again will show that nothing needs to be
> done.
>
> but that's not the case.  Both the sphinx-based documentation and the
> ruby notmuch.so are always rebuilt, i think due to some sort of
> dependency loop.
>
> But i don't really understand the dependencies there.  Maybe someone who
> understands either ruby or sphinx better than i do can clean it up?
> Having clean dependency tracking and quick rebuilds makes a project a
> lot more fun to hack on.
>
> Here's a trace of the rebuild process:
>
> 0 dkg@alice:~/src/notmuch/notmuch$ ./configure && make
>  […]
> make[1]: Leaving directory '/home/dkg/src/notmuch/notmuch/bindings/ruby'
> 0 dkg@alice:~/src/notmuch/notmuch$ make --trace
> doc/Makefile.local:53: update target 'sphinx-html' due to: docstring.stamp
> sphinx-build -b html -d doc/_build/doctrees -q ./doc doc/_build/html
> doc/Makefile.local:56: update target 'sphinx-texinfo' due to: docstring.stamp
> sphinx-build -b texinfo -d doc/_build/doctrees -q ./doc doc/_build/texinfo
> doc/Makefile.local:59: update target 'sphinx-info' due to: sphinx-texinfo
> make -C doc/_build/texinfo info
> make[1]: Entering directory '/home/dkg/src/notmuch/notmuch/doc/_build/texinfo'
> Makefile:32: update target 'notmuch-search-terms.info' due to: notmuch-search-terms.texi

This is not our Makefile, but something generated by sphinx; it would
not be that hard to replace if the problem was there. Alas I think the
underlying problem seems to be that "sphinx-build -b texinfo" is
regenerating (or at least touching) all of the texi files. I suspect
that's a limitation of sphinx-builder texinfo output.

> cd bindings/ruby && \
> 	EXTRA_LDFLAGS="-Wl,--no-undefined" \
> 	LIBNOTMUCH="../../lib/libnotmuch.so" \
> 	NOTMUCH_SRCDIR='/home/dkg/src/notmuch/notmuch' \
> 	ruby extconf.rb --vendor
> creating Makefile
> make -C bindings/ruby
> make[1]: Entering directory '/home/dkg/src/notmuch/notmuch/bindings/ruby'
> Makefile:258: update target 'notmuch.so' due to: Makefile
> echo linking shared-object notmuch.so
> linking shared-object notmuch.so
> rm -f notmuch.so
> gcc -shared -o notmuch.so database.o directory.o filenames.o init.o message.o messages.o query.o status.o tags.o thread.o threads.o -L. -L/usr/lib/x86_64-linux-gnu -L. -Wl,-z,relro -Wl,-z,now -fstack-protector -rdynamic -Wl,-export-dynamic -Wl,--no-undefined -Wl,-z,relro -Wl,-z,now -Wl,--compress-debug-sections=zlib   ../../lib/libnotmuch.so -lruby-2.5  -lpthread -lgmp -ldl -lcrypt -lm   -lc
> make[1]: Leaving directory '/home/dkg/src/notmuch/notmuch/bindings/ruby'
> 0 dkg@alice:~/src/notmuch/notmuch$

This Makefile is generated by "ruby extconf.rb --vendor". It includes a
dependency on itself, so it always fires after running "ruby
extconf.rb". It might be only running "ruby extconf.rb" if
bindings/ruby/Makefile does not exist would fix this particular
issue. That sounds more gnu make specific than ruby specific.

d

Re: subsequent rebuilds of notmuch always re-build sphinx and ruby

[Daniel Kahn Gillmor]

[-- Attachment #1: Type: text/plain, Size: 2925 bytes --]

thanks for the review, Bremner!

On Sat 2019-04-20 21:12:07 -0300, David Bremner wrote:
> Daniel Kahn Gillmor <dkg@fifthhorseman.net> writes:
>> 0 dkg@alice:~/src/notmuch/notmuch$ ./configure && make
>>  […]
>> make[1]: Leaving directory '/home/dkg/src/notmuch/notmuch/bindings/ruby'
>> 0 dkg@alice:~/src/notmuch/notmuch$ make --trace
>> doc/Makefile.local:53: update target 'sphinx-html' due to: docstring.stamp
>> sphinx-build -b html -d doc/_build/doctrees -q ./doc doc/_build/html
>> doc/Makefile.local:56: update target 'sphinx-texinfo' due to: docstring.stamp
>> sphinx-build -b texinfo -d doc/_build/doctrees -q ./doc doc/_build/texinfo
>> doc/Makefile.local:59: update target 'sphinx-info' due to: sphinx-texinfo
>> make -C doc/_build/texinfo info
>> make[1]: Entering directory '/home/dkg/src/notmuch/notmuch/doc/_build/texinfo'
>> Makefile:32: update target 'notmuch-search-terms.info' due to: notmuch-search-terms.texi
>
> This is not our Makefile, but something generated by sphinx; it would
> not be that hard to replace if the problem was there. Alas I think the
> underlying problem seems to be that "sphinx-build -b texinfo" is
> regenerating (or at least touching) all of the texi files. I suspect
> that's a limitation of sphinx-builder texinfo output.

but it's not just texinfo, right?  it starts with the html build
itself.  can we at least diagnose why that's happening?

>> cd bindings/ruby && \
>> 	EXTRA_LDFLAGS="-Wl,--no-undefined" \
>> 	LIBNOTMUCH="../../lib/libnotmuch.so" \
>> 	NOTMUCH_SRCDIR='/home/dkg/src/notmuch/notmuch' \
>> 	ruby extconf.rb --vendor
>> creating Makefile
>> make -C bindings/ruby
>> make[1]: Entering directory '/home/dkg/src/notmuch/notmuch/bindings/ruby'
>> Makefile:258: update target 'notmuch.so' due to: Makefile
>> echo linking shared-object notmuch.so
>> linking shared-object notmuch.so
>> rm -f notmuch.so
>> gcc -shared -o notmuch.so database.o directory.o filenames.o init.o message.o messages.o query.o status.o tags.o thread.o threads.o -L. -L/usr/lib/x86_64-linux-gnu -L. -Wl,-z,relro -Wl,-z,now -fstack-protector -rdynamic -Wl,-export-dynamic -Wl,--no-undefined -Wl,-z,relro -Wl,-z,now -Wl,--compress-debug-sections=zlib   ../../lib/libnotmuch.so -lruby-2.5  -lpthread -lgmp -ldl -lcrypt -lm   -lc
>> make[1]: Leaving directory '/home/dkg/src/notmuch/notmuch/bindings/ruby'
>> 0 dkg@alice:~/src/notmuch/notmuch$
>
> This Makefile is generated by "ruby extconf.rb --vendor". It includes a
> dependency on itself, so it always fires after running "ruby
> extconf.rb". It might be only running "ruby extconf.rb" if
> bindings/ruby/Makefile does not exist would fix this particular
> issue. That sounds more gnu make specific than ruby specific.

I'd be happy to test any proposed patches.  I don't really understand
this toolchain, or why anyone would build a makefile that rewrites
itself :/

       --dkg

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: subsequent rebuilds of notmuch always re-build sphinx and ruby

[David Bremner]

Daniel Kahn Gillmor <dkg@fifthhorseman.net> writes:

> but it's not just texinfo, right?  it starts with the html build
> itself.  can we at least diagnose why that's happening?
>

Yes, although the html rebuild is much faster than the texinfo + info rebuilds.

>> This Makefile is generated by "ruby extconf.rb --vendor". It includes a
>> dependency on itself, so it always fires after running "ruby
>> extconf.rb". It might be only running "ruby extconf.rb" if
>> bindings/ruby/Makefile does not exist would fix this particular
>> issue. That sounds more gnu make specific than ruby specific.

> I'd be happy to test any proposed patches.  I don't really understand
> this toolchain, or why anyone would build a makefile that rewrites
> itself :/

I've posted some patches for the sphinx-doc issues a couple of hours ago
(id:20190421171245.19729-1-david@tethera.net). 

Currently the ruby rebuild doesn't seem to be slowing things down much
for me.

╭─ convex:~/software/upstream/notmuch 
╰─ (git)-[wip/make-docs]-% /usr/bin/time make ruby-bindings 1>/dev/null
0.13user 0.02system 0:00.15elapsed 101%CPU (0avgtext+0avgdata 11504maxresident)k
0inputs+208outputs (0major+6523minor)pagefaults 0swaps

That's with an SSD, so maybe there's more of hit for other environments.

d

Re: subsequent rebuilds of notmuch always re-build sphinx and ruby

[Daniel Kahn Gillmor]

[-- Attachment #1: Type: text/plain, Size: 1856 bytes --]

On Sun 2019-04-21 16:29:02 -0300, David Bremner wrote:
> the html rebuild is much faster than the texinfo + info rebuilds.

agreed, in the runs that i've been doing as well.  I was concerned that
the html rebuild itself may have been *triggering* the rebuild of the
texinfo stuff, though.  Sounds like you don't think that's the case.

> I've posted some patches for the sphinx-doc issues a couple of hours ago
> (id:20190421171245.19729-1-david@tethera.net). 

thanks!  your own commentary on that series seems to acknowledge that
there are problems with it (though i don't understand the tradeoffs
well).  i'm not super comfortable with make-style "stamping": my
experience with it is that it creates a synchronization problem, and
it's easy for the "sync" between the stamp and the generated artifacts
to break, at which point the safest thing is to "make clean" and start
fully over.

Is there no way to give make itself full visibility into the specific
generated files so it can do its comparisons directly?  I'm obviously
not asking you to rewrite the entire native sphinx build system, i'm
just observing that at present it seems suboptimal, though i don't know
how to fix it either :/

> Currently the ruby rebuild doesn't seem to be slowing things down much
> for me.
>
> ╭─ convex:~/software/upstream/notmuch 
> ╰─ (git)-[wip/make-docs]-% /usr/bin/time make ruby-bindings 1>/dev/null
> 0.13user 0.02system 0:00.15elapsed 101%CPU (0avgtext+0avgdata 11504maxresident)k
> 0inputs+208outputs (0major+6523minor)pagefaults 0swaps
>
> That's with an SSD, so maybe there's more of hit for other environments.

I agree, this one isn't particularly slow, and it appears to be a leaf
dependency (for now), so it's not the worst thing.  But it's still
pretty clearly a bug that this thing loops.

       --dkg

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: subsequent rebuilds of notmuch always re-build sphinx and ruby

[David Bremner]

Daniel Kahn Gillmor <dkg@fifthhorseman.net> writes:

> On Sun 2019-04-21 16:29:02 -0300, David Bremner wrote:
>> the html rebuild is much faster than the texinfo + info rebuilds.
>
> agreed, in the runs that i've been doing as well.  I was concerned that
> the html rebuild itself may have been *triggering* the rebuild of the
> texinfo stuff, though.  Sounds like you don't think that's the case.
>
>> I've posted some patches for the sphinx-doc issues a couple of hours ago
>> (id:20190421171245.19729-1-david@tethera.net). 
>
> thanks!  your own commentary on that series seems to acknowledge that
> there are problems with it (though i don't understand the tradeoffs
> well).

There was a problem with the first patch, which I replaced with two more.


> Is there no way to give make itself full visibility into the specific
> generated files so it can do its comparisons directly?  I'm obviously
> not asking you to rewrite the entire native sphinx build system, i'm
> just observing that at present it seems suboptimal, though i don't know
> how to fix it either :/

I'm open to ideas, but keep in mind we want to support parallel make,
which means we have to be careful not to trigger multiple invocations of
sphinx-build in parallel.

Re: subsequent rebuilds of notmuch always re-build sphinx and ruby

[Daniel Kahn Gillmor]

[-- Attachment #1: Type: text/plain, Size: 5748 bytes --]

On Mon 2019-04-22 21:03:05 -0300, David Bremner wrote:
> There was a problem with the first patch, which I replaced with two more.

thanks.  i've reviewed and published my review on that series.  I think
it should probably be merged.

> I'm open to ideas, but keep in mind we want to support parallel make,
> which means we have to be careful not to trigger multiple invocations of
> sphinx-build in parallel.

hm, i'm not entirely sure why sphinx-build can't be run in parallel, if
it could target the creation of specific files (but maybe it can't).

I do note that (independent of this series), if i run the following
loop:


     while make -j4 --trace; do
        python3 -c 'print("="*100)'
        touch doc/man1/notmuch-reply.rst
     done

then only every second run of make contains this info:


doc/Makefile.local:38: update target 'doc/_build/man/man1/notmuch-reply.1.gz' due to: doc/_build/man/man1/notmuch-reply.1
rm -f doc/_build/man/man1/notmuch-reply.1.gz && gzip --stdout doc/_build/man/man1/notmuch-reply.1 > doc/_build/man/man1/notmuch-reply.1.gz
doc/Makefile.local:38: update target 'doc/_build/man/man1/notmuch-reindex.1.gz' due to: doc/_build/man/man1/notmuch-reindex.1
rm -f doc/_build/man/man1/notmuch-reindex.1.gz && gzip --stdout doc/_build/man/man1/notmuch-reindex.1 > doc/_build/man/man1/notmuch-reindex.1.gz
doc/Makefile.local:38: update target 'doc/_build/man/man1/notmuch-restore.1.gz' due to: doc/_build/man/man1/notmuch-restore.1
rm -f doc/_build/man/man1/notmuch-restore.1.gz && gzip --stdout doc/_build/man/man1/notmuch-restore.1 > doc/_build/man/man1/notmuch-restore.1.gz
doc/Makefile.local:38: update target 'doc/_build/man/man1/notmuch-new.1.gz' due to: doc/_build/man/man1/notmuch-new.1
rm -f doc/_build/man/man1/notmuch-new.1.gz && gzip --stdout doc/_build/man/man1/notmuch-new.1 > doc/_build/man/man1/notmuch-new.1.gz
doc/Makefile.local:38: update target 'doc/_build/man/man1/notmuch-dump.1.gz' due to: doc/_build/man/man1/notmuch-dump.1
rm -f doc/_build/man/man1/notmuch-dump.1.gz && gzip --stdout doc/_build/man/man1/notmuch-dump.1 > doc/_build/man/man1/notmuch-dump.1.gz
doc/Makefile.local:38: update target 'doc/_build/man/man1/notmuch-address.1.gz' due to: doc/_build/man/man1/notmuch-address.1
rm -f doc/_build/man/man1/notmuch-address.1.gz && gzip --stdout doc/_build/man/man1/notmuch-address.1 > doc/_build/man/man1/notmuch-address.1.gz
doc/Makefile.local:38: update target 'doc/_build/man/man1/notmuch-tag.1.gz' due to: doc/_build/man/man1/notmuch-tag.1
rm -f doc/_build/man/man1/notmuch-tag.1.gz && gzip --stdout doc/_build/man/man1/notmuch-tag.1 > doc/_build/man/man1/notmuch-tag.1.gz
doc/Makefile.local:38: update target 'doc/_build/man/man1/notmuch-count.1.gz' due to: doc/_build/man/man1/notmuch-count.1
rm -f doc/_build/man/man1/notmuch-count.1.gz && gzip --stdout doc/_build/man/man1/notmuch-count.1 > doc/_build/man/man1/notmuch-count.1.gz
doc/Makefile.local:38: update target 'doc/_build/man/man1/notmuch-search.1.gz' due to: doc/_build/man/man1/notmuch-search.1
rm -f doc/_build/man/man1/notmuch-search.1.gz && gzip --stdout doc/_build/man/man1/notmuch-search.1 > doc/_build/man/man1/notmuch-search.1.gz
doc/Makefile.local:38: update target 'doc/_build/man/man1/notmuch-emacs-mua.1.gz' due to: doc/_build/man/man1/notmuch-emacs-mua.1
rm -f doc/_build/man/man1/notmuch-emacs-mua.1.gz && gzip --stdout doc/_build/man/man1/notmuch-emacs-mua.1 > doc/_build/man/man1/notmuch-emacs-mua.1.gz
doc/Makefile.local:38: update target 'doc/_build/man/man1/notmuch-show.1.gz' due to: doc/_build/man/man1/notmuch-show.1
rm -f doc/_build/man/man1/notmuch-show.1.gz && gzip --stdout doc/_build/man/man1/notmuch-show.1 > doc/_build/man/man1/notmuch-show.1.gz
doc/Makefile.local:38: update target 'doc/_build/man/man1/notmuch-insert.1.gz' due to: doc/_build/man/man1/notmuch-insert.1
rm -f doc/_build/man/man1/notmuch-insert.1.gz && gzip --stdout doc/_build/man/man1/notmuch-insert.1 > doc/_build/man/man1/notmuch-insert.1.gz
doc/Makefile.local:38: update target 'doc/_build/man/man1/notmuch.1.gz' due to: doc/_build/man/man1/notmuch.1
rm -f doc/_build/man/man1/notmuch.1.gz && gzip --stdout doc/_build/man/man1/notmuch.1 > doc/_build/man/man1/notmuch.1.gz
doc/Makefile.local:38: update target 'doc/_build/man/man1/notmuch-compact.1.gz' due to: doc/_build/man/man1/notmuch-compact.1
rm -f doc/_build/man/man1/notmuch-compact.1.gz && gzip --stdout doc/_build/man/man1/notmuch-compact.1 > doc/_build/man/man1/notmuch-compact.1.gz
doc/Makefile.local:38: update target 'doc/_build/man/man1/notmuch-config.1.gz' due to: doc/_build/man/man1/notmuch-config.1
rm -f doc/_build/man/man1/notmuch-config.1.gz && gzip --stdout doc/_build/man/man1/notmuch-config.1 > doc/_build/man/man1/notmuch-config.1.gz
doc/Makefile.local:38: update target 'doc/_build/man/man5/notmuch-hooks.5.gz' due to: doc/_build/man/man5/notmuch-hooks.5
rm -f doc/_build/man/man5/notmuch-hooks.5.gz && gzip --stdout doc/_build/man/man5/notmuch-hooks.5 > doc/_build/man/man5/notmuch-hooks.5.gz
doc/Makefile.local:38: update target 'doc/_build/man/man7/notmuch-properties.7.gz' due to: doc/_build/man/man7/notmuch-properties.7
rm -f doc/_build/man/man7/notmuch-properties.7.gz && gzip --stdout doc/_build/man/man7/notmuch-properties.7 > doc/_build/man/man7/notmuch-properties.7.gz
doc/Makefile.local:38: update target 'doc/_build/man/man7/notmuch-search-terms.7.gz' due to: doc/_build/man/man7/notmuch-search-terms.7
rm -f doc/_build/man/man7/notmuch-search-terms.7.gz && gzip --stdout doc/_build/man/man7/notmuch-search-terms.7 > doc/_build/man/man7/notmuch-search-terms.7.gz

and the other times, it doesn't show up.  I'd expect this to be
idempotent, but i'm not sure why it does this.  it doesn't do it when
make is run with -j1.

     --dkg

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: subsequent rebuilds of notmuch always re-build sphinx and ruby

[David Bremner]

Daniel Kahn Gillmor <dkg@fifthhorseman.net> writes:

> On Mon 2019-04-22 21:03:05 -0300, David Bremner wrote:
>> There was a problem with the first patch, which I replaced with two more.
>
> thanks.  i've reviewed and published my review on that series.  I think
> it should probably be merged.
>
>> I'm open to ideas, but keep in mind we want to support parallel make,
>> which means we have to be careful not to trigger multiple invocations of
>> sphinx-build in parallel.

> hm, i'm not entirely sure why sphinx-build can't be run in parallel, if
> it could target the creation of specific files (but maybe it can't).

It can target specific files according to the documentation, but the
main issue is that it caches a bunch of state under
doc/_build/doctrees. It doesn't do any kind of locking, so multiple
writers leads to build failures.

>
> I do note that (independent of this series), if i run the following
> loop:
>
>
>      while make -j4 --trace; do
>         python3 -c 'print("="*100)'
>         touch doc/man1/notmuch-reply.rst
>      done
>
> then only every second run of make contains this info:
>

I think I see this also, but no idea yet what is going on.

Re: subsequent rebuilds of notmuch always re-build sphinx and ruby

[Daniel Kahn Gillmor]

On Tue 2019-04-23 22:09:23 -0300, David Bremner wrote:
> [dkg wrote:]
>> I do note that (independent of this series), if i run the following
>> loop:
>>
>>
>>      while make -j4 --trace; do
>>         python3 -c 'print("="*100)'
>>         touch doc/man1/notmuch-reply.rst
>>      done
>>
>> then only every second run of make contains this info:
>
> I think I see this also, but no idea yet what is going on.

I've tagged the previous mail with notmuch::bug so that we can keep
track of it, but i don't think it's a high priority.

thanks for having looked into it!

       --dkg

[PATCH] doc: add dep. on stamp file for rebuilding gzipped man pages.

[David Bremner]

In [1] Daniel observed that the gzipped man pages were only being
rebuild every second time when building with `make -j4'. This may be
caused by a race condition between sphinx-build rebuilding the roff
files and the recipe to gzip them. This commit sequentializes these
two steps by making the stamp file a prerequisite for (all of) the
gzip files.

[1]: id:87tveotn1g.fsf@fifthhorseman.net
---
 doc/Makefile.local | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/doc/Makefile.local b/doc/Makefile.local
index c2ae1743..d43ef269 100644
--- a/doc/Makefile.local
+++ b/doc/Makefile.local
@@ -117,6 +117,11 @@ build-man:
 install-man:
 	@echo "No sphinx, will not install man pages."
 else
+
+# it should be safe to depend on the stamp file, because it is created
+# after all roff files are moved into place.
+${MAN_GZIP_FILES}: ${DOCBUILDDIR}/.roff.stamp
+
 build-man: ${MAN_GZIP_FILES}
 install-man: ${MAN_GZIP_FILES}
 	mkdir -m0755 -p "$(DESTDIR)$(mandir)/man1"
-- 
2.34.1

Re: [PATCH] doc: add dep. on stamp file for rebuilding gzipped man pages.

[Tomi Ollila]

On Fri, Dec 24 2021, David Bremner wrote:

> In [1] Daniel observed that the gzipped man pages were only being
> rebuild every second time when building with `make -j4'. This may be
> caused by a race condition between sphinx-build rebuilding the roff
> files and the recipe to gzip them. This commit sequentializes these
> two steps by making the stamp file a prerequisite for (all of) the
> gzip files.
>
> [1]: id:87tveotn1g.fsf@fifthhorseman.net
> ---
>  doc/Makefile.local | 5 +++++
>  1 file changed, 5 insertions(+)
>
> diff --git a/doc/Makefile.local b/doc/Makefile.local
> index c2ae1743..d43ef269 100644
> --- a/doc/Makefile.local
> +++ b/doc/Makefile.local
> @@ -117,6 +117,11 @@ build-man:
>  install-man:
>  	@echo "No sphinx, will not install man pages."
>  else
> +
> +# it should be safe to depend on the stamp file, because it is created
> +# after all roff files are moved into place.
> +${MAN_GZIP_FILES}: ${DOCBUILDDIR}/.roff.stamp
> +

This thread becomes hard to follow...

... but the change looks sensible... the comment *should* be more
"assuritive" though ... :D


>  build-man: ${MAN_GZIP_FILES}
>  install-man: ${MAN_GZIP_FILES}
>  	mkdir -m0755 -p "$(DESTDIR)$(mandir)/man1"
> -- 
> 2.34.1

Re: [PATCH] doc: add dep. on stamp file for rebuilding gzipped man pages.

[David Bremner]

David Bremner <david@tethera.net> writes:

> In [1] Daniel observed that the gzipped man pages were only being
> rebuild every second time when building with `make -j4'. This may be
> caused by a race condition between sphinx-build rebuilding the roff
> files and the recipe to gzip them. This commit sequentializes these
> two steps by making the stamp file a prerequisite for (all of) the
> gzip files.
>
> [1]: id:87tveotn1g.fsf@fifthhorseman.net

Applied to master, wimpy commit message intact :P

d

Add stamp files to prevent rebuilds

[David Bremner]

There are several places where the current Makefiles either have a
dependency on a phony target, or on a directory. Both of these lead to
constant rebuilds.

Stamp files are not the most elegant solution, but it seems to prevent
the rebuilds.

[PATCH 1/3] doc: introduce stamp file for info build

[David Bremner]

This partially fixes (i.e. just for sphinx) the problem reported by
Daniel in id:87r29wwgq2.fsf@fifthhorseman.net.
---
 doc/Makefile.local | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/doc/Makefile.local b/doc/Makefile.local
index f476d1da..12e83140 100644
--- a/doc/Makefile.local
+++ b/doc/Makefile.local
@@ -58,8 +58,11 @@ $(DOCBUILDDIR)/.texi.stamp: $(ALL_RST_FILES)
 	$(SPHINXBUILD) -b texinfo -d $(DOCBUILDDIR)/texinfo_doctrees $(ALLSPHINXOPTS) $(DOCBUILDDIR)/texinfo
 	touch $@
 
-sphinx-info: sphinx-texinfo
+sphinx-info: $(DOCBUILDDIR)/.info.stamp
+
+$(DOCBUILDDIR)/.info.stamp: $(DOCBUILDDIR)/.texi.stamp
 	$(MAKE) -C $(DOCBUILDDIR)/texinfo info
+	touch $@
 
 # Use the man page converter that is available. We should never depend
 # on MAN_ROFF_FILES if a converter is not available.
@@ -141,5 +144,5 @@ $(dir)/config.dox: version.stamp
 	echo "INPUT=${srcdir}/lib/notmuch.h" >> $@
 
 CLEAN := $(CLEAN) $(DOCBUILDDIR) $(DOCBUILDDIR)/.roff.stamp $(DOCBUILDDIR)/.texi.stamp
-CLEAN := $(CLEAN) $(DOCBUILDDIR)/.html.stamp
+CLEAN := $(CLEAN) $(DOCBUILDDIR)/.html.stamp $(DOCBUILDDIR)/.info.stamp
 CLEAN := $(CLEAN) $(MAN_GZIP_FILES) $(MAN_ROFF_FILES) $(dir)/conf.pyc $(dir)/config.dox
-- 
2.33.0

[PATCH 2/3] ruby: don't use a directory as a target.

[David Bremner]

The directory is (neccesarily) not updated by the build, so it keeps
trying to build. The proposed fix is to use the name of the dynamic
library containing the extension.  This is a partial fix for the
rebuilding reported at [1].

[1]: id:87r29wwgq2.fsf@fifthhorseman.net
---
 bindings/Makefile.local | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/bindings/Makefile.local b/bindings/Makefile.local
index 3672e69f..1cdd28a0 100644
--- a/bindings/Makefile.local
+++ b/bindings/Makefile.local
@@ -3,21 +3,23 @@
 dir := bindings
 
 # force the shared library to be built
-ruby-bindings: lib/$(LINKER_NAME)
+ruby-bindings: $(dir)/ruby.stamp
+
+$(dir)/ruby.stamp: lib/$(LINKER_NAME)
 ifeq ($(HAVE_RUBY_DEV),1)
 	cd $(dir)/ruby && \
 		EXTRA_LDFLAGS="$(NO_UNDEFINED_LDFLAGS)" \
 		LIBNOTMUCH="../../lib/$(LINKER_NAME)" \
 		NOTMUCH_SRCDIR='$(NOTMUCH_SRCDIR)' \
 		$(RUBY) extconf.rb --vendor
-	$(MAKE) -C $(dir)/ruby CFLAGS="$(CFLAGS) -pipe -fno-plt -fPIC"
+	$(MAKE) -C $(dir)/ruby CFLAGS="$(CFLAGS) -pipe -fno-plt -fPIC" && touch $@
 endif
 
 python-cffi-bindings: lib/$(LINKER_NAME)
 ifeq ($(HAVE_PYTHON3_CFFI),1)
 	cd $(dir)/python-cffi && \
 		${PYTHON} setup.py build --build-lib build/stage && \
-		mkdir -p build/stage/tests && cp tests/*.py build/stage/tests
+		mkdir -p build/stage/tests && cp tests/*.py build/stage/tests && touch ../../$@
 endif
 
 CLEAN += $(patsubst %,$(dir)/ruby/%, \
@@ -26,6 +28,6 @@ CLEAN += $(patsubst %,$(dir)/ruby/%, \
 	init.o message.o messages.o mkmf.log notmuch.so query.o \
 	status.o tags.o thread.o threads.o)
 
-CLEAN += bindings/ruby/.vendorarchdir.time
+CLEAN += bindings/ruby/.vendorarchdir.time $(dir)/ruby.stamp
 
 CLEAN += bindings/python-cffi/build
-- 
2.33.0

[PATCH 3/3] python-cffi: introduce stamp file

[David Bremner]

Although the rebuild does not take long, it is a bit noisy, so assume
if it succeeds once, it doesn't need to re-invoke setup.py until the
shared library is rebuilt. This is a partial fix for [1].

[1]: id:87r29wwgq2.fsf@fifthhorseman.net
---
 bindings/Makefile.local | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)

diff --git a/bindings/Makefile.local b/bindings/Makefile.local
index 1cdd28a0..7b10af08 100644
--- a/bindings/Makefile.local
+++ b/bindings/Makefile.local
@@ -15,11 +15,14 @@ ifeq ($(HAVE_RUBY_DEV),1)
 	$(MAKE) -C $(dir)/ruby CFLAGS="$(CFLAGS) -pipe -fno-plt -fPIC" && touch $@
 endif
 
-python-cffi-bindings: lib/$(LINKER_NAME)
+python-cffi-bindings: $(dir)/python-cffi.stamp
+
+$(dir)/python-cffi.stamp: lib/$(LINKER_NAME)
 ifeq ($(HAVE_PYTHON3_CFFI),1)
 	cd $(dir)/python-cffi && \
 		${PYTHON} setup.py build --build-lib build/stage && \
-		mkdir -p build/stage/tests && cp tests/*.py build/stage/tests && touch ../../$@
+		mkdir -p build/stage/tests && cp tests/*.py build/stage/tests && \
+		touch ../python-cffi.stamp
 endif
 
 CLEAN += $(patsubst %,$(dir)/ruby/%, \
@@ -30,4 +33,4 @@ CLEAN += $(patsubst %,$(dir)/ruby/%, \
 
 CLEAN += bindings/ruby/.vendorarchdir.time $(dir)/ruby.stamp
 
-CLEAN += bindings/python-cffi/build
+CLEAN += bindings/python-cffi/build $(dir)/python-cffi.stamp
-- 
2.33.0

Re: Add stamp files to prevent rebuilds

[David Bremner]

David Bremner <david@tethera.net> writes:

> There are several places where the current Makefiles either have a
> dependency on a phony target, or on a directory. Both of these lead to
> constant rebuilds.
>
> Stamp files are not the most elegant solution, but it seems to prevent
> the rebuilds.

I applied patches 2 and 3 to master. Patch 1 doesn't seem quite right,
and in retrospect needs patches 2 and 3, so I will repost a new version
after this.

[PATCH 1/2] doc: replace phony target with variable

[David Bremner]

Depending on a phony target seems like a good way to always trigger a
recipe.
---
 doc/Makefile.local | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/doc/Makefile.local b/doc/Makefile.local
index 730ad4fb..1782c784 100644
--- a/doc/Makefile.local
+++ b/doc/Makefile.local
@@ -35,7 +35,7 @@ endif
 
 INFO_INFO_FILES := $(INFO_TEXI_FILES:.texi=.info)
 
-.PHONY: sphinx-html sphinx-texinfo sphinx-info doc-prereqs
+.PHONY: sphinx-html sphinx-texinfo sphinx-info
 
 .PHONY: install-man build-man apidocs install-apidocs
 
@@ -47,18 +47,20 @@ $(DOCBUILDDIR)/.roff.stamp $(DOCBUILDDIR)/.html.stamp $(DOCBUILDDIR)/.texi.stamp
 endif
 
 ifeq ($(HAVE_PYTHON3_CFFI),1)
-doc-prereqs: python-cffi-bindings
+DOC_PREREQS=bindings/python-cffi.stamp
+else
+DOC_PREREQS=
 endif
 
 sphinx-html: $(DOCBUILDDIR)/.html.stamp
 
-$(DOCBUILDDIR)/.html.stamp: $(ALL_RST_FILES) doc-prereqs
+$(DOCBUILDDIR)/.html.stamp: $(ALL_RST_FILES) $(DOC_PREREQS)
 	$(SPHINXBUILD) -b html -d $(DOCBUILDDIR)/html_doctrees $(ALLSPHINXOPTS) $(DOCBUILDDIR)/html
 	touch $@
 
 sphinx-texinfo: $(DOCBUILDDIR)/.texi.stamp
 
-$(DOCBUILDDIR)/.texi.stamp: $(ALL_RST_FILES) doc-prereqs
+$(DOCBUILDDIR)/.texi.stamp: $(ALL_RST_FILES) $(DOC_PREREQS)
 	$(SPHINXBUILD) -b texinfo -d $(DOCBUILDDIR)/texinfo_doctrees $(ALLSPHINXOPTS) $(DOCBUILDDIR)/texinfo
 	touch $@
 
-- 
2.33.0

[PATCH 2/2] doc: introduce stamp file for info build

[David Bremner]

This partially fixes (i.e. just for sphinx) the problem reported by
Daniel in id:87r29wwgq2.fsf@fifthhorseman.net.
---
 doc/Makefile.local | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)

diff --git a/doc/Makefile.local b/doc/Makefile.local
index 1782c784..c2ae1743 100644
--- a/doc/Makefile.local
+++ b/doc/Makefile.local
@@ -64,8 +64,11 @@ $(DOCBUILDDIR)/.texi.stamp: $(ALL_RST_FILES) $(DOC_PREREQS)
 	$(SPHINXBUILD) -b texinfo -d $(DOCBUILDDIR)/texinfo_doctrees $(ALLSPHINXOPTS) $(DOCBUILDDIR)/texinfo
 	touch $@
 
-sphinx-info: sphinx-texinfo
+sphinx-info: $(DOCBUILDDIR)/.info.stamp
+
+$(DOCBUILDDIR)/.info.stamp: $(DOCBUILDDIR)/.texi.stamp $(DOC_PREREQS)
 	$(MAKE) -C $(DOCBUILDDIR)/texinfo info
+	touch $@
 
 # Use the man page converter that is available. We should never depend
 # on MAN_ROFF_FILES if a converter is not available.
@@ -129,7 +132,7 @@ ifneq ($(HAVE_SPHINX)$(HAVE_MAKEINFO),11)
 build-info:
 	@echo "Missing sphinx or makeinfo, not building info pages"
 else
-build-info: sphinx-info
+build-info: $(DOCBUILDDIR)/.info.stamp
 endif
 
 ifneq ($(HAVE_SPHINX)$(HAVE_MAKEINFO)$(HAVE_INSTALL_INFO),111)
@@ -147,5 +150,5 @@ $(dir)/config.dox: version.stamp
 	echo "INPUT=${srcdir}/lib/notmuch.h" >> $@
 
 CLEAN := $(CLEAN) $(DOCBUILDDIR) $(DOCBUILDDIR)/.roff.stamp $(DOCBUILDDIR)/.texi.stamp
-CLEAN := $(CLEAN) $(DOCBUILDDIR)/.html.stamp
+CLEAN := $(CLEAN) $(DOCBUILDDIR)/.html.stamp $(DOCBUILDDIR)/.info.stamp
 CLEAN := $(CLEAN) $(MAN_GZIP_FILES) $(MAN_ROFF_FILES) $(dir)/conf.pyc $(dir)/config.dox
-- 
2.33.0

Re: [PATCH 1/2] doc: replace phony target with variable

[David Bremner]

David Bremner <david@tethera.net> writes:

> Depending on a phony target seems like a good way to always trigger a
> recipe.

Series applied to master, 

Marking the bug reported at id:87r29wwgq2.fsf@fifthhorseman.net as fixed
as of 031f4b4da5b317c580df474d002a8300d35a818e.