unofficial mirror of notmuch@notmuchmail.org
 help / color / mirror / code / Atom feed
* subsequent rebuilds of notmuch always re-build sphinx and ruby
@ 2019-04-20 20:43 Daniel Kahn Gillmor
  2019-04-21  0:12 ` David Bremner
  2021-10-30 20:48 ` Add stamp files to prevent rebuilds David Bremner
  0 siblings, 2 replies; 13+ messages in thread
From: Daniel Kahn Gillmor @ 2019-04-20 20:43 UTC (permalink / raw)
  To: Notmuch Mail

[-- Attachment #1: Type: text/plain, Size: 5169 bytes --]

Hi folks--

when i run "make" from my source tree, and it succeeds, i typically
expect that running "make" again will show that nothing needs to be
done.

but that's not the case.  Both the sphinx-based documentation and the
ruby notmuch.so are always rebuilt, i think due to some sort of
dependency loop.

But i don't really understand the dependencies there.  Maybe someone who
understands either ruby or sphinx better than i do can clean it up?
Having clean dependency tracking and quick rebuilds makes a project a
lot more fun to hack on.

Here's a trace of the rebuild process:

0 dkg@alice:~/src/notmuch/notmuch$ ./configure && make
 […]
make[1]: Leaving directory '/home/dkg/src/notmuch/notmuch/bindings/ruby'
0 dkg@alice:~/src/notmuch/notmuch$ make --trace
doc/Makefile.local:53: update target 'sphinx-html' due to: docstring.stamp
sphinx-build -b html -d doc/_build/doctrees -q ./doc doc/_build/html
doc/Makefile.local:56: update target 'sphinx-texinfo' due to: docstring.stamp
sphinx-build -b texinfo -d doc/_build/doctrees -q ./doc doc/_build/texinfo
doc/Makefile.local:59: update target 'sphinx-info' due to: sphinx-texinfo
make -C doc/_build/texinfo info
make[1]: Entering directory '/home/dkg/src/notmuch/notmuch/doc/_build/texinfo'
Makefile:32: update target 'notmuch-search-terms.info' due to: notmuch-search-terms.texi
makeinfo --no-split -o 'notmuch-search-terms.info' 'notmuch-search-terms.texi'
Makefile:32: update target 'notmuch-compact.info' due to: notmuch-compact.texi
makeinfo --no-split -o 'notmuch-compact.info' 'notmuch-compact.texi'
Makefile:32: update target 'notmuch-show.info' due to: notmuch-show.texi
makeinfo --no-split -o 'notmuch-show.info' 'notmuch-show.texi'
Makefile:32: update target 'notmuch-reply.info' due to: notmuch-reply.texi
makeinfo --no-split -o 'notmuch-reply.info' 'notmuch-reply.texi'
Makefile:32: update target 'notmuch-hooks.info' due to: notmuch-hooks.texi
makeinfo --no-split -o 'notmuch-hooks.info' 'notmuch-hooks.texi'
Makefile:32: update target 'notmuch-config.info' due to: notmuch-config.texi
makeinfo --no-split -o 'notmuch-config.info' 'notmuch-config.texi'
Makefile:32: update target 'notmuch-reindex.info' due to: notmuch-reindex.texi
makeinfo --no-split -o 'notmuch-reindex.info' 'notmuch-reindex.texi'
Makefile:32: update target 'notmuch-restore.info' due to: notmuch-restore.texi
makeinfo --no-split -o 'notmuch-restore.info' 'notmuch-restore.texi'
Makefile:32: update target 'notmuch-new.info' due to: notmuch-new.texi
makeinfo --no-split -o 'notmuch-new.info' 'notmuch-new.texi'
Makefile:32: update target 'notmuch-dump.info' due to: notmuch-dump.texi
makeinfo --no-split -o 'notmuch-dump.info' 'notmuch-dump.texi'
Makefile:32: update target 'notmuch-address.info' due to: notmuch-address.texi
makeinfo --no-split -o 'notmuch-address.info' 'notmuch-address.texi'
Makefile:32: update target 'notmuch-tag.info' due to: notmuch-tag.texi
makeinfo --no-split -o 'notmuch-tag.info' 'notmuch-tag.texi'
Makefile:32: update target 'notmuch-count.info' due to: notmuch-count.texi
makeinfo --no-split -o 'notmuch-count.info' 'notmuch-count.texi'
Makefile:32: update target 'notmuch-search.info' due to: notmuch-search.texi
makeinfo --no-split -o 'notmuch-search.info' 'notmuch-search.texi'
Makefile:32: update target 'notmuch-emacs-mua.info' due to: notmuch-emacs-mua.texi
makeinfo --no-split -o 'notmuch-emacs-mua.info' 'notmuch-emacs-mua.texi'
Makefile:32: update target 'notmuch-emacs.info' due to: notmuch-emacs.texi
makeinfo --no-split -o 'notmuch-emacs.info' 'notmuch-emacs.texi'
Makefile:32: update target 'notmuch-properties.info' due to: notmuch-properties.texi
makeinfo --no-split -o 'notmuch-properties.info' 'notmuch-properties.texi'
Makefile:32: update target 'notmuch-insert.info' due to: notmuch-insert.texi
makeinfo --no-split -o 'notmuch-insert.info' 'notmuch-insert.texi'
Makefile:32: update target 'notmuch.info' due to: notmuch.texi
makeinfo --no-split -o 'notmuch.info' 'notmuch.texi'
make[1]: Leaving directory '/home/dkg/src/notmuch/notmuch/doc/_build/texinfo'
bindings/Makefile.local:8: update target 'ruby-bindings' due to: lib/libnotmuch.so
cd bindings/ruby && \
	EXTRA_LDFLAGS="-Wl,--no-undefined" \
	LIBNOTMUCH="../../lib/libnotmuch.so" \
	NOTMUCH_SRCDIR='/home/dkg/src/notmuch/notmuch' \
	ruby extconf.rb --vendor
creating Makefile
make -C bindings/ruby
make[1]: Entering directory '/home/dkg/src/notmuch/notmuch/bindings/ruby'
Makefile:258: update target 'notmuch.so' due to: Makefile
echo linking shared-object notmuch.so
linking shared-object notmuch.so
rm -f notmuch.so
gcc -shared -o notmuch.so database.o directory.o filenames.o init.o message.o messages.o query.o status.o tags.o thread.o threads.o -L. -L/usr/lib/x86_64-linux-gnu -L. -Wl,-z,relro -Wl,-z,now -fstack-protector -rdynamic -Wl,-export-dynamic -Wl,--no-undefined -Wl,-z,relro -Wl,-z,now -Wl,--compress-debug-sections=zlib   ../../lib/libnotmuch.so -lruby-2.5  -lpthread -lgmp -ldl -lcrypt -lm   -lc
make[1]: Leaving directory '/home/dkg/src/notmuch/notmuch/bindings/ruby'
0 dkg@alice:~/src/notmuch/notmuch$


      --dkg

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: subsequent rebuilds of notmuch always re-build sphinx and ruby
  2019-04-20 20:43 subsequent rebuilds of notmuch always re-build sphinx and ruby Daniel Kahn Gillmor
@ 2019-04-21  0:12 ` David Bremner
  2019-04-21  3:14   ` Daniel Kahn Gillmor
  2021-10-30 20:48 ` Add stamp files to prevent rebuilds David Bremner
  1 sibling, 1 reply; 13+ messages in thread
From: David Bremner @ 2019-04-21  0:12 UTC (permalink / raw)
  To: Daniel Kahn Gillmor, Notmuch Mail

Daniel Kahn Gillmor <dkg@fifthhorseman.net> writes:

> Hi folks--
>
> when i run "make" from my source tree, and it succeeds, i typically
> expect that running "make" again will show that nothing needs to be
> done.
>
> but that's not the case.  Both the sphinx-based documentation and the
> ruby notmuch.so are always rebuilt, i think due to some sort of
> dependency loop.
>
> But i don't really understand the dependencies there.  Maybe someone who
> understands either ruby or sphinx better than i do can clean it up?
> Having clean dependency tracking and quick rebuilds makes a project a
> lot more fun to hack on.
>
> Here's a trace of the rebuild process:
>
> 0 dkg@alice:~/src/notmuch/notmuch$ ./configure && make
>  […]
> make[1]: Leaving directory '/home/dkg/src/notmuch/notmuch/bindings/ruby'
> 0 dkg@alice:~/src/notmuch/notmuch$ make --trace
> doc/Makefile.local:53: update target 'sphinx-html' due to: docstring.stamp
> sphinx-build -b html -d doc/_build/doctrees -q ./doc doc/_build/html
> doc/Makefile.local:56: update target 'sphinx-texinfo' due to: docstring.stamp
> sphinx-build -b texinfo -d doc/_build/doctrees -q ./doc doc/_build/texinfo
> doc/Makefile.local:59: update target 'sphinx-info' due to: sphinx-texinfo
> make -C doc/_build/texinfo info
> make[1]: Entering directory '/home/dkg/src/notmuch/notmuch/doc/_build/texinfo'
> Makefile:32: update target 'notmuch-search-terms.info' due to: notmuch-search-terms.texi

This is not our Makefile, but something generated by sphinx; it would
not be that hard to replace if the problem was there. Alas I think the
underlying problem seems to be that "sphinx-build -b texinfo" is
regenerating (or at least touching) all of the texi files. I suspect
that's a limitation of sphinx-builder texinfo output.

> cd bindings/ruby && \
> 	EXTRA_LDFLAGS="-Wl,--no-undefined" \
> 	LIBNOTMUCH="../../lib/libnotmuch.so" \
> 	NOTMUCH_SRCDIR='/home/dkg/src/notmuch/notmuch' \
> 	ruby extconf.rb --vendor
> creating Makefile
> make -C bindings/ruby
> make[1]: Entering directory '/home/dkg/src/notmuch/notmuch/bindings/ruby'
> Makefile:258: update target 'notmuch.so' due to: Makefile
> echo linking shared-object notmuch.so
> linking shared-object notmuch.so
> rm -f notmuch.so
> gcc -shared -o notmuch.so database.o directory.o filenames.o init.o message.o messages.o query.o status.o tags.o thread.o threads.o -L. -L/usr/lib/x86_64-linux-gnu -L. -Wl,-z,relro -Wl,-z,now -fstack-protector -rdynamic -Wl,-export-dynamic -Wl,--no-undefined -Wl,-z,relro -Wl,-z,now -Wl,--compress-debug-sections=zlib   ../../lib/libnotmuch.so -lruby-2.5  -lpthread -lgmp -ldl -lcrypt -lm   -lc
> make[1]: Leaving directory '/home/dkg/src/notmuch/notmuch/bindings/ruby'
> 0 dkg@alice:~/src/notmuch/notmuch$

This Makefile is generated by "ruby extconf.rb --vendor". It includes a
dependency on itself, so it always fires after running "ruby
extconf.rb". It might be only running "ruby extconf.rb" if
bindings/ruby/Makefile does not exist would fix this particular
issue. That sounds more gnu make specific than ruby specific.

d

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: subsequent rebuilds of notmuch always re-build sphinx and ruby
  2019-04-21  0:12 ` David Bremner
@ 2019-04-21  3:14   ` Daniel Kahn Gillmor
  2019-04-21 19:29     ` David Bremner
  0 siblings, 1 reply; 13+ messages in thread
From: Daniel Kahn Gillmor @ 2019-04-21  3:14 UTC (permalink / raw)
  To: David Bremner, Notmuch Mail

[-- Attachment #1: Type: text/plain, Size: 2925 bytes --]

thanks for the review, Bremner!

On Sat 2019-04-20 21:12:07 -0300, David Bremner wrote:
> Daniel Kahn Gillmor <dkg@fifthhorseman.net> writes:
>> 0 dkg@alice:~/src/notmuch/notmuch$ ./configure && make
>>  […]
>> make[1]: Leaving directory '/home/dkg/src/notmuch/notmuch/bindings/ruby'
>> 0 dkg@alice:~/src/notmuch/notmuch$ make --trace
>> doc/Makefile.local:53: update target 'sphinx-html' due to: docstring.stamp
>> sphinx-build -b html -d doc/_build/doctrees -q ./doc doc/_build/html
>> doc/Makefile.local:56: update target 'sphinx-texinfo' due to: docstring.stamp
>> sphinx-build -b texinfo -d doc/_build/doctrees -q ./doc doc/_build/texinfo
>> doc/Makefile.local:59: update target 'sphinx-info' due to: sphinx-texinfo
>> make -C doc/_build/texinfo info
>> make[1]: Entering directory '/home/dkg/src/notmuch/notmuch/doc/_build/texinfo'
>> Makefile:32: update target 'notmuch-search-terms.info' due to: notmuch-search-terms.texi
>
> This is not our Makefile, but something generated by sphinx; it would
> not be that hard to replace if the problem was there. Alas I think the
> underlying problem seems to be that "sphinx-build -b texinfo" is
> regenerating (or at least touching) all of the texi files. I suspect
> that's a limitation of sphinx-builder texinfo output.

but it's not just texinfo, right?  it starts with the html build
itself.  can we at least diagnose why that's happening?

>> cd bindings/ruby && \
>> 	EXTRA_LDFLAGS="-Wl,--no-undefined" \
>> 	LIBNOTMUCH="../../lib/libnotmuch.so" \
>> 	NOTMUCH_SRCDIR='/home/dkg/src/notmuch/notmuch' \
>> 	ruby extconf.rb --vendor
>> creating Makefile
>> make -C bindings/ruby
>> make[1]: Entering directory '/home/dkg/src/notmuch/notmuch/bindings/ruby'
>> Makefile:258: update target 'notmuch.so' due to: Makefile
>> echo linking shared-object notmuch.so
>> linking shared-object notmuch.so
>> rm -f notmuch.so
>> gcc -shared -o notmuch.so database.o directory.o filenames.o init.o message.o messages.o query.o status.o tags.o thread.o threads.o -L. -L/usr/lib/x86_64-linux-gnu -L. -Wl,-z,relro -Wl,-z,now -fstack-protector -rdynamic -Wl,-export-dynamic -Wl,--no-undefined -Wl,-z,relro -Wl,-z,now -Wl,--compress-debug-sections=zlib   ../../lib/libnotmuch.so -lruby-2.5  -lpthread -lgmp -ldl -lcrypt -lm   -lc
>> make[1]: Leaving directory '/home/dkg/src/notmuch/notmuch/bindings/ruby'
>> 0 dkg@alice:~/src/notmuch/notmuch$
>
> This Makefile is generated by "ruby extconf.rb --vendor". It includes a
> dependency on itself, so it always fires after running "ruby
> extconf.rb". It might be only running "ruby extconf.rb" if
> bindings/ruby/Makefile does not exist would fix this particular
> issue. That sounds more gnu make specific than ruby specific.

I'd be happy to test any proposed patches.  I don't really understand
this toolchain, or why anyone would build a makefile that rewrites
itself :/

       --dkg

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: subsequent rebuilds of notmuch always re-build sphinx and ruby
  2019-04-21  3:14   ` Daniel Kahn Gillmor
@ 2019-04-21 19:29     ` David Bremner
  2019-04-22 23:19       ` Daniel Kahn Gillmor
  0 siblings, 1 reply; 13+ messages in thread
From: David Bremner @ 2019-04-21 19:29 UTC (permalink / raw)
  To: Daniel Kahn Gillmor, Notmuch Mail

Daniel Kahn Gillmor <dkg@fifthhorseman.net> writes:

> but it's not just texinfo, right?  it starts with the html build
> itself.  can we at least diagnose why that's happening?
>

Yes, although the html rebuild is much faster than the texinfo + info rebuilds.

>> This Makefile is generated by "ruby extconf.rb --vendor". It includes a
>> dependency on itself, so it always fires after running "ruby
>> extconf.rb". It might be only running "ruby extconf.rb" if
>> bindings/ruby/Makefile does not exist would fix this particular
>> issue. That sounds more gnu make specific than ruby specific.

> I'd be happy to test any proposed patches.  I don't really understand
> this toolchain, or why anyone would build a makefile that rewrites
> itself :/

I've posted some patches for the sphinx-doc issues a couple of hours ago
(id:20190421171245.19729-1-david@tethera.net). 

Currently the ruby rebuild doesn't seem to be slowing things down much
for me.

╭─ convex:~/software/upstream/notmuch 
╰─ (git)-[wip/make-docs]-% /usr/bin/time make ruby-bindings 1>/dev/null
0.13user 0.02system 0:00.15elapsed 101%CPU (0avgtext+0avgdata 11504maxresident)k
0inputs+208outputs (0major+6523minor)pagefaults 0swaps

That's with an SSD, so maybe there's more of hit for other environments.

d

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: subsequent rebuilds of notmuch always re-build sphinx and ruby
  2019-04-21 19:29     ` David Bremner
@ 2019-04-22 23:19       ` Daniel Kahn Gillmor
  2019-04-23  0:03         ` David Bremner
  0 siblings, 1 reply; 13+ messages in thread
From: Daniel Kahn Gillmor @ 2019-04-22 23:19 UTC (permalink / raw)
  To: David Bremner, Notmuch Mail

[-- Attachment #1: Type: text/plain, Size: 1856 bytes --]

On Sun 2019-04-21 16:29:02 -0300, David Bremner wrote:
> the html rebuild is much faster than the texinfo + info rebuilds.

agreed, in the runs that i've been doing as well.  I was concerned that
the html rebuild itself may have been *triggering* the rebuild of the
texinfo stuff, though.  Sounds like you don't think that's the case.

> I've posted some patches for the sphinx-doc issues a couple of hours ago
> (id:20190421171245.19729-1-david@tethera.net). 

thanks!  your own commentary on that series seems to acknowledge that
there are problems with it (though i don't understand the tradeoffs
well).  i'm not super comfortable with make-style "stamping": my
experience with it is that it creates a synchronization problem, and
it's easy for the "sync" between the stamp and the generated artifacts
to break, at which point the safest thing is to "make clean" and start
fully over.

Is there no way to give make itself full visibility into the specific
generated files so it can do its comparisons directly?  I'm obviously
not asking you to rewrite the entire native sphinx build system, i'm
just observing that at present it seems suboptimal, though i don't know
how to fix it either :/

> Currently the ruby rebuild doesn't seem to be slowing things down much
> for me.
>
> ╭─ convex:~/software/upstream/notmuch 
> ╰─ (git)-[wip/make-docs]-% /usr/bin/time make ruby-bindings 1>/dev/null
> 0.13user 0.02system 0:00.15elapsed 101%CPU (0avgtext+0avgdata 11504maxresident)k
> 0inputs+208outputs (0major+6523minor)pagefaults 0swaps
>
> That's with an SSD, so maybe there's more of hit for other environments.

I agree, this one isn't particularly slow, and it appears to be a leaf
dependency (for now), so it's not the worst thing.  But it's still
pretty clearly a bug that this thing loops.

       --dkg

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: subsequent rebuilds of notmuch always re-build sphinx and ruby
  2019-04-22 23:19       ` Daniel Kahn Gillmor
@ 2019-04-23  0:03         ` David Bremner
  2019-04-23 21:43           ` Daniel Kahn Gillmor
  0 siblings, 1 reply; 13+ messages in thread
From: David Bremner @ 2019-04-23  0:03 UTC (permalink / raw)
  To: Daniel Kahn Gillmor, Notmuch Mail

Daniel Kahn Gillmor <dkg@fifthhorseman.net> writes:

> On Sun 2019-04-21 16:29:02 -0300, David Bremner wrote:
>> the html rebuild is much faster than the texinfo + info rebuilds.
>
> agreed, in the runs that i've been doing as well.  I was concerned that
> the html rebuild itself may have been *triggering* the rebuild of the
> texinfo stuff, though.  Sounds like you don't think that's the case.
>
>> I've posted some patches for the sphinx-doc issues a couple of hours ago
>> (id:20190421171245.19729-1-david@tethera.net). 
>
> thanks!  your own commentary on that series seems to acknowledge that
> there are problems with it (though i don't understand the tradeoffs
> well).

There was a problem with the first patch, which I replaced with two more.


> Is there no way to give make itself full visibility into the specific
> generated files so it can do its comparisons directly?  I'm obviously
> not asking you to rewrite the entire native sphinx build system, i'm
> just observing that at present it seems suboptimal, though i don't know
> how to fix it either :/

I'm open to ideas, but keep in mind we want to support parallel make,
which means we have to be careful not to trigger multiple invocations of
sphinx-build in parallel.

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: subsequent rebuilds of notmuch always re-build sphinx and ruby
  2019-04-23  0:03         ` David Bremner
@ 2019-04-23 21:43           ` Daniel Kahn Gillmor
  2019-04-24  1:09             ` David Bremner
  0 siblings, 1 reply; 13+ messages in thread
From: Daniel Kahn Gillmor @ 2019-04-23 21:43 UTC (permalink / raw)
  To: David Bremner, Notmuch Mail

[-- Attachment #1: Type: text/plain, Size: 5748 bytes --]

On Mon 2019-04-22 21:03:05 -0300, David Bremner wrote:
> There was a problem with the first patch, which I replaced with two more.

thanks.  i've reviewed and published my review on that series.  I think
it should probably be merged.

> I'm open to ideas, but keep in mind we want to support parallel make,
> which means we have to be careful not to trigger multiple invocations of
> sphinx-build in parallel.

hm, i'm not entirely sure why sphinx-build can't be run in parallel, if
it could target the creation of specific files (but maybe it can't).

I do note that (independent of this series), if i run the following
loop:


     while make -j4 --trace; do
        python3 -c 'print("="*100)'
        touch doc/man1/notmuch-reply.rst
     done

then only every second run of make contains this info:


doc/Makefile.local:38: update target 'doc/_build/man/man1/notmuch-reply.1.gz' due to: doc/_build/man/man1/notmuch-reply.1
rm -f doc/_build/man/man1/notmuch-reply.1.gz && gzip --stdout doc/_build/man/man1/notmuch-reply.1 > doc/_build/man/man1/notmuch-reply.1.gz
doc/Makefile.local:38: update target 'doc/_build/man/man1/notmuch-reindex.1.gz' due to: doc/_build/man/man1/notmuch-reindex.1
rm -f doc/_build/man/man1/notmuch-reindex.1.gz && gzip --stdout doc/_build/man/man1/notmuch-reindex.1 > doc/_build/man/man1/notmuch-reindex.1.gz
doc/Makefile.local:38: update target 'doc/_build/man/man1/notmuch-restore.1.gz' due to: doc/_build/man/man1/notmuch-restore.1
rm -f doc/_build/man/man1/notmuch-restore.1.gz && gzip --stdout doc/_build/man/man1/notmuch-restore.1 > doc/_build/man/man1/notmuch-restore.1.gz
doc/Makefile.local:38: update target 'doc/_build/man/man1/notmuch-new.1.gz' due to: doc/_build/man/man1/notmuch-new.1
rm -f doc/_build/man/man1/notmuch-new.1.gz && gzip --stdout doc/_build/man/man1/notmuch-new.1 > doc/_build/man/man1/notmuch-new.1.gz
doc/Makefile.local:38: update target 'doc/_build/man/man1/notmuch-dump.1.gz' due to: doc/_build/man/man1/notmuch-dump.1
rm -f doc/_build/man/man1/notmuch-dump.1.gz && gzip --stdout doc/_build/man/man1/notmuch-dump.1 > doc/_build/man/man1/notmuch-dump.1.gz
doc/Makefile.local:38: update target 'doc/_build/man/man1/notmuch-address.1.gz' due to: doc/_build/man/man1/notmuch-address.1
rm -f doc/_build/man/man1/notmuch-address.1.gz && gzip --stdout doc/_build/man/man1/notmuch-address.1 > doc/_build/man/man1/notmuch-address.1.gz
doc/Makefile.local:38: update target 'doc/_build/man/man1/notmuch-tag.1.gz' due to: doc/_build/man/man1/notmuch-tag.1
rm -f doc/_build/man/man1/notmuch-tag.1.gz && gzip --stdout doc/_build/man/man1/notmuch-tag.1 > doc/_build/man/man1/notmuch-tag.1.gz
doc/Makefile.local:38: update target 'doc/_build/man/man1/notmuch-count.1.gz' due to: doc/_build/man/man1/notmuch-count.1
rm -f doc/_build/man/man1/notmuch-count.1.gz && gzip --stdout doc/_build/man/man1/notmuch-count.1 > doc/_build/man/man1/notmuch-count.1.gz
doc/Makefile.local:38: update target 'doc/_build/man/man1/notmuch-search.1.gz' due to: doc/_build/man/man1/notmuch-search.1
rm -f doc/_build/man/man1/notmuch-search.1.gz && gzip --stdout doc/_build/man/man1/notmuch-search.1 > doc/_build/man/man1/notmuch-search.1.gz
doc/Makefile.local:38: update target 'doc/_build/man/man1/notmuch-emacs-mua.1.gz' due to: doc/_build/man/man1/notmuch-emacs-mua.1
rm -f doc/_build/man/man1/notmuch-emacs-mua.1.gz && gzip --stdout doc/_build/man/man1/notmuch-emacs-mua.1 > doc/_build/man/man1/notmuch-emacs-mua.1.gz
doc/Makefile.local:38: update target 'doc/_build/man/man1/notmuch-show.1.gz' due to: doc/_build/man/man1/notmuch-show.1
rm -f doc/_build/man/man1/notmuch-show.1.gz && gzip --stdout doc/_build/man/man1/notmuch-show.1 > doc/_build/man/man1/notmuch-show.1.gz
doc/Makefile.local:38: update target 'doc/_build/man/man1/notmuch-insert.1.gz' due to: doc/_build/man/man1/notmuch-insert.1
rm -f doc/_build/man/man1/notmuch-insert.1.gz && gzip --stdout doc/_build/man/man1/notmuch-insert.1 > doc/_build/man/man1/notmuch-insert.1.gz
doc/Makefile.local:38: update target 'doc/_build/man/man1/notmuch.1.gz' due to: doc/_build/man/man1/notmuch.1
rm -f doc/_build/man/man1/notmuch.1.gz && gzip --stdout doc/_build/man/man1/notmuch.1 > doc/_build/man/man1/notmuch.1.gz
doc/Makefile.local:38: update target 'doc/_build/man/man1/notmuch-compact.1.gz' due to: doc/_build/man/man1/notmuch-compact.1
rm -f doc/_build/man/man1/notmuch-compact.1.gz && gzip --stdout doc/_build/man/man1/notmuch-compact.1 > doc/_build/man/man1/notmuch-compact.1.gz
doc/Makefile.local:38: update target 'doc/_build/man/man1/notmuch-config.1.gz' due to: doc/_build/man/man1/notmuch-config.1
rm -f doc/_build/man/man1/notmuch-config.1.gz && gzip --stdout doc/_build/man/man1/notmuch-config.1 > doc/_build/man/man1/notmuch-config.1.gz
doc/Makefile.local:38: update target 'doc/_build/man/man5/notmuch-hooks.5.gz' due to: doc/_build/man/man5/notmuch-hooks.5
rm -f doc/_build/man/man5/notmuch-hooks.5.gz && gzip --stdout doc/_build/man/man5/notmuch-hooks.5 > doc/_build/man/man5/notmuch-hooks.5.gz
doc/Makefile.local:38: update target 'doc/_build/man/man7/notmuch-properties.7.gz' due to: doc/_build/man/man7/notmuch-properties.7
rm -f doc/_build/man/man7/notmuch-properties.7.gz && gzip --stdout doc/_build/man/man7/notmuch-properties.7 > doc/_build/man/man7/notmuch-properties.7.gz
doc/Makefile.local:38: update target 'doc/_build/man/man7/notmuch-search-terms.7.gz' due to: doc/_build/man/man7/notmuch-search-terms.7
rm -f doc/_build/man/man7/notmuch-search-terms.7.gz && gzip --stdout doc/_build/man/man7/notmuch-search-terms.7 > doc/_build/man/man7/notmuch-search-terms.7.gz

and the other times, it doesn't show up.  I'd expect this to be
idempotent, but i'm not sure why it does this.  it doesn't do it when
make is run with -j1.

     --dkg

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: subsequent rebuilds of notmuch always re-build sphinx and ruby
  2019-04-23 21:43           ` Daniel Kahn Gillmor
@ 2019-04-24  1:09             ` David Bremner
  2019-04-24  5:11               ` Daniel Kahn Gillmor
  0 siblings, 1 reply; 13+ messages in thread
From: David Bremner @ 2019-04-24  1:09 UTC (permalink / raw)
  To: Daniel Kahn Gillmor, Notmuch Mail

Daniel Kahn Gillmor <dkg@fifthhorseman.net> writes:

> On Mon 2019-04-22 21:03:05 -0300, David Bremner wrote:
>> There was a problem with the first patch, which I replaced with two more.
>
> thanks.  i've reviewed and published my review on that series.  I think
> it should probably be merged.
>
>> I'm open to ideas, but keep in mind we want to support parallel make,
>> which means we have to be careful not to trigger multiple invocations of
>> sphinx-build in parallel.

> hm, i'm not entirely sure why sphinx-build can't be run in parallel, if
> it could target the creation of specific files (but maybe it can't).

It can target specific files according to the documentation, but the
main issue is that it caches a bunch of state under
doc/_build/doctrees. It doesn't do any kind of locking, so multiple
writers leads to build failures.

>
> I do note that (independent of this series), if i run the following
> loop:
>
>
>      while make -j4 --trace; do
>         python3 -c 'print("="*100)'
>         touch doc/man1/notmuch-reply.rst
>      done
>
> then only every second run of make contains this info:
>

I think I see this also, but no idea yet what is going on.

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: subsequent rebuilds of notmuch always re-build sphinx and ruby
  2019-04-24  1:09             ` David Bremner
@ 2019-04-24  5:11               ` Daniel Kahn Gillmor
  0 siblings, 0 replies; 13+ messages in thread
From: Daniel Kahn Gillmor @ 2019-04-24  5:11 UTC (permalink / raw)
  To: David Bremner, Notmuch Mail

On Tue 2019-04-23 22:09:23 -0300, David Bremner wrote:
> [dkg wrote:]
>> I do note that (independent of this series), if i run the following
>> loop:
>>
>>
>>      while make -j4 --trace; do
>>         python3 -c 'print("="*100)'
>>         touch doc/man1/notmuch-reply.rst
>>      done
>>
>> then only every second run of make contains this info:
>
> I think I see this also, but no idea yet what is going on.

I've tagged the previous mail with notmuch::bug so that we can keep
track of it, but i don't think it's a high priority.

thanks for having looked into it!

       --dkg

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Add stamp files to prevent rebuilds
  2019-04-20 20:43 subsequent rebuilds of notmuch always re-build sphinx and ruby Daniel Kahn Gillmor
  2019-04-21  0:12 ` David Bremner
@ 2021-10-30 20:48 ` David Bremner
  2021-10-30 20:48   ` [PATCH 1/3] doc: introduce stamp file for info build David Bremner
                     ` (2 more replies)
  1 sibling, 3 replies; 13+ messages in thread
From: David Bremner @ 2021-10-30 20:48 UTC (permalink / raw)
  To: Daniel Kahn Gillmor, Notmuch Mail

There are several places where the current Makefiles either have a
dependency on a phony target, or on a directory. Both of these lead to
constant rebuilds.

Stamp files are not the most elegant solution, but it seems to prevent
the rebuilds.


^ permalink raw reply	[flat|nested] 13+ messages in thread

* [PATCH 1/3] doc: introduce stamp file for info build
  2021-10-30 20:48 ` Add stamp files to prevent rebuilds David Bremner
@ 2021-10-30 20:48   ` David Bremner
  2021-10-30 20:48   ` [PATCH 2/3] ruby: don't use a directory as a target David Bremner
  2021-10-30 20:48   ` [PATCH 3/3] python-cffi: introduce stamp file David Bremner
  2 siblings, 0 replies; 13+ messages in thread
From: David Bremner @ 2021-10-30 20:48 UTC (permalink / raw)
  To: Daniel Kahn Gillmor, Notmuch Mail

This partially fixes (i.e. just for sphinx) the problem reported by
Daniel in id:87r29wwgq2.fsf@fifthhorseman.net.
---
 doc/Makefile.local | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/doc/Makefile.local b/doc/Makefile.local
index f476d1da..12e83140 100644
--- a/doc/Makefile.local
+++ b/doc/Makefile.local
@@ -58,8 +58,11 @@ $(DOCBUILDDIR)/.texi.stamp: $(ALL_RST_FILES)
 	$(SPHINXBUILD) -b texinfo -d $(DOCBUILDDIR)/texinfo_doctrees $(ALLSPHINXOPTS) $(DOCBUILDDIR)/texinfo
 	touch $@
 
-sphinx-info: sphinx-texinfo
+sphinx-info: $(DOCBUILDDIR)/.info.stamp
+
+$(DOCBUILDDIR)/.info.stamp: $(DOCBUILDDIR)/.texi.stamp
 	$(MAKE) -C $(DOCBUILDDIR)/texinfo info
+	touch $@
 
 # Use the man page converter that is available. We should never depend
 # on MAN_ROFF_FILES if a converter is not available.
@@ -141,5 +144,5 @@ $(dir)/config.dox: version.stamp
 	echo "INPUT=${srcdir}/lib/notmuch.h" >> $@
 
 CLEAN := $(CLEAN) $(DOCBUILDDIR) $(DOCBUILDDIR)/.roff.stamp $(DOCBUILDDIR)/.texi.stamp
-CLEAN := $(CLEAN) $(DOCBUILDDIR)/.html.stamp
+CLEAN := $(CLEAN) $(DOCBUILDDIR)/.html.stamp $(DOCBUILDDIR)/.info.stamp
 CLEAN := $(CLEAN) $(MAN_GZIP_FILES) $(MAN_ROFF_FILES) $(dir)/conf.pyc $(dir)/config.dox
-- 
2.33.0

^ permalink raw reply	[flat|nested] 13+ messages in thread

* [PATCH 2/3] ruby: don't use a directory as a target.
  2021-10-30 20:48 ` Add stamp files to prevent rebuilds David Bremner
  2021-10-30 20:48   ` [PATCH 1/3] doc: introduce stamp file for info build David Bremner
@ 2021-10-30 20:48   ` David Bremner
  2021-10-30 20:48   ` [PATCH 3/3] python-cffi: introduce stamp file David Bremner
  2 siblings, 0 replies; 13+ messages in thread
From: David Bremner @ 2021-10-30 20:48 UTC (permalink / raw)
  To: Daniel Kahn Gillmor, Notmuch Mail

The directory is (neccesarily) not updated by the build, so it keeps
trying to build. The proposed fix is to use the name of the dynamic
library containing the extension.  This is a partial fix for the
rebuilding reported at [1].

[1]: id:87r29wwgq2.fsf@fifthhorseman.net
---
 bindings/Makefile.local | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/bindings/Makefile.local b/bindings/Makefile.local
index 3672e69f..1cdd28a0 100644
--- a/bindings/Makefile.local
+++ b/bindings/Makefile.local
@@ -3,21 +3,23 @@
 dir := bindings
 
 # force the shared library to be built
-ruby-bindings: lib/$(LINKER_NAME)
+ruby-bindings: $(dir)/ruby.stamp
+
+$(dir)/ruby.stamp: lib/$(LINKER_NAME)
 ifeq ($(HAVE_RUBY_DEV),1)
 	cd $(dir)/ruby && \
 		EXTRA_LDFLAGS="$(NO_UNDEFINED_LDFLAGS)" \
 		LIBNOTMUCH="../../lib/$(LINKER_NAME)" \
 		NOTMUCH_SRCDIR='$(NOTMUCH_SRCDIR)' \
 		$(RUBY) extconf.rb --vendor
-	$(MAKE) -C $(dir)/ruby CFLAGS="$(CFLAGS) -pipe -fno-plt -fPIC"
+	$(MAKE) -C $(dir)/ruby CFLAGS="$(CFLAGS) -pipe -fno-plt -fPIC" && touch $@
 endif
 
 python-cffi-bindings: lib/$(LINKER_NAME)
 ifeq ($(HAVE_PYTHON3_CFFI),1)
 	cd $(dir)/python-cffi && \
 		${PYTHON} setup.py build --build-lib build/stage && \
-		mkdir -p build/stage/tests && cp tests/*.py build/stage/tests
+		mkdir -p build/stage/tests && cp tests/*.py build/stage/tests && touch ../../$@
 endif
 
 CLEAN += $(patsubst %,$(dir)/ruby/%, \
@@ -26,6 +28,6 @@ CLEAN += $(patsubst %,$(dir)/ruby/%, \
 	init.o message.o messages.o mkmf.log notmuch.so query.o \
 	status.o tags.o thread.o threads.o)
 
-CLEAN += bindings/ruby/.vendorarchdir.time
+CLEAN += bindings/ruby/.vendorarchdir.time $(dir)/ruby.stamp
 
 CLEAN += bindings/python-cffi/build
-- 
2.33.0

^ permalink raw reply	[flat|nested] 13+ messages in thread

* [PATCH 3/3] python-cffi: introduce stamp file
  2021-10-30 20:48 ` Add stamp files to prevent rebuilds David Bremner
  2021-10-30 20:48   ` [PATCH 1/3] doc: introduce stamp file for info build David Bremner
  2021-10-30 20:48   ` [PATCH 2/3] ruby: don't use a directory as a target David Bremner
@ 2021-10-30 20:48   ` David Bremner
  2 siblings, 0 replies; 13+ messages in thread
From: David Bremner @ 2021-10-30 20:48 UTC (permalink / raw)
  To: Daniel Kahn Gillmor, Notmuch Mail

Although the rebuild does not take long, it is a bit noisy, so assume
if it succeeds once, it doesn't need to re-invoke setup.py until the
shared library is rebuilt. This is a partial fix for [1].

[1]: id:87r29wwgq2.fsf@fifthhorseman.net
---
 bindings/Makefile.local | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)

diff --git a/bindings/Makefile.local b/bindings/Makefile.local
index 1cdd28a0..7b10af08 100644
--- a/bindings/Makefile.local
+++ b/bindings/Makefile.local
@@ -15,11 +15,14 @@ ifeq ($(HAVE_RUBY_DEV),1)
 	$(MAKE) -C $(dir)/ruby CFLAGS="$(CFLAGS) -pipe -fno-plt -fPIC" && touch $@
 endif
 
-python-cffi-bindings: lib/$(LINKER_NAME)
+python-cffi-bindings: $(dir)/python-cffi.stamp
+
+$(dir)/python-cffi.stamp: lib/$(LINKER_NAME)
 ifeq ($(HAVE_PYTHON3_CFFI),1)
 	cd $(dir)/python-cffi && \
 		${PYTHON} setup.py build --build-lib build/stage && \
-		mkdir -p build/stage/tests && cp tests/*.py build/stage/tests && touch ../../$@
+		mkdir -p build/stage/tests && cp tests/*.py build/stage/tests && \
+		touch ../python-cffi.stamp
 endif
 
 CLEAN += $(patsubst %,$(dir)/ruby/%, \
@@ -30,4 +33,4 @@ CLEAN += $(patsubst %,$(dir)/ruby/%, \
 
 CLEAN += bindings/ruby/.vendorarchdir.time $(dir)/ruby.stamp
 
-CLEAN += bindings/python-cffi/build
+CLEAN += bindings/python-cffi/build $(dir)/python-cffi.stamp
-- 
2.33.0

^ permalink raw reply	[flat|nested] 13+ messages in thread

end of thread, other threads:[~2021-10-30 20:49 UTC | newest]

Thread overview: 13+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-04-20 20:43 subsequent rebuilds of notmuch always re-build sphinx and ruby Daniel Kahn Gillmor
2019-04-21  0:12 ` David Bremner
2019-04-21  3:14   ` Daniel Kahn Gillmor
2019-04-21 19:29     ` David Bremner
2019-04-22 23:19       ` Daniel Kahn Gillmor
2019-04-23  0:03         ` David Bremner
2019-04-23 21:43           ` Daniel Kahn Gillmor
2019-04-24  1:09             ` David Bremner
2019-04-24  5:11               ` Daniel Kahn Gillmor
2021-10-30 20:48 ` Add stamp files to prevent rebuilds David Bremner
2021-10-30 20:48   ` [PATCH 1/3] doc: introduce stamp file for info build David Bremner
2021-10-30 20:48   ` [PATCH 2/3] ruby: don't use a directory as a target David Bremner
2021-10-30 20:48   ` [PATCH 3/3] python-cffi: introduce stamp file David Bremner

Code repositories for project(s) associated with this inbox:

	notmuch.git.git (no URL configured)

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).