[RFC PATCH] rfc: make 'make distclean' always use current Makefile.config

[RFC PATCH] rfc: make 'make distclean' always use current Makefile.config

[Tomi Ollila]

... and not recreate it (slowly) if 'configure' changed, just to
be deleteted later during 'make distclean'.

For cleaning the old Makefile.config might actually be more accurate
if configure changes what will be cleaned later...
---
 Makefile | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/Makefile b/Makefile
index 0ef57fa..be527e9 100644
--- a/Makefile
+++ b/Makefile
@@ -22,7 +22,13 @@ include Makefile.config
 global_deps = Makefile Makefile.config Makefile.local \
 	$(subdirs:%=%/Makefile) $(subdirs:%=%/Makefile.local)
 
-Makefile.config: $(srcdir)/configure
+# In case of 'make distclean' have configure as order-only prerequisite so
+# that Makefile.config is not recreated when configure is newer than it is.
+ifeq ($(MAKECMDGOALS),distclean)
+Makefile.config: | $(srcdir)/configure
+else
+Makefile.config:   $(srcdir)/configure
+endif
 ifeq ($(configure_options),)
 	@echo ""
 	@echo "Note: Calling ./configure with no command-line arguments. This is often fine,"
-- 
2.7.4

Re: [RFC PATCH] rfc: make 'make distclean' always use current Makefile.config

[David Bremner]

Tomi Ollila <tomi.ollila@iki.fi> writes:

I like the idea here; it's an annoying wait. Could we do something
similar for make clean?

> ... and not recreate it (slowly) if 'configure' changed, just to
> be deleteted later during 'make distclean'.
spellink

> +# In case of 'make distclean' have configure as order-only prerequisite so
> +# that Makefile.config is not recreated when configure is newer than it is.
> +ifeq ($(MAKECMDGOALS),distclean)

Two comments

- do we want to check for exactly distclean, or distclean one of the
  goals?

- it turns out that specifying e.g. "make distclean all" is currently
  broken (before your patch)

Re: [RFC PATCH] rfc: make 'make distclean' always use current Makefile.config

[Tomi Ollila]

On Sun, Jan 08 2017, David Bremner <david@tethera.net> wrote:

> Tomi Ollila <tomi.ollila@iki.fi> writes:
>
> I like the idea here; it's an annoying wait. Could we do something
> similar for make clean?

yes exact `make clean`, see below.

>
>> ... and not recreate it (slowly) if 'configure' changed, just to
>> be deleteted later during 'make distclean'.
> spellink
>
>> +# In case of 'make distclean' have configure as order-only prerequisite so
>> +# that Makefile.config is not recreated when configure is newer than it is.
>> +ifeq ($(MAKECMDGOALS),distclean)
>
> Two comments
>
> - do we want to check for exactly distclean, or distclean one of the
>   goals?
>
> - it turns out that specifying e.g. "make distclean all" is currently
>   broken (before your patch)

I tried to make support for 'make distclean all', but it just doesn't work
w/o heavy trickery.

After `make distclean all` has removed Makefile.config the prerequisities
of 'all' starts to run; it mysteriously fails (but if it did not, new
Makefile.config was not created -- but if new Makefile.config was
created, make would re-exec itself as `make distclean all` since a file
it included changed (and it would re-remove Makefile.config...)).

I tried an alternative for `make clean all`, where 'all:' rule line
in the beginning of Makefile.local had an ifeq alternative
`$(MAKE) all configure_options=$(configure_options)`
and without prerequisities 'notmuch notmuch-shared build-man ruby-bindings'.

"Unfortunately" e.g. emacs/Makefile.local also adds prerequisities for all:
-- and with those building emacs/notmuch-version.el fails
(I don't understand why, but it does). Setting all these prerequisities
could be ifneq()'d out but that is just complexity.


So, for developer convenience we can add cleanup speedup for
`make clean`, `make distclean` (and `make dataclean` if desired),
and user has to explicitly execute e.g. `make distclean && make all`
to do this kind of multiple targets...

Tomi

PS: I'd be interested to know whether it is possible to
`make distclean all` with autoconf-generated Makefiles...

[RFC PATCH 2] opportunistic support for make {, dist, data}clean {all, test, install}

[Tomi Ollila]

Makes make clean, make distclean and make dataclean faster if
Makefile.config exists but configure is newer.

Makes any combination of make {,dist,data}clean {all,test,install}
work (probably, some ad-hoc hand testing done).
---

Well, could not resist the temptation to dig further...

This may be as fragile as out-of-tree build, but should
not make the status quo worse.


Makefile                  | 15 +++++++++++++++
 Makefile.local            | 14 ++++++++++++--
 completion/Makefile.local |  2 +-
 emacs/Makefile.local      |  6 +++---
 lib/Makefile.local        |  2 +-
 test/Makefile.local       |  5 +++++
 6 files changed, 37 insertions(+), 7 deletions(-)

diff --git a/Makefile b/Makefile
index 0ef57fa..874a5ae 100644
--- a/Makefile
+++ b/Makefile
@@ -22,7 +22,22 @@ include Makefile.config
 global_deps = Makefile Makefile.config Makefile.local \
 	$(subdirs:%=%/Makefile) $(subdirs:%=%/Makefile.local)
 
+ifneq ($(filter clean distclean dataclean, $(word 1, $(MAKECMDGOALS))),)
+ifneq ($(filter-out all test install, $(wordlist 2, 99, $(MAKECMDGOALS))),)
+$(error With "$(word 1, $(MAKECMDGOALS))" only "all", "test" and "install" may work)
+endif
+WITH_CLEAN := yes
+else
+WITH_CLEAN := no
+endif
+
+# Potentially speedup make clean, distclean and dataclean ; avoid
+# re-creation of Makefile.config if it exists but configure is newer.
+ifeq ($(WITH_CLEAN),yes)
+Makefile.config: | $(srcdir)/configure
+else
 Makefile.config: $(srcdir)/configure
+endif
 ifeq ($(configure_options),)
 	@echo ""
 	@echo "Note: Calling ./configure with no command-line arguments. This is often fine,"
diff --git a/Makefile.local b/Makefile.local
index 3548ed9..a9e831a 100644
--- a/Makefile.local
+++ b/Makefile.local
@@ -1,7 +1,11 @@
 # -*- makefile -*-
 
 .PHONY: all
-all: notmuch notmuch-shared build-man ruby-bindings
+ifeq ($(WITH_CLEAN),yes)
+all:
+	$(MAKE) $@ configure_options=$(configure_options)
+else
+all: _all notmuch notmuch-shared build-man ruby-bindings
 ifeq ($(MAKECMDGOALS),)
 ifeq ($(shell cat .first-build-message 2>/dev/null),)
 	@NOTMUCH_FIRST_BUILD=1 $(MAKE) --no-print-directory all
@@ -16,6 +20,7 @@ ifeq ($(shell cat .first-build-message 2>/dev/null),)
 	@echo Printed > .first-build-message
 endif
 endif
+endif
 
 # Depend (also) on the file 'version'. In case of ifeq ($(IS_GIT),yes)
 # this file may already have been updated.
@@ -249,7 +254,11 @@ notmuch-shared: $(notmuch_client_modules) lib/$(LINKER_NAME)
 	$(call quiet,$(FINAL_NOTMUCH_LINKER) $(CFLAGS)) $(notmuch_client_modules) $(FINAL_NOTMUCH_LDFLAGS) -o $@
 
 .PHONY: install
-install: all install-man
+ifeq ($(WITH_CLEAN),yes)
+install:
+	$(MAKE) $@ configure_options=$(configure_options)
+else
+install: _install all install-man
 	mkdir -p "$(DESTDIR)$(prefix)/bin/"
 	install notmuch-shared "$(DESTDIR)$(prefix)/bin/notmuch"
 ifeq ($(MAKECMDGOALS), install)
@@ -273,6 +282,7 @@ ifeq ($(WITH_EMACS), 1)
 	@echo "the command \"M-x notmuch\" from within emacs."
 endif
 endif
+endif
 
 SRCS  := $(SRCS) $(notmuch_client_srcs)
 CLEAN := $(CLEAN) notmuch notmuch-shared $(notmuch_client_modules)
diff --git a/completion/Makefile.local b/completion/Makefile.local
index dfc1271..e0a1893 100644
--- a/completion/Makefile.local
+++ b/completion/Makefile.local
@@ -8,7 +8,7 @@ dir := completion
 bash_script := $(srcdir)/$(dir)/notmuch-completion.bash
 zsh_script := $(srcdir)/$(dir)/notmuch-completion.zsh
 
-install: install-$(dir)
+_install: install-$(dir)
 
 install-$(dir):
 	@echo $@
diff --git a/emacs/Makefile.local b/emacs/Makefile.local
index 040e64d..e6a2b1b 100644
--- a/emacs/Makefile.local
+++ b/emacs/Makefile.local
@@ -35,7 +35,7 @@ $(dir)/notmuch-pkg.el: $(srcdir)/$(dir)/notmuch-pkg.el.tmpl
 	@sed -e 's/%AG%/Generated file (from $(<F)) -- do not edit!/' \
 	     -e 's/%VERSION%/"$(ELPA_VERSION)"/' $< > $@
 
-all: $(dir)/notmuch-pkg.el
+_all: $(dir)/notmuch-pkg.el
 install-emacs: $(dir)/notmuch-pkg.el
 
 emacs_mua := $(dir)/notmuch-emacs-mua
@@ -88,11 +88,11 @@ notmuch-emacs-%.tar: ${elpa_sources}
 
 ifeq ($(WITH_EMACS),1)
 ifeq ($(HAVE_EMACS),1)
-all: $(emacs_bytecode)
+_all: $(emacs_bytecode)
 install-emacs: $(emacs_bytecode)
 endif
 
-install: install-emacs
+_install: install-emacs
 endif
 
 .PHONY: install-emacs
diff --git a/lib/Makefile.local b/lib/Makefile.local
index b77e578..479c60c 100644
--- a/lib/Makefile.local
+++ b/lib/Makefile.local
@@ -71,7 +71,7 @@ $(dir)/$(SONAME): $(dir)/$(LIBNAME)
 $(dir)/$(LINKER_NAME): $(dir)/$(SONAME)
 	ln -sf $(LIBNAME) $@
 
-install: install-$(dir)
+_install: install-$(dir)
 
 install-$(dir): $(dir)/$(LIBNAME)
 	mkdir -p "$(DESTDIR)$(libdir)/"
diff --git a/test/Makefile.local b/test/Makefile.local
index f8cf90d..fbb395f 100644
--- a/test/Makefile.local
+++ b/test/Makefile.local
@@ -59,6 +59,10 @@ TEST_BINARIES := $(TEST_BINARIES:.cc=)
 
 test-binaries: $(TEST_BINARIES)
 
+ifeq ($(WITH_CLEAN),yes)
+test:
+	$(MAKE) $@ configure_options=$(configure_options)
+else
 test:	all test-binaries
 ifeq ($V,)
 	@echo 'Use "$(MAKE) V=1" to print test headings and PASSing results.'
@@ -71,6 +75,7 @@ else
 	@${test_src_dir}/notmuch-test $(OPTIONS)
 endif
 endif
+endif
 
 check: test
 
-- 
2.9.3

[RFC PATCH 3] support goals after make *clean, *clean with current Makefile.config

[Tomi Ollila]

Makes make clean, make distclean and make dataclean faster if
Makefile.config exists but configure is newer.

After 1st target being *clean, any non-*clean targets may follow
it -- these are built in sub-make -- and when control returns
to main make, it just recognizes targers already made.
---

smoke-tested.

goal vs. target terminology is hard always

Makefile       | 15 +++++++++++++++
 Makefile.local | 27 ++++++++++++++++++++-------
 2 files changed, 35 insertions(+), 7 deletions(-)

diff --git a/Makefile b/Makefile
index 0ef57fa..7540903 100644
--- a/Makefile
+++ b/Makefile
@@ -22,7 +22,22 @@ include Makefile.config
 global_deps = Makefile Makefile.config Makefile.local \
 	$(subdirs:%=%/Makefile) $(subdirs:%=%/Makefile.local)
 
+ifneq ($(filter clean distclean dataclean, $(word 1, $(MAKECMDGOALS))),)
+ifneq ($(filter clean distclean dataclean, $(wordlist 2, 99, $(MAKECMDGOALS))),)
+$(error With "$(word 1, $(MAKECMDGOALS))" no further *clean targets works)
+endif
+WITH_CLEAN := yes
+else
+WITH_CLEAN := no
+endif
+
+# Potentially speedup make clean, distclean and dataclean ; avoid
+# re-creation of Makefile.config if it exists but configure is newer.
+ifeq ($(WITH_CLEAN),yes)
+Makefile.config: | $(srcdir)/configure
+else
 Makefile.config: $(srcdir)/configure
+endif
 ifeq ($(configure_options),)
 	@echo ""
 	@echo "Note: Calling ./configure with no command-line arguments. This is often fine,"
diff --git a/Makefile.local b/Makefile.local
index 3548ed9..e33af45 100644
--- a/Makefile.local
+++ b/Makefile.local
@@ -198,18 +198,31 @@ quiet ?= $($(shell echo $1 | sed -e s'/ .*//'))
 	@mkdir -p $(patsubst %/.,%,.deps/$(@D))
 	$(call quiet,CC $(CPPFLAGS) $(CFLAGS)) -c $(FINAL_CFLAGS) $< -o $@ -MD -MP -MF .deps/$*.d
 
-.PHONY : clean
-clean:
-	rm -rf $(CLEAN); rm -rf .deps
+.PHONY: _clean
+_clean:
+	rm -rf $(CLEAN)
+	rm -rf .deps
 
-.PHONY: distclean
-distclean: clean
+.PHONY: _distclean
+_distclean: _clean
 	rm -rf $(DISTCLEAN)
 
-.PHONY: dataclean
-dataclean: distclean
+.PHONY: _dataclean
+_dataclean: _distclean
 	rm -rf $(DATACLEAN)
 
+clean: _clean
+distclean: _distclean
+dataclean: _dataclean
+
+ifeq ($(WITH_CLEAN),yes)
+ifneq ($(word 2, $(MAKECMDGOALS)),)
+clean distclean dataclean: goals_after_clean
+goals_after_clean:
+	$(MAKE) $(wordlist 2, 99, $(MAKECMDGOALS)) configure_options=$(configure_options)
+endif
+endif
+
 notmuch_client_srcs =		\
 	command-line-arguments.c\
 	debugger.c		\
-- 
2.9.3

[RFC PATCH 5] support make goals after initial *clean goal(s)

[Tomi Ollila]

Now make goal combinations starting with *clean goals,
ending with *clean coals, and having non-*clean goals in between
should work. What does not work is non-clean -- clean -- non-clean
goal combination.
Also, initial *clean goals do not request re-creation of
Makefile.config if it exists and ./configure is newer.
---

If someone did not lost count, those can notice that I did not send
patch #4. it was quick fix for #3 -- but both had problem (which I
could not test) that phony/targets that do not stamp itself to the
filesystem would be run multiple time in #3.

Now, this is even simpler, more robust and everything -- just
that some more testing is needed (if not for anything else, for
verifying that my claims in the commit message are accurate).

and now I also got the difference between goals and targets.
goals are those which user give from command line (or default,
if none) -- targets are those defined in the makefiles, which
defines how to reach these goals.

 Makefile | 25 +++++++++++++++++++++++++
 1 file changed, 25 insertions(+)

diff --git a/Makefile b/Makefile
index 0ef57fa..ffab290 100644
--- a/Makefile
+++ b/Makefile
@@ -22,7 +22,30 @@ include Makefile.config
 global_deps = Makefile Makefile.config Makefile.local \
 	$(subdirs:%=%/Makefile) $(subdirs:%=%/Makefile.local)
 
+INCLUDE_MORE := yes
+ifneq ($(filter clean distclean dataclean, $(word 1, $(MAKECMDGOALS))),)
+CLEAN_GOAL := $(word 1, $(MAKECMDGOALS))
+
+ifneq ($(word 2, $(MAKECMDGOALS)),)
+INCLUDE_MORE := no
+$(CLEAN_GOAL): make_in_parts
+.PHONY: make_in_parts
+make_in_parts:
+	$(MAKE) $(CLEAN_GOAL)
+	$(MAKE) $(wordlist 2, 99, $(MAKECMDGOALS)) configure_options=$(configure_options)
+endif
+
+else
+CLEAN_GOAL :=
+endif
+
+# Potentially speedup make clean, distclean and dataclean ; avoid
+# re-creation of Makefile.config if it exists but configure is newer.
+ifneq ($(CLEAN_GOAL),)
+Makefile.config: | $(srcdir)/configure
+else
 Makefile.config: $(srcdir)/configure
+endif
 ifeq ($(configure_options),)
 	@echo ""
 	@echo "Note: Calling ./configure with no command-line arguments. This is often fine,"
@@ -33,9 +56,11 @@ ifeq ($(configure_options),)
 endif
 	$(srcdir)/configure $(configure_options)
 
+ifeq ($(INCLUDE_MORE),yes)
 # runtime variable definitions available in all subdirs
 include $(srcdir)/Makefile.global
 # Finally, include all of the Makefile.local fragments where all the
 # real work is done.
 
 include $(subdirs:%=%/Makefile.local) Makefile.local
+endif
-- 
2.9.3

[RFC PATCH 6] support make goals after initial {'', dist, data}clean goal(s)

[Tomi Ollila]

Now make goal combinations starting with *clean goals,
ending with *clean coals, and having non-*clean goals in between
should work. What does not expected to work are
non-*clean - *clean - *non-clean goal combinations.

Also, if first goals are *clean goals, re-creation of Makefile.config
is inhibited when Makefile.config exists and ./configure is newer.
---

Now there is more probability I don't figure out anything that
could be improved (but, if things go as usual, i know that
tomorrow morning ;)

 Makefile | 30 ++++++++++++++++++++++++++++++
 1 file changed, 30 insertions(+)

diff --git a/Makefile b/Makefile
index 0ef57fa..d2010fe 100644
--- a/Makefile
+++ b/Makefile
@@ -22,7 +22,35 @@ include Makefile.config
 global_deps = Makefile Makefile.config Makefile.local \
 	$(subdirs:%=%/Makefile) $(subdirs:%=%/Makefile.local)
 
+INCLUDE_MORE := yes
+ifneq ($(filter clean distclean dataclean, $(word 1, $(MAKECMDGOALS))),)
+CLEAN_GOAL := $(word 1, $(MAKECMDGOALS))
+
+# If there are more goals following CLEAN_GOAL, run $(MAKE)s in parts.
+ifneq ($(word 2, $(MAKECMDGOALS)),)
+INCLUDE_MORE := no
+FOLLOWING_GOALS := $(wordlist 2, 99, $(MAKECMDGOALS))
+
+.PHONY: $(FOLLOWING_GOALS) make_in_parts
+$(FOLLOWING_GOALS):
+	@true
+$(CLEAN_GOAL): make_in_parts
+make_in_parts:
+	$(MAKE) $(CLEAN_GOAL)
+	$(MAKE) $(FOLLOWING_GOALS) configure_options="$(configure_options)"
+endif
+
+else
+CLEAN_GOAL :=
+endif
+
+# Potentially speedup make clean, distclean and dataclean ; avoid
+# re-creating Makefile.config if it exists but configure is newer.
+ifneq ($(CLEAN_GOAL),)
+Makefile.config: | $(srcdir)/configure
+else
 Makefile.config: $(srcdir)/configure
+endif
 ifeq ($(configure_options),)
 	@echo ""
 	@echo "Note: Calling ./configure with no command-line arguments. This is often fine,"
@@ -33,9 +61,11 @@ ifeq ($(configure_options),)
 endif
 	$(srcdir)/configure $(configure_options)
 
+ifeq ($(INCLUDE_MORE),yes)
 # runtime variable definitions available in all subdirs
 include $(srcdir)/Makefile.global
 # Finally, include all of the Makefile.local fragments where all the
 # real work is done.
 
 include $(subdirs:%=%/Makefile.local) Makefile.local
+endif
-- 
2.9.3

Re: [RFC PATCH 6] support make goals after initial {'', dist, data}clean goal(s)

[David Bremner]

Tomi Ollila <tomi.ollila@iki.fi> writes:

> Now make goal combinations starting with *clean goals,
> ending with *clean coals, and having non-*clean goals in between
> should work. What does not expected to work are
> non-*clean - *clean - *non-clean goal combinations.
>
> Also, if first goals are *clean goals, re-creation of Makefile.config
> is inhibited when Makefile.config exists and ./configure is newer.
> ---
>
> Now there is more probability I don't figure out anything that
> could be improved (but, if things go as usual, i know that
> tomorrow morning ;)

Hi Tomi;

No more improvements for this? Maybe we should call it non-RFC now?

d

Re: [RFC PATCH 6] support make goals after initial {'', dist, data}clean goal(s)

[Tomi Ollila]

On Thu, Mar 09 2017, David Bremner <david@tethera.net> wrote:

> Tomi Ollila <tomi.ollila@iki.fi> writes:
>
>> Now make goal combinations starting with *clean goals,
>> ending with *clean coals, and having non-*clean goals in between
>> should work. What does not expected to work are
>> non-*clean - *clean - *non-clean goal combinations.
>>
>> Also, if first goals are *clean goals, re-creation of Makefile.config
>> is inhibited when Makefile.config exists and ./configure is newer.
>> ---
>>
>> Now there is more probability I don't figure out anything that
>> could be improved (but, if things go as usual, i know that
>> tomorrow morning ;)
>
> Hi Tomi;
>
> No more improvements for this? Maybe we should call it non-RFC now?

This is as good as I can make it. Yes, I think it matured beyond the
RFC status so IMO it can be released for wider reviewer audience ;)

Tomi

>
> d

Re: [RFC PATCH 6] support make goals after initial {'', dist, data}clean goal(s)

[David Bremner]

Tomi Ollila <tomi.ollila@iki.fi> writes:

> Now make goal combinations starting with *clean goals,
> ending with *clean coals, and having non-*clean goals in between
> should work. What does not expected to work are
> non-*clean - *clean - *non-clean goal combinations.
>
> Also, if first goals are *clean goals, re-creation of Makefile.config
> is inhibited when Makefile.config exists and ./configure is newer.

It's a bit painful to read, but that's Makefiles for you I guess.

d

Re: [RFC PATCH 6] support make goals after initial {'', dist, data}clean goal(s)

[David Bremner]

Tomi Ollila <tomi.ollila@iki.fi> writes:

> Now make goal combinations starting with *clean goals,
> ending with *clean coals, and having non-*clean goals in between
> should work. What does not expected to work are
> non-*clean - *clean - *non-clean goal combinations.
>

pushed, already appreciating the saved step

d