mtime of installed files

mtime of installed files

[Ludovic Courtès]

Hello,

Background: Guile 1.9/2.0 has an auto-compilation feature whereby if a
source file has no corresponding object file in the search path, or if
the object file is older than the source file, then the source file is
automatically recompiled and stored in ~/.cache/guile/ccache.

Packages that use Guile can also choose to pre-compile all their source
files and install both the source and object files.  This saves the need
for users to auto-compile the source.  (This is what Guile does with its
own source files.)

For this to work, we need “make install” to guarantee the relation
mtime(installed-object) >= mtime(installed-source), assuming we have
mtime(builddir-object) >= mtime(srcdir-source), which will always be the
case unless the computer’s clock is skewed.

Do Automake-generated makefiles provide such a guarantee?  Regardless of
the ‘make’ implementation, OS, etc.?

Thanks,
Ludo’.

Re: mtime of installed files

[Ralf Wildenhues]

Hi Ludovic,

* Ludovic Courtès wrote on Tue, Jul 20, 2010 at 06:28:46PM CEST:
> Background: Guile 1.9/2.0 has an auto-compilation feature whereby if a
> source file has no corresponding object file in the search path, or if
> the object file is older than the source file, then the source file is
> automatically recompiled and stored in ~/.cache/guile/ccache.

I assume this part has nothing to do with Automake per se, right?

> Packages that use Guile can also choose to pre-compile all their source
> files and install both the source and object files.  This saves the need
> for users to auto-compile the source.  (This is what Guile does with its
> own source files.)

I assume this, too, is currently done without help of Automake, or make,
for that matter.

> For this to work, we need “make install” to guarantee the relation
> mtime(installed-object) >= mtime(installed-source), assuming we have
> mtime(builddir-object) >= mtime(srcdir-source), which will always be the
> case unless the computer’s clock is skewed.
> 
> Do Automake-generated makefiles provide such a guarantee?  Regardless of
> the ‘make’ implementation, OS, etc.?

This is actually a tough question; it has little to do with Automake,
much more with any or all of make, tar, cp, touch, install, libc, file
system, and the kernel.

`info Autoconf --index timestamp' leads to two nodes describing some of
the problems with time stamps.  The gist is that in the worst case, any
of the above tools independently may have full or only limited timestamp
resolution (nowadays that would mostly be either nanosecond or 1 second).
Even well-run software distributions without user-compiled tools
typically can't afford to (or don't) update all of these tools at once.

Tools with limited resolution are expected to cut off fractional bits,
but there has been at least one report on autotools lists (ping me if
you need a pointer) of tools that seem to round, which violates Posix
and leads to disaster.  I do not recollect how wide-spread they are.

Anyway, most of the time, these issues can be worked around, for example
by sleeping for a second before building builddir objects, or between
installing sources and installing object files, depending on whether
install -C is used or not.

Hope that helps.  It might be a bit tricky or ugly to actually put this
sleep command in a normal automake Makefile.am, so if you know what
exactly you need, we can create an example.

Cheers,
Ralf

Re: mtime of installed files

[Ludovic Courtès]

Hi Ralf,

Ralf Wildenhues <Ralf.Wildenhues@gmx.de> writes:

> * Ludovic Courtès wrote on Tue, Jul 20, 2010 at 06:28:46PM CEST:
>> Background: Guile 1.9/2.0 has an auto-compilation feature whereby if a
>> source file has no corresponding object file in the search path, or if
>> the object file is older than the source file, then the source file is
>> automatically recompiled and stored in ~/.cache/guile/ccache.
>
> I assume this part has nothing to do with Automake per se, right?

Right.

>> Packages that use Guile can also choose to pre-compile all their source
>> files and install both the source and object files.  This saves the need
>> for users to auto-compile the source.  (This is what Guile does with its
>> own source files.)
>
> I assume this, too, is currently done without help of Automake, or make,
> for that matter.

Correct.

>> For this to work, we need “make install” to guarantee the relation
>> mtime(installed-object) >= mtime(installed-source), assuming we have
>> mtime(builddir-object) >= mtime(srcdir-source), which will always be the
>> case unless the computer’s clock is skewed.
>> 
>> Do Automake-generated makefiles provide such a guarantee?  Regardless of
>> the ‘make’ implementation, OS, etc.?
>
> This is actually a tough question; it has little to do with Automake,
> much more with any or all of make, tar, cp, touch, install, libc, file
> system, and the kernel.
>
> `info Autoconf --index timestamp' leads to two nodes describing some of
> the problems with time stamps.  The gist is that in the worst case, any
> of the above tools independently may have full or only limited timestamp
> resolution (nowadays that would mostly be either nanosecond or 1 second).
> Even well-run software distributions without user-compiled tools
> typically can't afford to (or don't) update all of these tools at once.

Indeed.  Thanks for the pointer.

[...]

> Anyway, most of the time, these issues can be worked around, for example
> by sleeping for a second before building builddir objects, or between
> installing sources and installing object files, depending on whether
> install -C is used or not.

The latter would be faster IIUC (sleep 1 second instead of N seconds,
with N the number of source files.)

The ‘install’ command is chosen by Automake or the user-specified
$INSTALL, so we can’t really determine whether it uses ‘-C’, right?

> Hope that helps.  It might be a bit tricky or ugly to actually put this
> sleep command in a normal automake Makefile.am, so if you know what
> exactly you need, we can create an example.

A typical Makefile.am looks like this:

--8<---------------cut here---------------start------------->8---
dist_foobar_SOURCES = foo.scm bar.scm
nodist_foobar_DATA  = foo.go bar.go

.scm.go:
        guile-tools compile -o $@ $<
--8<---------------cut here---------------end--------------->8---

Actually we currently have this hook, which Andy added some time ago
(here $(moddir) contains installed source files and $(ccachedir) is for
installed object files):

--8<---------------cut here---------------start------------->8---
install-data-hook:
	@$(am__vpath_adj_setup) \
	list='$(nobase_mod_DATA)'; for p in $$list; do \
	  if test -f "$$p"; then d=; else d="$(srcdir)/"; fi; \
	  $(am__vpath_adj) \
	  echo " touch -r '$$d$$p' '$(DESTDIR)$(moddir)/$$f'"; \
	  touch -r "$$d$$p" "$(DESTDIR)$(moddir)/$$f"; \
	done
	@$(am__vpath_adj_setup) \
	list='$(nobase_ccache_DATA)'; for p in $$list; do \
	  if test -f "$$p"; then d=; else d="$(srcdir)/"; fi; \
	  $(am__vpath_adj) \
	  echo " touch -r '$$d$$p' '$(DESTDIR)$(ccachedir)/$$f'"; \
	  touch -r "$$d$$p" "$(DESTDIR)$(ccachedir)/$$f"; \
	done
--8<---------------cut here---------------end--------------->8---

IIUC it copies timestamps from $(srcdir) and $(builddir) to the
corresponding installed files.

It looks like this is all we need, right?

Problem is, each package that installs Guile object and source files
needs this hook...

Thanks,
Ludo’.

Re: mtime of installed files

[Ralf Wildenhues]

Hi Ludovic,

* Ludovic Courtès wrote on Wed, Jul 21, 2010 at 06:27:46PM CEST:
> Ralf Wildenhues writes:
> > * Ludovic Courtès wrote on Tue, Jul 20, 2010 at 06:28:46PM CEST:
> >> For this to work, we need “make install” to guarantee the relation
> >> mtime(installed-object) >= mtime(installed-source), assuming we have
> >> mtime(builddir-object) >= mtime(srcdir-source), which will always be the
> >> case unless the computer’s clock is skewed.
> >> 
> >> Do Automake-generated makefiles provide such a guarantee?  Regardless of
> >> the ‘make’ implementation, OS, etc.?

> > Anyway, most of the time, these issues can be worked around, for example
> > by sleeping for a second before building builddir objects, or between
> > installing sources and installing object files, depending on whether
> > install -C is used or not.
> 
> The latter would be faster IIUC (sleep 1 second instead of N seconds,
> with N the number of source files.)

I'm not sure I understand.  I didn't mean you should sleep separately
for each of the objects.  Sleeping just once overall should be enough.
This semantics should not (need to) depend on whether 'install -C' is
used or not.

> The ‘install’ command is chosen by Automake or the user-specified
> $INSTALL, so we can’t really determine whether it uses ‘-C’, right?

Typically, that is right; well, at least we don't set it, the user does.

I'm not sure how -C helps you, though: it just doesn't update the time
stamp of a file already installed with the same contents; there is no
relation between stamps of different installed files.

> > Hope that helps.  It might be a bit tricky or ugly to actually put this
> > sleep command in a normal automake Makefile.am, so if you know what
> > exactly you need, we can create an example.
> 
> A typical Makefile.am looks like this:
> 
> --8<---------------cut here---------------start------------->8---
> dist_foobar_SOURCES = foo.scm bar.scm
> nodist_foobar_DATA  = foo.go bar.go
> 
> .scm.go:
>         guile-tools compile -o $@ $<
> --8<---------------cut here---------------end--------------->8---
> 
> Actually we currently have this hook, which Andy added some time ago
> (here $(moddir) contains installed source files and $(ccachedir) is for
> installed object files):
> 
> --8<---------------cut here---------------start------------->8---
> install-data-hook:
> 	@$(am__vpath_adj_setup) \
> 	list='$(nobase_mod_DATA)'; for p in $$list; do \
> 	  if test -f "$$p"; then d=; else d="$(srcdir)/"; fi; \
> 	  $(am__vpath_adj) \
> 	  echo " touch -r '$$d$$p' '$(DESTDIR)$(moddir)/$$f'"; \
> 	  touch -r "$$d$$p" "$(DESTDIR)$(moddir)/$$f"; \
> 	done
> 	@$(am__vpath_adj_setup) \
> 	list='$(nobase_ccache_DATA)'; for p in $$list; do \
> 	  if test -f "$$p"; then d=; else d="$(srcdir)/"; fi; \
> 	  $(am__vpath_adj) \
> 	  echo " touch -r '$$d$$p' '$(DESTDIR)$(ccachedir)/$$f'"; \
> 	  touch -r "$$d$$p" "$(DESTDIR)$(ccachedir)/$$f"; \
> 	done
> --8<---------------cut here---------------end--------------->8---
> 
> IIUC it copies timestamps from $(srcdir) and $(builddir) to the
> corresponding installed files.
> 
> It looks like this is all we need, right?

The code depends on internal Automake APIs am__vpath_adj_setup and
am__vpath_adj so is in principle not safe against Automake upgrades.

For extensibility (the user might need an install-data-hook for other
reasons too) you should write it somehow like:

  install-data-hook: guile-install-data-hook
  guile-install-data-hook:
        ... commands ...

  .PHONY: guile-install-data-hook

because only one set of rule commands is allowed with a non-double-colon
rule.

Then, the code ensures installed files have the same stamps as the
uninstalled ones, but that is not actually what you need: you need
to have newer timestamps on the installed objects over the installed
sources.

Now I need to ask in more detail: do you actually need "newer" or just
"not older"?  Because if the latter, then you could just let the rule to
install $(nodist_foobar_DATA) depend on the rule to install
$(dist_foobar_SOURCES), the latter of which I assume you still need to
write, and hook to install-data-local.  Or you do something like
  mod_DATA = $(dist_foobar_SOURCES)
  ccache_DATA = $(nodist_foobar_DATA)

The install rules then still have internal names
(install-nodist_foobarDATA or install-ccacheDATA or so) but at least the
right thing happens, as long as the same $(INSTALL_DATA) command is used
in both cases.

And if you need "newer" then you could "sleep 1" as last bit of your
hand-written rule to install $(dist_foobar_SOURCES), or let the
install-ccacheDATA depend on a guile-install-sleep helper rule which
itself depends on the install-nodist_foobarDATA rule or so.
However, that doesn't help you when "install -C" is used.

Now, there's one more technical complication for adding the dependency:
automake will not put out its own install-ccacheDATA rule if you use it
as target anywhere (on the grounds that it's the only method to override
an automake-provided target), which you don't want here; you can trick
automake by assigning the name to some variable though:
  guile_install_ccacheDATA = install-ccacheDATA
  $(guile_install_ccacheDATA): install-modDATA

> Problem is, each package that installs Guile object and source files
> needs this hook...

Well, for that you could put the code into a guile-rules.am fragment and
advise your users to put something like

include $(top_srcdir)/build-aux/guile-rules.am

into their Makefile.am files that need it.

Cheers,
Ralf

Re: mtime of installed files

[Ludovic Courtès]

Hello!

Ralf Wildenhues <Ralf.Wildenhues@gmx.de> writes:

> Now I need to ask in more detail: do you actually need "newer" or just
> "not older"?  Because if the latter, then you could just let the rule to
> install $(nodist_foobar_DATA) depend on the rule to install
> $(dist_foobar_SOURCES), the latter of which I assume you still need to
> write, and hook to install-data-local.  Or you do something like
>   mod_DATA = $(dist_foobar_SOURCES)
>   ccache_DATA = $(nodist_foobar_DATA)
>
> The install rules then still have internal names
> (install-nodist_foobarDATA or install-ccacheDATA or so) but at least the
> right thing happens, as long as the same $(INSTALL_DATA) command is used
> in both cases.
>
> And if you need "newer" then you could "sleep 1" as last bit of your
> hand-written rule to install $(dist_foobar_SOURCES), or let the
> install-ccacheDATA depend on a guile-install-sleep helper rule which
> itself depends on the install-nodist_foobarDATA rule or so.
> However, that doesn't help you when "install -C" is used.
>
> Now, there's one more technical complication for adding the dependency:
> automake will not put out its own install-ccacheDATA rule if you use it
> as target anywhere (on the grounds that it's the only method to override
> an automake-provided target), which you don't want here; you can trick
> automake by assigning the name to some variable though:
>   guile_install_ccacheDATA = install-ccacheDATA
>   $(guile_install_ccacheDATA): install-modDATA

For the record, this was fixed along these lines.  The auto-compile
mtime checks were also changed to checked for >= instead of ==:

  http://git.savannah.gnu.org/cgit/guile.git/commit/?id=fefd60ba4ba751712c45c95362bbc2f858890678

Ralf: can you formally state that the rule names (‘install-ccacheDATA’,
etc.) are not going to change?  :-)

Thank you!

Ludo’.

Re: mtime of installed files

[Ralf Wildenhues]

Hi Ludo,

* Ludovic Courtès wrote on Tue, Jul 27, 2010 at 10:18:39AM CEST:
> Ralf Wildenhues writes:
> 
> >   guile_install_ccacheDATA = install-ccacheDATA
> >   $(guile_install_ccacheDATA): install-modDATA
> 
> For the record, this was fixed along these lines.  The auto-compile
> mtime checks were also changed to checked for >= instead of ==:
> 
>   http://git.savannah.gnu.org/cgit/guile.git/commit/?id=fefd60ba4ba751712c45c95362bbc2f858890678
> 
> Ralf: can you formally state that the rule names (‘install-ccacheDATA’,
> etc.) are not going to change?  :-)

No, sorry.  But I can state something that should be almost as good:
It's quite unlikely that these names will change, and if I ever do that
change, then it will be only for a very good reason, and at that point
I will either document the new names and/or provide another documented
method to specify ordering requirements on installation rules.

Hope that helps.

Cheers,
Ralf