Re: Generating compiled scm (.go) files as part of LilyPond build

[Andy Wingo <wingo@pobox.com>]

Greets,

> On Tue 19 Jul 2011 15:28:41 BST, Andy Wingo wrote:
>> On Tue 19 Jul 2011 15:18, Ian Hulin <ian@hulin.org.uk> writes:
>> 
>>> It may boil down to a matter of taste, but I find double and
>>> triple extensions on a filename intrinsically nasty.  I've normally
>>> come across them before on Windows systems where a filename such as
>>> thing.htm.exe usually means there's malware or a trojan or suchlike
>>> on your system. See also comments below.
>> 
>> Consider it an opaque key into a cache.  We could change in the
>> future to default to the SHA1 of the whole path.  That does have
>> some advantages regarding flattening the directory hierarchy,
>> resulting in fewer stat operations.
>> 
> Sorry, but how does this benefit LilyPond as a user of the Guile API?
> I'm used to viewing file extensions as an indication of the format of
> the file content. In our simple view scheme sources are *.scm, and
> compiled scheme files are *.go.

Not sure I understand here; LilyPond would benefit from such a scheme
due to having various operations be faster, and by continuing to not
have to worry about the form of these file names.

> Guile documentation implies something similar (Chap 4 Environment
> Variable sections on GUILE_AUTO_COMPILE and GUILE_LOAD_COMPILED_PATH)
>
> "If a compiled (‘.go’) file corresponding to a ‘.scm’ file is not found
> or is not newer than the ‘.scm’ file, the ‘.scm’ file will be compiled
> on the fly, and the
> resulting ‘.go’ file stored away. An advisory note will be printed on
> the console."
>
> "GUILE_LOAD_COMPILED_PATH
> This variable may be used to augment the path that is searched for
> compiled Scheme files (‘.go’ files) when loading"
>
> Neither of these sections leads the reader to expect an actual file name
> generated to be
> ;;; compiled
> /long/path/name/testing.scm.go
>                        ^^^^^^^

I think that we need to be clear here, that Guile really cannot commit
to the form of the names of auto-compiled files.  That is not part of
Guile's API.

The reason for a simple string-append is that it is very possible to
compile a file named /long/path/name/testing *and also* a file named
/long/path/name/testing.scm.  Appending a suffix is a simple and
effective means to ensuring a unique cache key.

>>> ian@nanny-ogg:~/src/Guile/guile-2.0.2$ meta/guile -L $PWD
>> 
>> This will set XDG_CACHE_HOME=${top_builddir}/cache.
>> 
> The Guile Reference Manual Chapter 4 p 38 specifically says this
> defaults to $HOME/cache

Not when running uninstalled, with meta/guile.  Many things are
different in that case.

>>> The problem is knowing where the cache is. For Lilypond, we need to
>>> have a common root directory off of which we can hang the compiled
>>> files in scm/out.
>> 
>> Why do you care?  (Honest question.)
>> 
> (Honest answer)
> Because if Lilypond is run up with the Guile default of --auto-compile
> it will check for and generate compiled files in a different place to
> where we want to have produced them at build-time.  Also once for a
> project like LilyPond these files would need to be in a shared directory
> on the file system. Relying on the default compiled file spec would have
> a set of cached files somewhere on each user's cache in their home
> directory.

I don't understand why this isn't working for you.  I think we need to
be very specific.  Let's look at am/guilec from Guile itself.  It has:

    # -*- makefile -*-
    GOBJECTS = $(SOURCES:%.scm=%.go)

    GUILE_WARNINGS = -Wunbound-variable -Warity-mismatch -Wformat

    moddir = $(pkgdatadir)/$(GUILE_EFFECTIVE_VERSION)/$(modpath)
    nobase_mod_DATA = $(SOURCES) $(NOCOMP_SOURCES)
    ccachedir = $(pkglibdir)/$(GUILE_EFFECTIVE_VERSION)/ccache/$(modpath)
    nobase_ccache_DATA = $(GOBJECTS)
    EXTRA_DIST = $(SOURCES) $(NOCOMP_SOURCES)
    ETAGS_ARGS = $(SOURCES) $(NOCOMP_SOURCES)

    CLEANFILES = $(GOBJECTS)

    # Make sure source files are installed first, so that the mtime of
    # installed compiled files is greater than that of installed source
    # files.  See
    # <http://lists.gnu.org/archive/html/guile-devel/2010-07/msg00125.html>
    # for details.
    guile_install_go_files = install-nobase_ccacheDATA
    $(guile_install_go_files): install-nobase_modDATA

    AM_V_GUILEC = $(AM_V_GUILEC_$(V))
    AM_V_GUILEC_ = $(AM_V_GUILEC_$(AM_DEFAULT_VERBOSITY))
    AM_V_GUILEC_0 = @echo "  GUILEC" $@;

    SUFFIXES = .scm .go
    .scm.go:
    	$(AM_V_GUILEC)GUILE_AUTO_COMPILE=0				\
    	$(top_builddir)/meta/uninstalled-env			\
    	guild compile $(GUILE_WARNINGS) -o "$@" "$<"

Then module/Makefile.am just has to:

  include $(top_srcdir)/am/guilec
  # We're at the root of the module hierarchy.
  modpath =
  SOURCES =					\
    ice-9/psyntax-pp.scm				\
    ice-9/boot-9.scm				\
    ...

That will result in .go files getting compiled for all Scheme files.
They are compiled in an "uninstalled env", which adds the srcdir to the
GUILE_LOAD_PATH, and the builddir to the GUILE_LOAD_COMPILED_PATH, while
things are being compiled.  The bit turning off auto-compilation isn't
needed for packages outside of Guile itself; Guile only needs it because
its compiler is written in Scheme.

The .go files are installed to the correct place.  You would probably
use $(datadir)/guile/site/$(modpath) instead.  The .go files would get
installed to
$(libdir)/guile/$(GUILE_EFFECTIVE_VERSION)/ccache/$(modpath).  They are
shared among users.  They will be seen as fresh; users will not cause
auto-compilation, except for Scheme executable scripts, which, since
they are not searched for in a path, don't have a place for their
compiled files to go.  But that is a minor concern compared to what
you're working with right now.

>> To me there are two cases:
>> 
>> 1) You compile the files yourself.  You either ensure that the .go 
>> files end up in the right places, or adjust 
>> GUILE_LOAD_COMPILED_PATH.
>> 
> But GUILE_LOAD_COMPILED_PATH setting won't prevent the vanilla
> compiled-file-name from checking in the wrong place for at compile time.

If Guile loads a file from the load path, and finds an up-to-date .go
file in the GUILE_LOAD_COMPILED_PATH, it will not look in the fallback
path.

> OK we can brew our own expectation of where the file will be, and
> provide our own ly:load and intelligence to pick up the compiled file,
> but we have to do this in conjunction with a
> scm_putenv("GUILE_AUTO_COMPILE=0") call in our initialization code.

You certainly might need to do this, in the end.  It's a possibility.
Hopefully not, though.

Does this clarify things for you?

Regards,

Andy
-- 
http://wingolog.org/