Re: Generating compiled scm (.go) files as part of LilyPond build

[Ian Hulin <ian@hulin.org.uk>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 29/01/11 11:21, Andy Wingo wrote:
> Hi Ian,
> 
> [re-adding the list; please keep the list CC'd on all guile-related 
> topics]
> 
> On Fri 28 Jan 2011 22:37, Ian Hulin <ian@hulin.org.uk> writes:
> 
>> Are you going to fix the double extension bug for defaulted/cached 
>> output names? I.e. compiling c++.scm will produce an output
>> filename of c++.scm.go in the cache.
> 
> As far as I'm concerned this isn't a bug.  It only happens for writes
> to the fallback path.  Appending an extension instead of replacing
> an extension makes the algorithm more robust (e.g, doesn't confuse
> `foo' and `foo.scm'.
> 
It may boil down to a matter of taste, but I find double and triple
extensions on a filename intrinsically nasty.  I've normally come across
them before on Windows systems where a filename such as thing.htm.exe
usually means there's malware or a trojan or suchlike on your system.
See also comments below.
>>> Basically compiled-file-name doesn't do any path searches.  It
>>> simply computes a place in ~/.cache in which to cache the result
>>> of compiling FILE.
>>> 
>> Yuk! This means we will need to have custom routines in our code to
>> work out and specify the output path we want.
> 
> There is not a DWIM answer, for various reasons -- but the
> combinatorics between %load-path, %load-compiled-path,
> %load-extensions, %load-compiled-extensions, and the actual `load'
> procedure should give you an idea.  The autocompilation strategy is
> as good as we can get without you actually telling Guile where you
> want your files.
> 
>> It also means that autocompilation is a hindrance rather than a
>> help, since it hi-jacks the compiled files into the cache.
> 
> What is the problem with writing to the cache?  It is a cache.  It's 
> doing it's job.
> 
The problem is knowing where the cache is. For Lilypond, we need to have
a common root directory off of which we can hang the compiled files in
scm/out.

I've been able to find out from Guile V2.02 that the cache root can be
controlled by an environment variable XDG_CACHE_HOME, which is defaulted
by Guile to $HOME/.cache if not defined.  The documentation also says
that compiled files for will be placed in $XDG_CACHE_HOME/guile/ccache
but this is what it actually does:

ian@nanny-ogg:~/src/Guile/guile-2.0.2$ meta/guile -L $PWD
GNU Guile 2.0.2
Copyright (C) 1995-2011 Free Software Foundation, Inc.

Guile comes with ABSOLUTELY NO WARRANTY; for details type `,show w'.
This program is free software, and you are welcome to redistribute it
under certain conditions; type `,show c' for details.

scheme@(guile-user)> (load "testing.scm")
;;; note: auto-compilation is enabled, set GUILE_AUTO_COMPILE=0
;;;       or pass the --no-auto-compile argument to disable.
;;; compiling /home/ian/src/Guile/guile-2.0.2/testing.scm
;;; compiled
/home/ian/src/Guile/guile-2.0.2/cache/guile/ccache/2.0-LE-4-2.0/home/ian/src/Guile/guile-2.0.2/testing.scm.go
scheme@(guile-user)>

The output filename looks like it's derived like this

The first section of the path
"/home/ian/src/Guile/guile-2.0.2/cache"
is controlled by XDG_CACHE_HOME or defaulted, but instead of
$HOME/.cache it defaults to $PWD/cache here.

The first two sections of the path
/home/ian/src/Guile/guile-2.0.2/cache/guile/ccache/2.0-LE-4-2.0
is controlled by %compile-fallback-path using the value derived or
defaulted from $XDG_CACHE_HOME for the first section.

The third section of the path
/home/ian/src/Guile/guile-2.0.2/
is the absolute path of the incoming filename concatenated onto the
first two sections.  This is unhelpful for what we want to do in Lilypond.

If the first two sections are of the path point to a cache base
directory, at this point section three of the path should be derived
from the *relative* path in the incoming filename.

testing.scm.go
Lose the .scm bit, it also is unhelpful.

If the compiled-file-name used the incoming relative path for the third
section we could do the following in our initialization code before
calling guile ($LILY_DATADIR is the the root LilyPond directory).

1. Insert $LILY_DATADIR/scm at the head of %load-path and
$LILY_DATADIR/scm/out at the head of %load-compiled-path
2. setenv $XDG_CACHE_HOME to $LILY_DATADIR
3. set %compile-fallback-path to $LILY_DATADIR

If the double extension "feature" was also removed,

Then the equivalent of
(load "testing.scm") with auto-compilation on would generate

$LILY_DATADIR/scm/testing.go

It wouldn't give us _quite_ what we want as the compiled files would be
in the same directory as the source files, but we could live with that
if it allows us either to use auto-compile, or to rely in Guile's
in-built check of the the source and compiled file's modified times.

>>> Load (system base compile) then, no?
>>> 
>> OK, this is another V1.8/V2.0 conditional kludge we need to put in
>> the LilyPond initialization code.
> 
> Compiling files is a 2.0 thing, of course...
> 
>>>> It looks like (compiled-file-name) uses a hard-coded default
>>>> setting for %load-compiled-files to use as the root for cache.
>>> 
>>> It doesn't use that path at all.  It uses
>>> %compile-fallback-path.
>>> 
>> Is this now documented?
> 
> No.  Would you like to document it?
> 
> Things are not perfect as they are, but they get better by people 
> writing code and documentation.  Contributions will help us get
> there.
> 
Only if compiled-file-name does something we can use :-).

Cheers,
Ian


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQEcBAEBAgAGBQJOJYQ6AAoJEBqidDirZqAS7U8IAJxhji+H13gnYVLfRcUrfXOV
JNKSG/HKaZMK9L4PN8ykYghyEzCMdwWEENXKoNAQKKjLYQMsN77SfTHChPU0Hy4Z
jK+DlKzPpNiGLKHjpcW34e7lrFiYV0xgsp72GqAKhdfmPnhyA7fE0GKsHCbP2YTc
6/xYJxBtQnpP3KICOVVfiCty78pkeXVriUzyKZVf8LgJzL80us10OoOKZVjVYy3z
suHzKt/w/2K6iaAI6f48CgpDAab6ZYaO9WlGcmG19YURU5kztV9usCZzbdqnLJfy
InLqJhFro6o9G55dt1txh+QwgJ70axRZLqEzqBX2JmecvZnwINE1gbLC+TWXxqg=
=N4RP
-----END PGP SIGNATURE-----