Generating compiled scm (.go) files as part of LilyPond build

Generating compiled scm (.go) files as part of LilyPond build

[Ian Hulin]

Firstly, apologies if there is anywhere obvious I've failed to look such
as guile and/or guile mailing list histories.

LilyPond has a large number of .scm files it used and loads into its own
'lily' module using an initialization scheme script lily.scm.  Lily.scm
is the one that does the heavy lifting loading the .scm files.

Our long-term aim, when we are able to move to using Guile V2.0 as an
infrastructure, is to byte-compile as many of these as possible during
the Lily build using something like.
$ guile-tool compile <scm file>
   --output-file=<compiled-scm-dir><.go file>

(In the following paras, <lilypond-root> is the run-time base directory
being used by LilyPond.)
At the moment, during Lily initialization we prefix the guile path
%load-path with <lilypond-root>/scm as this is where we keep all the
LilyPond-specific scm files. This works fine when interpreting
everything using V1.8.7.  It also works reasonably well with 1.9/2.0 if
we use AUTOCOMPILE, as then guile seems to keep a private cache of
.scm.go files in somewhere like:
/home/<username>/.cache/guile/ccache/2.0-R-LE-4/
  home/<username>//lilypond/scm.

When we ask to load the <lilypond-root>/scm/<file>,scm using
primitive-load-path, Guile does all its file date modification magic,
against the <file>.scm.go file in the cache to decide if it needs to
generate newer compiled file.

OK, here's the question:  if we decide not to rely on AUTOCOMPILE, and
we are able generate our own .go files in, say, <lilypond-root>/out/scm,
how do we get guile to use the compiled version
<lilypond-root>/out/scm/<file>.go in preference to a possibly
non-existent .scm.go file in the cache?  There are some hints of
configure/type variables LOAD_PATH and COMPILED_LOAD_PATH in NEWS, and I
hoped that there might me a %compiled-load-path run-time equivalent to
COMPILED_LOAD_PATH just like %load-path corresponds to LOAD_PATH, but it
doesn't exist as at V1.9.13.  If there was, we could possibly prefix
<lilypond-root>/out/scm to the %compiled-load-path in the same way we
fiddle with %load-path for the interpreter.  Of course, I'm open to any
more elegant solutions.

Thanks in advance for any help and/or suggestions,

Cheers,

Ian Hulin

Re: Generating compiled scm (.go) files as part of LilyPond build

[Neil Jerram]

Ian Hulin <ian@hulin.org.uk> writes:

> OK, here's the question:  if we decide not to rely on AUTOCOMPILE, and
> we are able generate our own .go files in, say, <lilypond-root>/out/scm,
> how do we get guile to use the compiled version
> <lilypond-root>/out/scm/<file>.go in preference to a possibly
> non-existent .scm.go file in the cache?  There are some hints of
> configure/type variables LOAD_PATH and COMPILED_LOAD_PATH in NEWS, and I
> hoped that there might me a %compiled-load-path run-time equivalent to
> COMPILED_LOAD_PATH just like %load-path corresponds to LOAD_PATH, but it
> doesn't exist as at V1.9.13.  If there was, we could possibly prefix
> <lilypond-root>/out/scm to the %compiled-load-path in the same way we
> fiddle with %load-path for the interpreter.  Of course, I'm open to any
> more elegant solutions.

From some code grepping, I think the answer is to use
(primitive-load-path ...), and that the path for compiled files needs to
be put in %load-compiled-path.  Would you like to try that?

If this turns out to be correct, I'll update the doc for
primitive-load-path to cover it - so please report back.

Regards,
        Neil

Re: Generating compiled scm (.go) files as part of LilyPond build

[Ian Hulin]

Hi Neil,
On 28/11/10 12:01, Neil Jerram wrote:
> Ian Hulin <ian@hulin.org.uk> writes:
> 
>> OK, here's the question:  if we decide not to rely on AUTOCOMPILE, and
>> we are able generate our own .go files in, say, <lilypond-root>/out/scm,
>> how do we get guile to use the compiled version
>> <lilypond-root>/out/scm/<file>.go in preference to a possibly
>> non-existent .scm.go file in the cache?  There are some hints of
>> configure/type variables LOAD_PATH and COMPILED_LOAD_PATH in NEWS, and I
>> hoped that there might me a %compiled-load-path run-time equivalent to
>> COMPILED_LOAD_PATH just like %load-path corresponds to LOAD_PATH, but it
>> doesn't exist as at V1.9.13.  If there was, we could possibly prefix
>> <lilypond-root>/out/scm to the %compiled-load-path in the same way we
>> fiddle with %load-path for the interpreter.  Of course, I'm open to any
>> more elegant solutions.
> 
> From some code grepping, I think the answer is to use
> (primitive-load-path ...), and that the path for compiled files needs to
> be put in %load-compiled-path.  Would you like to try that?

This looks useful, together with the $GUILE_LOAD_COMPILED_PATH thingy
Andy mentioned (presumably something we can generate in our config
settings?

However, the code that loads in our .scm files also uses
%search-load-path. Will this find things in %load-compiled-path as well
as %load-path in V2.0?

> 
> If this turns out to be correct, I'll update the doc for
> primitive-load-path to cover it - so please report back.
> 
Working . . .

Cheers,
Ian Hulin

Re: Generating compiled scm (.go) files as part of LilyPond build

[Andy Wingo]

On Sat 27 Nov 2010 18:42, Ian Hulin <ian@hulin.org.uk> writes:

> Our long-term aim, when we are able to move to using Guile V2.0 as an
> infrastructure, is to byte-compile as many of these as possible during
> the Lily build using something like.
> $ guile-tool compile <scm file>
>    --output-file=<compiled-scm-dir><.go file>

Ralf has promised to add Guile 2.0 support to automake, so this should
be partly automated.

> At the moment, during Lily initialization we prefix the guile path
> %load-path with <lilypond-root>/scm as this is where we keep all the
> LilyPond-specific scm files.

There is also $GUILE_LOAD_COMPILED_PATH in 2.0, which you can set to the
proper path in your $builddir... which, ummm, doesn't appear to be
documented. Sorry about that. Want to patch our docs? :)

Ciao,

Andy
-- 
http://wingolog.org/

Re: Generating compiled scm (.go) files as part of LilyPond build

[Ian Hulin]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi Andy,
On 29/11/10 21:17, Andy Wingo wrote:
> On Sat 27 Nov 2010 18:42, Ian Hulin <ian@hulin.org.uk> writes:
> 
>> Our long-term aim, when we are able to move to using Guile V2.0 as an
>> infrastructure, is to byte-compile as many of these as possible during
>> the Lily build using something like.
>> $ guile-tool compile <scm file>
>>    --output-file=<compiled-scm-dir><.go file>
> 
> Ralf has promised to add Guile 2.0 support to automake, so this should
> be partly automated.
> 
>> At the moment, during Lily initialization we prefix the guile path
>> %load-path with <lilypond-root>/scm as this is where we keep all the
>> LilyPond-specific scm files.
> 
> There is also $GUILE_LOAD_COMPILED_PATH in 2.0, which you can set to the
> proper path in your $builddir... which, ummm, doesn't appear to be
> documented. Sorry about that. Want to patch our docs? :)

It doesn't appear to work using 1.9.13:
(PWD is /home/ian/lilypond)
ian@greebo$ echo $GUILE_LOAD_COMPILED_PATH/home/ian/lilypond/scm/out
ian@greebo$ ls $GUILE_LOAD_COMPILED_PATH
dummy.dep
ian@greebo$ guile-tools compile scm/lily-library.scm
wrote
`/home/ian/.cache/guile/ccache/2.0-0.S-LE-4/home/ian/lilypond/scm/lily-library.scm.go'
ian@greebo$

Cheers,
Ian

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQEcBAEBAgAGBQJM9rxEAAoJEBqidDirZqASGEMIAJnbJDLwp/Gv/ORrEnyB+rSp
aG/Wa1UC6nrjcAPH2tJ8+410RGMHEDp5YWagSP37JEKBqH6sjJTpZ4cXbUaUyJCO
+Fktd04uL2yr48wZzi71fjUXJOjNSe0WtJ4DqokbJ75jESj1nNLmKkTr+OJhPBvU
Z18NejAWWT8Tsob8nIJfyPnfQQ7tB55aUsbjvF3VvcNCAJ59i/CS8JZ0dCHZPueW
ARHbzL8raLCLBk1/A9kNUpCy8Gl5pjOkBWe4WS2QJwY6ObWyAwLUrUPeW3U4sXYB
TB67DzKfNQTth7+xiB5+gQNHNLaQVSbjpaGy2Vlv9eL7p648p2OXSFULgGrKhZk=
=NNrA
-----END PGP SIGNATURE-----

Re: Generating compiled scm (.go) files as part of LilyPond build

[Andy Wingo]

On Wed 01 Dec 2010 22:21, Ian Hulin <ian@hulin.org.uk> writes:

> ian@greebo$ guile-tools compile scm/lily-library.scm
> wrote
> `/home/ian/.cache/guile/ccache/2.0-0.S-LE-4/home/ian/lilypond/scm/lily-library.scm.go'

You will want to set the -o option, as in GCC. See guile-tools compile
--help.

Cheers,

Andy
-- 
http://wingolog.org/

Re: Generating compiled scm (.go) files as part of LilyPond build

[Ian Hulin]

Hi Andy,
On 01/12/10 22:08, Andy Wingo wrote:
> On Wed 01 Dec 2010 22:21, Ian Hulin <ian@hulin.org.uk> writes:
> 
>> ian@greebo$ guile-tools compile scm/lily-library.scm
>> wrote
>> `/home/ian/.cache/guile/ccache/2.0-0.S-LE-4/home/ian/lilypond/scm/lily-library.scm.go'
> 
> You will want to set the -o option, as in GCC. See guile-tools compile
> --help.
> 

Thanks for the work-round, but I think there are problems with
compiled-file-name, which in turn affect guile-tool compile,
load-from-path and what happens when autocompile is in force.

Documentation for compiled-file-name
"— Scheme Procedure: compiled-file-name file

    Compute an appropriate name for a compiled version of a Scheme file
named file.

    Usually, the result will be the original file name with the .scm
suffix replaced with .go, but the exact behavior depends on the contents
of the %load-extensions and %load-compiled-extensions lists. "

1. compiled-file-name doesn't respect any settings of
%load-compiled-path (which could be picked up from
GUILE_LOAD_COMPILED_PATH setting.

2.  compiled-file-name doesn't replace a .scm source extension with the
default .go extension - it appends it to the complete filename.

3. compiled-file-name isn't available at the API until either you've
used compile-file or else explicitly loaded
module (system base compile).

ian@greebo:~/lilypond$ echo $GUILE_LOAD_COMPILED_PATH
/home/ian/lilypond/scm/out
ian@greebo:~/lilypond$ian@greebo:~/lilypond$ guile -L /home/ian/lilypond/scm
GNU Guile 1.9.14
Copyright (C) 1995-2010 Free Software Foundation, Inc.

Guile comes with ABSOLUTELY NO WARRANTY; for details type `,show w'.
This program is free software, and you are welcome to redistribute it
under certain conditions; type `,show c' for details.

Enter `,help' for help.
scheme@(guile-user)> ,use (ice-9 readline)
scheme@(guile-user)> (activate-readline)
scheme@(guile-user)> ,use (system base compile)
scheme@(guile-user)> %load-compiled-path
$1 = ("/home/ian/lilypond/scm/out" "/usr/local/lib/guile/2.0/ccache")
scheme@(guile-user)> (compiled-file-name (%search-load-path "c++.scm"))
$1 =
"/home/ian/.cache/guile/ccache/2.0-0.T-LE-4/home/ian/lilypond/scm/c++.scm.go"
scheme@(guile-user)>

It looks like (compiled-file-name) uses a hard-coded default setting for
%load-compiled-files to use as the root for cache.

For LilyPond purposes it would be preferable if (compiled-file-name)
worked like this (this is only a prototype suggestion):

(define (ly-compiled-file-name filename)
	(let* ( (pathdef (car %load-compiled-files))
		(sextdef (car %load-extensions))
		(oextdef (car %load-compiled-extensions))
		(path (dirname filename))
		(outname (basename filename sextdef)))
	(if (equal? path "." )
		(set! path pathdef))
	(string-append path "/" outname oextdef)))

Cheers and Happy New Year,

Ian Hulin

Re: Generating compiled scm (.go) files as part of LilyPond build

[Andy Wingo]

Hi Ian,

On Thu 30 Dec 2010 12:43, Ian Hulin <ian@hulin.org.uk> writes:

> On 01/12/10 22:08, Andy Wingo wrote:
>> On Wed 01 Dec 2010 22:21, Ian Hulin <ian@hulin.org.uk> writes:
>> 
>>> ian@greebo$ guile-tools compile scm/lily-library.scm
>>> wrote
>>> `/home/ian/.cache/guile/ccache/2.0-0.S-LE-4/home/ian/lilypond/scm/lily-library.scm.go'
>> 
>> You will want to set the -o option, as in GCC. See guile-tools compile
>> --help.
>
> Thanks for the work-round, but I think there are problems with
> compiled-file-name, which in turn affect guile-tool compile,
> load-from-path and what happens when autocompile is in force.
>
> Documentation for compiled-file-name
> "— Scheme Procedure: compiled-file-name file
>
>     Compute an appropriate name for a compiled version of a Scheme file
> named file.
>
>     Usually, the result will be the original file name with the .scm
> suffix replaced with .go, but the exact behavior depends on the contents
> of the %load-extensions and %load-compiled-extensions lists. "

This documentation was out-of-date.  I have just committed updates to
master, which are not very good, but at least more truthy.

Basically compiled-file-name doesn't do any path searches.  It simply
computes a place in ~/.cache in which to cache the result of compiling
FILE.

As the comment in compile.scm says:

;;; This function is among the trickiest I've ever written. I tried many
;;; variants. In the end, simple is best, of course.
;;;
;;; After turning this around a number of times, it seems that the the
;;; desired behavior is that .go files should exist in a path, for
;;; searching. That is orthogonal to this function. For writing .go
;;; files, either you know where they should go, in which case you tell
;;; compile-file explicitly, as in the srcdir != builddir case; or you
;;; don't know, in which case this function is called, and we just put
;;; them in your own ccache dir in ~/.guile-ccache.

> 3. compiled-file-name isn't available at the API until either you've
> used compile-file or else explicitly loaded
> module (system base compile).

Load (system base compile) then, no?

> It looks like (compiled-file-name) uses a hard-coded default setting for
> %load-compiled-files to use as the root for cache.

It doesn't use that path at all.  It uses %compile-fallback-path.

Cheers,

Andy
-- 
http://wingolog.org/

Re: Generating compiled scm (.go) files as part of LilyPond build

[Andy Wingo]

Hi Ian,

[re-adding the list; please keep the list CC'd on all guile-related
topics]

On Fri 28 Jan 2011 22:37, Ian Hulin <ian@hulin.org.uk> writes:

> Are you going to fix the double extension bug for defaulted/cached
> output names? I.e. compiling c++.scm will produce an output filename of
> c++.scm.go in the cache.

As far as I'm concerned this isn't a bug.  It only happens for writes to
the fallback path.  Appending an extension instead of replacing an
extension makes the algorithm more robust (e.g, doesn't confuse `foo'
and `foo.scm'.

>> Basically compiled-file-name doesn't do any path searches.  It simply
>> computes a place in ~/.cache in which to cache the result of compiling
>> FILE.
>> 
> Yuk! This means we will need to have custom routines in our code to work
> out and specify the output path we want.

There is not a DWIM answer, for various reasons -- but the combinatorics
between %load-path, %load-compiled-path, %load-extensions,
%load-compiled-extensions, and the actual `load' procedure should give
you an idea.  The autocompilation strategy is as good as we can get
without you actually telling Guile where you want your files.

> It also means that autocompilation is a hindrance rather than a help,
> since it hi-jacks the compiled files into the cache.

What is the problem with writing to the cache?  It is a cache.  It's
doing it's job.

>> Load (system base compile) then, no?
>> 
> OK, this is another V1.8/V2.0 conditional kludge we need to put in the
> LilyPond initialization code.

Compiling files is a 2.0 thing, of course...

>>> It looks like (compiled-file-name) uses a hard-coded default setting for
>>> %load-compiled-files to use as the root for cache.
>> 
>> It doesn't use that path at all.  It uses %compile-fallback-path.
>> 
> Is this now documented?

No.  Would you like to document it?

Things are not perfect as they are, but they get better by people
writing code and documentation.  Contributions will help us get there.

Thanks,

Andy
-- 
http://wingolog.org/

Re: Generating compiled scm (.go) files as part of LilyPond build

[Ian Hulin]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 29/01/11 11:21, Andy Wingo wrote:
> Hi Ian,
> 
> [re-adding the list; please keep the list CC'd on all guile-related 
> topics]
> 
> On Fri 28 Jan 2011 22:37, Ian Hulin <ian@hulin.org.uk> writes:
> 
>> Are you going to fix the double extension bug for defaulted/cached 
>> output names? I.e. compiling c++.scm will produce an output
>> filename of c++.scm.go in the cache.
> 
> As far as I'm concerned this isn't a bug.  It only happens for writes
> to the fallback path.  Appending an extension instead of replacing
> an extension makes the algorithm more robust (e.g, doesn't confuse
> `foo' and `foo.scm'.
> 
It may boil down to a matter of taste, but I find double and triple
extensions on a filename intrinsically nasty.  I've normally come across
them before on Windows systems where a filename such as thing.htm.exe
usually means there's malware or a trojan or suchlike on your system.
See also comments below.
>>> Basically compiled-file-name doesn't do any path searches.  It
>>> simply computes a place in ~/.cache in which to cache the result
>>> of compiling FILE.
>>> 
>> Yuk! This means we will need to have custom routines in our code to
>> work out and specify the output path we want.
> 
> There is not a DWIM answer, for various reasons -- but the
> combinatorics between %load-path, %load-compiled-path,
> %load-extensions, %load-compiled-extensions, and the actual `load'
> procedure should give you an idea.  The autocompilation strategy is
> as good as we can get without you actually telling Guile where you
> want your files.
> 
>> It also means that autocompilation is a hindrance rather than a
>> help, since it hi-jacks the compiled files into the cache.
> 
> What is the problem with writing to the cache?  It is a cache.  It's 
> doing it's job.
> 
The problem is knowing where the cache is. For Lilypond, we need to have
a common root directory off of which we can hang the compiled files in
scm/out.

I've been able to find out from Guile V2.02 that the cache root can be
controlled by an environment variable XDG_CACHE_HOME, which is defaulted
by Guile to $HOME/.cache if not defined.  The documentation also says
that compiled files for will be placed in $XDG_CACHE_HOME/guile/ccache
but this is what it actually does:

ian@nanny-ogg:~/src/Guile/guile-2.0.2$ meta/guile -L $PWD
GNU Guile 2.0.2
Copyright (C) 1995-2011 Free Software Foundation, Inc.

Guile comes with ABSOLUTELY NO WARRANTY; for details type `,show w'.
This program is free software, and you are welcome to redistribute it
under certain conditions; type `,show c' for details.

scheme@(guile-user)> (load "testing.scm")
;;; note: auto-compilation is enabled, set GUILE_AUTO_COMPILE=0
;;;       or pass the --no-auto-compile argument to disable.
;;; compiling /home/ian/src/Guile/guile-2.0.2/testing.scm
;;; compiled
/home/ian/src/Guile/guile-2.0.2/cache/guile/ccache/2.0-LE-4-2.0/home/ian/src/Guile/guile-2.0.2/testing.scm.go
scheme@(guile-user)>

The output filename looks like it's derived like this

The first section of the path
"/home/ian/src/Guile/guile-2.0.2/cache"
is controlled by XDG_CACHE_HOME or defaulted, but instead of
$HOME/.cache it defaults to $PWD/cache here.

The first two sections of the path
/home/ian/src/Guile/guile-2.0.2/cache/guile/ccache/2.0-LE-4-2.0
is controlled by %compile-fallback-path using the value derived or
defaulted from $XDG_CACHE_HOME for the first section.

The third section of the path
/home/ian/src/Guile/guile-2.0.2/
is the absolute path of the incoming filename concatenated onto the
first two sections.  This is unhelpful for what we want to do in Lilypond.

If the first two sections are of the path point to a cache base
directory, at this point section three of the path should be derived
from the *relative* path in the incoming filename.

testing.scm.go
Lose the .scm bit, it also is unhelpful.

If the compiled-file-name used the incoming relative path for the third
section we could do the following in our initialization code before
calling guile ($LILY_DATADIR is the the root LilyPond directory).

1. Insert $LILY_DATADIR/scm at the head of %load-path and
$LILY_DATADIR/scm/out at the head of %load-compiled-path
2. setenv $XDG_CACHE_HOME to $LILY_DATADIR
3. set %compile-fallback-path to $LILY_DATADIR

If the double extension "feature" was also removed,

Then the equivalent of
(load "testing.scm") with auto-compilation on would generate

$LILY_DATADIR/scm/testing.go

It wouldn't give us _quite_ what we want as the compiled files would be
in the same directory as the source files, but we could live with that
if it allows us either to use auto-compile, or to rely in Guile's
in-built check of the the source and compiled file's modified times.

>>> Load (system base compile) then, no?
>>> 
>> OK, this is another V1.8/V2.0 conditional kludge we need to put in
>> the LilyPond initialization code.
> 
> Compiling files is a 2.0 thing, of course...
> 
>>>> It looks like (compiled-file-name) uses a hard-coded default
>>>> setting for %load-compiled-files to use as the root for cache.
>>> 
>>> It doesn't use that path at all.  It uses
>>> %compile-fallback-path.
>>> 
>> Is this now documented?
> 
> No.  Would you like to document it?
> 
> Things are not perfect as they are, but they get better by people 
> writing code and documentation.  Contributions will help us get
> there.
> 
Only if compiled-file-name does something we can use :-).

Cheers,
Ian


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQEcBAEBAgAGBQJOJYQ6AAoJEBqidDirZqAS7U8IAJxhji+H13gnYVLfRcUrfXOV
JNKSG/HKaZMK9L4PN8ykYghyEzCMdwWEENXKoNAQKKjLYQMsN77SfTHChPU0Hy4Z
jK+DlKzPpNiGLKHjpcW34e7lrFiYV0xgsp72GqAKhdfmPnhyA7fE0GKsHCbP2YTc
6/xYJxBtQnpP3KICOVVfiCty78pkeXVriUzyKZVf8LgJzL80us10OoOKZVjVYy3z
suHzKt/w/2K6iaAI6f48CgpDAab6ZYaO9WlGcmG19YURU5kztV9usCZzbdqnLJfy
InLqJhFro6o9G55dt1txh+QwgJ70axRZLqEzqBX2JmecvZnwINE1gbLC+TWXxqg=
=N4RP
-----END PGP SIGNATURE-----

Re: Generating compiled scm (.go) files as part of LilyPond build

[Andy Wingo]

Hi Ian,

On Tue 19 Jul 2011 15:18, Ian Hulin <ian@hulin.org.uk> writes:

> It may boil down to a matter of taste, but I find double and triple
> extensions on a filename intrinsically nasty.  I've normally come across
> them before on Windows systems where a filename such as thing.htm.exe
> usually means there's malware or a trojan or suchlike on your system.
> See also comments below.

Consider it an opaque key into a cache.  We could change in the future
to default to the SHA1 of the whole path.  That does have some
advantages regarding flattening the directory hierarchy, resulting in
fewer stat operations.

> ian@nanny-ogg:~/src/Guile/guile-2.0.2$ meta/guile -L $PWD

This will set XDG_CACHE_HOME=${top_builddir}/cache.

> The problem is knowing where the cache is. For Lilypond, we need to have
> a common root directory off of which we can hang the compiled files in
> scm/out.

Why do you care?  (Honest question.)

To me there are two cases:

 1) You compile the files yourself.  You either ensure that the .go
    files end up in the right places, or adjust
    GUILE_LOAD_COMPILED_PATH.

 2) You rely on autocompilation.  In this case there is no guarantee
    about where the files go, besides residing in the
    XDG_CACHE_DIR/guile.

Regards,

Andy
-- 
http://wingolog.org/

Re: Generating compiled scm (.go) files as part of LilyPond build

[Ian Hulin]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Tue 19 Jul 2011 15:28:41 BST, Andy Wingo wrote:
> Hi Ian,
> 
> On Tue 19 Jul 2011 15:18, Ian Hulin <ian@hulin.org.uk> writes:
> 
>> It may boil down to a matter of taste, but I find double and
>> triple extensions on a filename intrinsically nasty.  I've normally
>> come across them before on Windows systems where a filename such as
>> thing.htm.exe usually means there's malware or a trojan or suchlike
>> on your system. See also comments below.
> 
> Consider it an opaque key into a cache.  We could change in the
> future to default to the SHA1 of the whole path.  That does have
> some advantages regarding flattening the directory hierarchy,
> resulting in fewer stat operations.
> 
Sorry, but how does this benefit LilyPond as a user of the Guile API?
I'm used to viewing file extensions as an indication of the format of
the file content. In our simple view scheme sources are *.scm, and
compiled scheme files are *.go.

Guile documentation implies something similar (Chap 4 Environment
Variable sections on GUILE_AUTO_COMPILE and GUILE_LOAD_COMPILED_PATH)

"If a compiled (‘.go’) file corresponding to a ‘.scm’ file is not found
or is not newer than the ‘.scm’ file, the ‘.scm’ file will be compiled
on the fly, and the
resulting ‘.go’ file stored away. An advisory note will be printed on
the console."

"GUILE_LOAD_COMPILED_PATH
This variable may be used to augment the path that is searched for
compiled Scheme files (‘.go’ files) when loading"

Neither of these sections leads the reader to expect an actual file name
generated to be
;;; compiled
/long/path/name/testing.scm.go
                       ^^^^^^^
>> ian@nanny-ogg:~/src/Guile/guile-2.0.2$ meta/guile -L $PWD
> 
> This will set XDG_CACHE_HOME=${top_builddir}/cache.
> 
The Guile Reference Manual Chapter 4 p 38 specifically says this
defaults to $HOME/cache

>> The problem is knowing where the cache is. For Lilypond, we need to
>> have a common root directory off of which we can hang the compiled
>> files in scm/out.
> 
> Why do you care?  (Honest question.)
> 
(Honest answer)
Because if Lilypond is run up with the Guile default of --auto-compile
it will check for and generate compiled files in a different place to
where we want to have produced them at build-time.  Also once for a
project like LilyPond these files would need to be in a shared directory
on the file system. Relying on the default compiled file spec would have
a set of cached files somewhere on each user's cache in their home
directory.


> To me there are two cases:
> 
> 1) You compile the files yourself.  You either ensure that the .go 
> files end up in the right places, or adjust 
> GUILE_LOAD_COMPILED_PATH.
> 
But GUILE_LOAD_COMPILED_PATH setting won't prevent the vanilla
compiled-file-name from checking in the wrong place for at compile time.
OK we can brew our own expectation of where the file will be, and
provide our own ly:load and intelligence to pick up the compiled file,
but we have to do this in conjunction with a
scm_putenv("GUILE_AUTO_COMPILE=0") call in our initialization code.

> 2) You rely on autocompilation.  In this case there is no guarantee 
> about where the files go, besides residing in the 
> XDG_CACHE_DIR/guile.
> 
Looks like this is a no-no.  We can't control what the files are called,
or if they'd be recognized once they've been moved to their location
after /make install/ for LilyPond. Also there's the issue of embedded
Scheme statements in Lily source files (which could contain their own
(load "xxxx") statements).

It's a real shame because it looks like auto-compile is a feature
designed to make life easy for Guile users (including layered projects
like LilyPond). Unfortunately its default settings prevent us from using
it. :-(.

I've also got a question re the (include) procedure which Ludovic
mentioned in a post on guile-users.  I'll start a different thread for
that one.

Thanks for the help so far,

Cheers,
Ian

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQEcBAEBAgAGBQJOJg5cAAoJEBqidDirZqASMj0H/2jn/jiNG+MtT+YEc762baxL
8pMC4ijNMwWEnMwzuWm1LHhHiJqG5roxvinPoe5bjeC8RBNSMOW7SKGkJ+ao5pTK
vYfBm55WqdpAvAVQA9OaDkL6J33TEa6z1ioNYxj2yFOHHsj0HHVnhUM29RDNYgmw
9duK9lKtv37AGT0W9aj6bGx8FRGFUOFgRUnHMCtn7usE1DboXQgjaY3IQ46XYs6n
XGfuFz1JBXqhKEjjK7Fbu5M44A0eD2yWGJ01K3eVQqB1OmIrz7zy7wU495Wqd4im
/eop+BCJRnpRAdHIR0y/LgrrlPSpD+mulBk9BOuoAglXM6N8spaQQY/mj8MkrKU=
=igTg
-----END PGP SIGNATURE-----

Re: Generating compiled scm (.go) files as part of LilyPond build

[Andy Wingo]

Greets,

> On Tue 19 Jul 2011 15:28:41 BST, Andy Wingo wrote:
>> On Tue 19 Jul 2011 15:18, Ian Hulin <ian@hulin.org.uk> writes:
>> 
>>> It may boil down to a matter of taste, but I find double and
>>> triple extensions on a filename intrinsically nasty.  I've normally
>>> come across them before on Windows systems where a filename such as
>>> thing.htm.exe usually means there's malware or a trojan or suchlike
>>> on your system. See also comments below.
>> 
>> Consider it an opaque key into a cache.  We could change in the
>> future to default to the SHA1 of the whole path.  That does have
>> some advantages regarding flattening the directory hierarchy,
>> resulting in fewer stat operations.
>> 
> Sorry, but how does this benefit LilyPond as a user of the Guile API?
> I'm used to viewing file extensions as an indication of the format of
> the file content. In our simple view scheme sources are *.scm, and
> compiled scheme files are *.go.

Not sure I understand here; LilyPond would benefit from such a scheme
due to having various operations be faster, and by continuing to not
have to worry about the form of these file names.

> Guile documentation implies something similar (Chap 4 Environment
> Variable sections on GUILE_AUTO_COMPILE and GUILE_LOAD_COMPILED_PATH)
>
> "If a compiled (‘.go’) file corresponding to a ‘.scm’ file is not found
> or is not newer than the ‘.scm’ file, the ‘.scm’ file will be compiled
> on the fly, and the
> resulting ‘.go’ file stored away. An advisory note will be printed on
> the console."
>
> "GUILE_LOAD_COMPILED_PATH
> This variable may be used to augment the path that is searched for
> compiled Scheme files (‘.go’ files) when loading"
>
> Neither of these sections leads the reader to expect an actual file name
> generated to be
> ;;; compiled
> /long/path/name/testing.scm.go
>                        ^^^^^^^

I think that we need to be clear here, that Guile really cannot commit
to the form of the names of auto-compiled files.  That is not part of
Guile's API.

The reason for a simple string-append is that it is very possible to
compile a file named /long/path/name/testing *and also* a file named
/long/path/name/testing.scm.  Appending a suffix is a simple and
effective means to ensuring a unique cache key.

>>> ian@nanny-ogg:~/src/Guile/guile-2.0.2$ meta/guile -L $PWD
>> 
>> This will set XDG_CACHE_HOME=${top_builddir}/cache.
>> 
> The Guile Reference Manual Chapter 4 p 38 specifically says this
> defaults to $HOME/cache

Not when running uninstalled, with meta/guile.  Many things are
different in that case.

>>> The problem is knowing where the cache is. For Lilypond, we need to
>>> have a common root directory off of which we can hang the compiled
>>> files in scm/out.
>> 
>> Why do you care?  (Honest question.)
>> 
> (Honest answer)
> Because if Lilypond is run up with the Guile default of --auto-compile
> it will check for and generate compiled files in a different place to
> where we want to have produced them at build-time.  Also once for a
> project like LilyPond these files would need to be in a shared directory
> on the file system. Relying on the default compiled file spec would have
> a set of cached files somewhere on each user's cache in their home
> directory.

I don't understand why this isn't working for you.  I think we need to
be very specific.  Let's look at am/guilec from Guile itself.  It has:

    # -*- makefile -*-
    GOBJECTS = $(SOURCES:%.scm=%.go)

    GUILE_WARNINGS = -Wunbound-variable -Warity-mismatch -Wformat

    moddir = $(pkgdatadir)/$(GUILE_EFFECTIVE_VERSION)/$(modpath)
    nobase_mod_DATA = $(SOURCES) $(NOCOMP_SOURCES)
    ccachedir = $(pkglibdir)/$(GUILE_EFFECTIVE_VERSION)/ccache/$(modpath)
    nobase_ccache_DATA = $(GOBJECTS)
    EXTRA_DIST = $(SOURCES) $(NOCOMP_SOURCES)
    ETAGS_ARGS = $(SOURCES) $(NOCOMP_SOURCES)

    CLEANFILES = $(GOBJECTS)

    # Make sure source files are installed first, so that the mtime of
    # installed compiled files is greater than that of installed source
    # files.  See
    # <http://lists.gnu.org/archive/html/guile-devel/2010-07/msg00125.html>
    # for details.
    guile_install_go_files = install-nobase_ccacheDATA
    $(guile_install_go_files): install-nobase_modDATA

    AM_V_GUILEC = $(AM_V_GUILEC_$(V))
    AM_V_GUILEC_ = $(AM_V_GUILEC_$(AM_DEFAULT_VERBOSITY))
    AM_V_GUILEC_0 = @echo "  GUILEC" $@;

    SUFFIXES = .scm .go
    .scm.go:
    	$(AM_V_GUILEC)GUILE_AUTO_COMPILE=0				\
    	$(top_builddir)/meta/uninstalled-env			\
    	guild compile $(GUILE_WARNINGS) -o "$@" "$<"

Then module/Makefile.am just has to:

  include $(top_srcdir)/am/guilec
  # We're at the root of the module hierarchy.
  modpath =
  SOURCES =					\
    ice-9/psyntax-pp.scm				\
    ice-9/boot-9.scm				\
    ...

That will result in .go files getting compiled for all Scheme files.
They are compiled in an "uninstalled env", which adds the srcdir to the
GUILE_LOAD_PATH, and the builddir to the GUILE_LOAD_COMPILED_PATH, while
things are being compiled.  The bit turning off auto-compilation isn't
needed for packages outside of Guile itself; Guile only needs it because
its compiler is written in Scheme.

The .go files are installed to the correct place.  You would probably
use $(datadir)/guile/site/$(modpath) instead.  The .go files would get
installed to
$(libdir)/guile/$(GUILE_EFFECTIVE_VERSION)/ccache/$(modpath).  They are
shared among users.  They will be seen as fresh; users will not cause
auto-compilation, except for Scheme executable scripts, which, since
they are not searched for in a path, don't have a place for their
compiled files to go.  But that is a minor concern compared to what
you're working with right now.

>> To me there are two cases:
>> 
>> 1) You compile the files yourself.  You either ensure that the .go 
>> files end up in the right places, or adjust 
>> GUILE_LOAD_COMPILED_PATH.
>> 
> But GUILE_LOAD_COMPILED_PATH setting won't prevent the vanilla
> compiled-file-name from checking in the wrong place for at compile time.

If Guile loads a file from the load path, and finds an up-to-date .go
file in the GUILE_LOAD_COMPILED_PATH, it will not look in the fallback
path.

> OK we can brew our own expectation of where the file will be, and
> provide our own ly:load and intelligence to pick up the compiled file,
> but we have to do this in conjunction with a
> scm_putenv("GUILE_AUTO_COMPILE=0") call in our initialization code.

You certainly might need to do this, in the end.  It's a possibility.
Hopefully not, though.

Does this clarify things for you?

Regards,

Andy
-- 
http://wingolog.org/

Re: Generating compiled scm (.go) files as part of LilyPond build

[Ludovic Courtès]

Hi Ian,

Ian Hulin <ian@hulin.org.uk> skribis:

> On Tue 19 Jul 2011 15:28:41 BST, Andy Wingo wrote:

>> On Tue 19 Jul 2011 15:18, Ian Hulin <ian@hulin.org.uk> writes:

>>> The problem is knowing where the cache is. For Lilypond, we need to
>>> have a common root directory off of which we can hang the compiled
>>> files in scm/out.
>>
>> Why do you care?  (Honest question.)
>>
> (Honest answer)
> Because if Lilypond is run up with the Guile default of --auto-compile
> it will check for and generate compiled files in a different place to
> where we want to have produced them at build-time.

Would it work for you to have makefile rules to build/install .go files
where you want?

For Automake, something like this:
<http://git.savannah.gnu.org/cgit/guile-rpc.git/tree/modules/Makefile.am#n86>.

Thanks,
Ludo’.