Autocompilation/LilyPond

Autocompilation/LilyPond

[David Kastrup]

Hi,

with the stable release 2.16 of LilyPond looming around the corner, it
will become imminent soon to think about supporting Guile 2.0.

Previous attempts have mostly exploded around the problem that we have
something like

(for-each ly:load init-scheme-files)

in our lily.scm file, and the auto-compiler attempts to compile all of
those files independently as far as I understand.  Unfortunately, some
of them contain macro definitions that other files rely on.

Personally, I think it would make sense if we could get the autocompiler
to treat the whole blob of files as _one_ unit, and recompile the unit
if it gets out of date.  That would save us from trying to factor out
macro dependencies into separate files (and since our markup system
defines a macro for every markup function, and since the macros are
needed when building markups in Scheme, this is actually rather hard to
do).  You might say that it is LilyPond's own fault that it has reverted
a bit more to macro programming than feasible for its own good.

I would not actually say that you are wrong.  However, there is the
problem of lifting a whole bunch of working code base under active
development into the Guilev2 era, and if we could tackle design or
maldesign questions mostly independently and in bite-sized chunks rather
than humongous patches moving material around, this would help a lot in
getting Guilev2 support on track.

Suggestions?

-- 
David Kastrup

Re: Autocompilation/LilyPond

[Mark H Weaver]

David Kastrup <dak@gnu.org> writes:

> with the stable release 2.16 of LilyPond looming around the corner, it
> will become imminent soon to think about supporting Guile 2.0.
>
> Previous attempts have mostly exploded around the problem that we have
> something like
>
> (for-each ly:load init-scheme-files)
>
> in our lily.scm file, and the auto-compiler attempts to compile all of
> those files independently as far as I understand.  Unfortunately, some
> of them contain macro definitions that other files rely on.
>
> Personally, I think it would make sense if we could get the autocompiler
> to treat the whole blob of files as _one_ unit, and recompile the unit
> if it gets out of date.

I'm not sure that would help much.  There's a deeper problem that you
should be aware of.  In Guile 1.x, macro uses within procedures are not
expanded until the procedure is evaluated.  In Guile 2, macros are
expanded as soon as the procedure is defined, even if compilation is
turned off.  This means that functions can only use macros that were
previously defined.

For example, the following works in Guile 1.8 but not in Guile 2:

  (define (foo x) (bar x))
  (define-macro (bar x) `(quote ,x))

In Guile 2, you must put the definition of 'bar' before 'foo'.

So you'll need to load your code in the right order so that macros
always come before their uses.

One clarification is in order.  You might conclude from this that it is
not possible to define mutually-recursive macros, but that's not true.
For example, the following is fine:

(define-macro (beep x) `(boop ,x))
(define-macro (boop x) `(quote ,x))

That's because 'beep' doesn't use 'boop', it merely produces a use of
'boop' in its expansion.  'boop' is merely part of a quoted literal
within 'beep'.

   Regards,
     Mark

Re: Autocompilation/LilyPond

[David Kastrup]

Mark H Weaver <mhw@netris.org> writes:

> David Kastrup <dak@gnu.org> writes:
>
>> with the stable release 2.16 of LilyPond looming around the corner, it
>> will become imminent soon to think about supporting Guile 2.0.
>>
>> Previous attempts have mostly exploded around the problem that we have
>> something like
>>
>> (for-each ly:load init-scheme-files)
>>
>> in our lily.scm file, and the auto-compiler attempts to compile all
>> of those files independently as far as I understand.  Unfortunately,
>> some of them contain macro definitions that other files rely on.
>>
>> Personally, I think it would make sense if we could get the
>> autocompiler to treat the whole blob of files as _one_ unit, and
>> recompile the unit if it gets out of date.
>
> I'm not sure that would help much.  There's a deeper problem that you
> should be aware of.  In Guile 1.x, macro uses within procedures are
> not expanded until the procedure is evaluated.  In Guile 2, macros are
> expanded as soon as the procedure is defined, even if compilation is
> turned off.  This means that functions can only use macros that were
> previously defined.

I don't think that making this condition hold would be really hard.
LilyPond has a rather carefully selected load order in several stages,
so use-before-definition, whether in the context of macros or not,
should be more the exception than the rule, and only require smaller
rearrangements.

> One clarification is in order.  You might conclude from this that it
> is not possible to define mutually-recursive macros, but that's not
> true.

I don't think we use them, anyway.  Most problems are more due to small
oversights rather than maliciously clever overdesign.

-- 
David Kastrup

Re: Autocompilation/LilyPond

[Mark H Weaver]

David Kastrup <dak@gnu.org> writes:

> Mark H Weaver <mhw@netris.org> writes:
>
>> David Kastrup <dak@gnu.org> writes:
>>
>>> with the stable release 2.16 of LilyPond looming around the corner, it
>>> will become imminent soon to think about supporting Guile 2.0.
>>>
>>> Previous attempts have mostly exploded around the problem that we have
>>> something like
>>>
>>> (for-each ly:load init-scheme-files)
>>>
>>> in our lily.scm file, and the auto-compiler attempts to compile all
>>> of those files independently as far as I understand.  Unfortunately,
>>> some of them contain macro definitions that other files rely on.
>>>
>>> Personally, I think it would make sense if we could get the
>>> autocompiler to treat the whole blob of files as _one_ unit, and
>>> recompile the unit if it gets out of date.
>>
>> I'm not sure that would help much.  There's a deeper problem that you
>> should be aware of.  In Guile 1.x, macro uses within procedures are
>> not expanded until the procedure is evaluated.  In Guile 2, macros are
>> expanded as soon as the procedure is defined, even if compilation is
>> turned off.  This means that functions can only use macros that were
>> previously defined.
>
> I don't think that making this condition hold would be really hard.
> LilyPond has a rather carefully selected load order in several stages,
> so use-before-definition, whether in the context of macros or not,
> should be more the exception than the rule, and only require smaller
> rearrangements.

Excellent!  As long as you load everything in the right order, such that
macros are defined before they are used, I don't see why there should be
any other problems related to macros and compilation.

     Mark

Re: Autocompilation/LilyPond

[David Kastrup]

Mark H Weaver <mhw@netris.org> writes:

> David Kastrup <dak@gnu.org> writes:
>
>> Mark H Weaver <mhw@netris.org> writes:
>>
>>> David Kastrup <dak@gnu.org> writes:
>>>
>>>> with the stable release 2.16 of LilyPond looming around the corner, it
>>>> will become imminent soon to think about supporting Guile 2.0.
>>>>
>>>> Previous attempts have mostly exploded around the problem that we have
>>>> something like
>>>>
>>>> (for-each ly:load init-scheme-files)
>>>>
>>>> in our lily.scm file, and the auto-compiler attempts to compile all
>>>> of those files independently as far as I understand.  Unfortunately,
>>>> some of them contain macro definitions that other files rely on.
>>>>
>>>> Personally, I think it would make sense if we could get the
>>>> autocompiler to treat the whole blob of files as _one_ unit, and
>>>> recompile the unit if it gets out of date.
>>>
>>> I'm not sure that would help much.  There's a deeper problem that you
>>> should be aware of.  In Guile 1.x, macro uses within procedures are
>>> not expanded until the procedure is evaluated.  In Guile 2, macros are
>>> expanded as soon as the procedure is defined, even if compilation is
>>> turned off.  This means that functions can only use macros that were
>>> previously defined.
>>
>> I don't think that making this condition hold would be really hard.
>> LilyPond has a rather carefully selected load order in several stages,
>> so use-before-definition, whether in the context of macros or not,
>> should be more the exception than the rule, and only require smaller
>> rearrangements.
>
> Excellent!  As long as you load everything in the right order, such that
> macros are defined before they are used, I don't see why there should be
> any other problems related to macros and compilation.

Because the individual files are not independent from one another?
That's why I wrote:

    Personally, I think it would make sense if we could get the
    autocompiler to treat the whole blob of files as _one_ unit, and
    recompile the unit if it gets out of date.

-- 
David Kastrup

Re: Autocompilation/LilyPond

[Mark H Weaver]

David Kastrup <dak@gnu.org> writes:

> Mark H Weaver <mhw@netris.org> writes:
>
>> Excellent!  As long as you load everything in the right order, such that
>> macros are defined before they are used, I don't see why there should be
>> any other problems related to macros and compilation.
>
> Because the individual files are not independent from one another?
> That's why I wrote:
>
>     Personally, I think it would make sense if we could get the
>     autocompiler to treat the whole blob of files as _one_ unit, and
>     recompile the unit if it gets out of date.

There's no problem with them being dependent on one another.  When you
load a file, even with auto-compilation, the macro expander will make
use of whatever macros are already bound in the current module.  The
rest of the compiler sees only the output of the macro expander.

Try the following experiment: put "(define-macro (bar x) `(quote ,x))"
into "foo1.scm", and "(define (foo x) (bar x))" into "foo2.scm", and
then within a REPL type: (load "foo1.scm") (load "foo2.scm") and observe
that everything works as it should.

If you really want everything to be compiled as one unit, you can use
'include' (which acts essentially the same as #include in C), though
beware that Guile is not yet smart enough to auto-recompile when one of
the included files gets updated.

I don't see any compelling benefit to compiling everything as one unit,
but do as you wish :)

   Regards,
     Mark

Re: Autocompilation/LilyPond

[David Kastrup]

Mark H Weaver <mhw@netris.org> writes:

> David Kastrup <dak@gnu.org> writes:
>
>> Mark H Weaver <mhw@netris.org> writes:
>>
>>> Excellent!  As long as you load everything in the right order, such that
>>> macros are defined before they are used, I don't see why there should be
>>> any other problems related to macros and compilation.
>>
>> Because the individual files are not independent from one another?
>> That's why I wrote:
>>
>>     Personally, I think it would make sense if we could get the
>>     autocompiler to treat the whole blob of files as _one_ unit, and
>>     recompile the unit if it gets out of date.
>
> There's no problem with them being dependent on one another.  When you
> load a file, even with auto-compilation, the macro expander will make
> use of whatever macros are already bound in the current module.  The
> rest of the compiler sees only the output of the macro expander.
>
> Try the following experiment: put "(define-macro (bar x) `(quote ,x))"
> into "foo1.scm", and "(define (foo x) (bar x))" into "foo2.scm", and
> then within a REPL type: (load "foo1.scm") (load "foo2.scm") and observe
> that everything works as it should.
>
> If you really want everything to be compiled as one unit, you can use
> 'include' (which acts essentially the same as #include in C), though
> beware that Guile is not yet smart enough to auto-recompile when one of
> the included files gets updated.
>
> I don't see any compelling benefit to compiling everything as one unit,
> but do as you wish :)

How is this supposed to work for compiling and installing a package?
Basically,

make all
sudo make install

The usual case will be that the user calling lilypond will not have
write permission in the installed directories (and even if he did, like
when calling lilypond as root, lilypond should not stomp over the
installed files).

So what would make all do to generate one or more .go files?

-- 
David Kastrup

Re: Autocompilation/LilyPond

[Mark H Weaver]

David Kastrup <dak@gnu.org> writes:

> How is this supposed to work for compiling and installing a package?
> Basically,
>
> make all
> sudo make install
>
> The usual case will be that the user calling lilypond will not have
> write permission in the installed directories (and even if he did, like
> when calling lilypond as root, lilypond should not stomp over the
> installed files).
>
> So what would make all do to generate one or more .go files?

I'm not very familiar with the build system, so it would be great if
Ludovic or Andy could chime in here, but as I understand it, the way
it's meant to work is as follows:

Ideally, each file contains a single module, whose name matches its own
pathname relative to an element of the GUILE_LOAD_PATH, with a
(define-module ...) header at the top declaring its dependencies on
other modules (using #:use-module) and its exports (using #:export and
#:export-syntax).  See guile-2.0.5/modules/* for many examples, and
section 6.19.3 (Creating Guile Modules) for reference.

Then, when you boot Lilypond, instead of using 'load' to load these
files, you'd instead use 'use-modules', which would both load the .go
files and import all of their exported definitions into the main
Lilypond module.

If you do this, then you don't have to worry about what order you use to
compile or load things, and you can use the 'guild compile' command to
compile each file to a .go file.  See section 6.17.5 (Compiling Scheme
Code) in the manual for more details.  For example:

  $ guild compile -o foo.go foo.scm
  wrote `foo.go'

In the long run, I think this is probably your best way forward, but
admittedly it would require more work to make this transition.  So now I
will outline a couple of other options that require much less work.

You could write a little Scheme script that gets run by the Lilypond
build system to create the .go files.  This script would first set the
current module to whatever it will be when the Lilypond scheme files are
loaded at runtime, and then compile and load the .scm files in the
appropriate order, using something like this (untested) code:

  (set-current-module (resolve-module '(LILYPOND MODULE NAME)))
  (for-each (lambda (base-name)
              (let ((scm-file-name (string-append base-name ".scm"))
                    (go-file-name  (string-append base-name ".go")))
                (compile-file scm-file-name
                              #:output-file go-file-name
                              #:env (current-module)
                              #:opts %auto-compilation-options)
                (load scm-file-name)))
            <LIST-OF-BASE-NAMES>)

By compiling and loading each file in sequence, the macro definitions of
the earlier files will be available to the later files.

Alternatively, you could simply run Lilypond itself during the build
process, with the XDG_CACHE_HOME environment set to something in the
build directory so that the auto-compiled .go files will end up there.
However, this solution seems a bit less robust to me, as I could imagine
some day changing our policy of where the auto-compiled files are
placed.

Again, this is not my area of expertise, so hopefully Ludovic or Andy
could take a look at what I've written here and let us know if I made
any mistakes.

     Best,
      Mark

Re: Autocompilation/LilyPond

[David Kastrup]

Mark H Weaver <mhw@netris.org> writes:

> David Kastrup <dak@gnu.org> writes:
>
>> How is this supposed to work for compiling and installing a package?
>> Basically,
>>
>> make all
>> sudo make install
>>
>> The usual case will be that the user calling lilypond will not have
>> write permission in the installed directories (and even if he did, like
>> when calling lilypond as root, lilypond should not stomp over the
>> installed files).
>>
>> So what would make all do to generate one or more .go files?
>
> I'm not very familiar with the build system, so it would be great if
> Ludovic or Andy could chime in here, but as I understand it, the way
> it's meant to work is as follows:
>
> Ideally, each file contains a single module, whose name matches its own
> pathname relative to an element of the GUILE_LOAD_PATH, with a
> (define-module ...) header at the top declaring its dependencies on
> other modules (using #:use-module) and its exports (using #:export and
> #:export-syntax).  See guile-2.0.5/modules/* for many examples, and
> section 6.19.3 (Creating Guile Modules) for reference.

That's not really all that feasible: many of the dependencies are
implied in the load order, and it would be rather awkward to put an
explicit #:use-module for every part previous in the order.

> If you do this, then you don't have to worry about what order you use
> to compile or load things, and you can use the 'guild compile' command
> to compile each file to a .go file.  See section 6.17.5 (Compiling
> Scheme Code) in the manual for more details.  For example:
>
>   $ guild compile -o foo.go foo.scm
>   wrote `foo.go'
>
> In the long run, I think this is probably your best way forward, but
> admittedly it would require more work to make this transition.

The main problem is that it requires such a large reorganisation of the
LilyPond sources that the attempts to do it in that manner tended to
consist of outdated nontrivially rearranged parts before getting through
peer review successfully.  LilyPond is quite a moving target.

Ian Hulin has mostly worked on Guilev2 migration in that manner, and it
has caused him to do lots of futile work, a major cause of frustration
for him and of worry for others because the large reorganisations made
the work hard to verify.

So while this might be the "if everything was written from scratch, it
would make most sense to do it this way" approach, it has proven less
than fabulous as a migration strategy and is probably the major cause
that Ian and myself probably don't have the most happy thoughts when
thinking about each other.

So an approach that does not require manual declaration of dependencies
would certainly help.

> So now I will outline a couple of other options that require much less
> work.

I'll have to look at them.

-- 
David Kastrup

Re: Autocompilation/LilyPond

[Mark H Weaver]

David Kastrup <dak@gnu.org> writes:

>> In the long run, I think this is probably your best way forward, but
>> admittedly it would require more work to make this transition.
>
> The main problem is that it requires such a large reorganisation of the
> LilyPond sources that the attempts to do it in that manner tended to
> consist of outdated nontrivially rearranged parts before getting through
> peer review successfully.  LilyPond is quite a moving target.
>
> Ian Hulin has mostly worked on Guilev2 migration in that manner, and it
> has caused him to do lots of futile work, a major cause of frustration
> for him and of worry for others because the large reorganisations made
> the work hard to verify.
>
> So while this might be the "if everything was written from scratch, it
> would make most sense to do it this way" approach, it has proven less
> than fabulous as a migration strategy [...]

Okay, understood.  The other alternatives are workable, with the build
script written in Scheme probably being the most future-proof of the
remaining options.

It occurs to me that if the "lilypond module" has not already been set
up, then it will be completely empty and not suitable for compiling
anything, so in that case you'll want to do something closer to this:

  (define lilypond-module (make-fresh-user-module))
  (module-use! lilypond-module (resolve-interface '(guile)))
  (set-module-name! lilypond-module '(LILYPOND MODULE NAME))
  (set-current-module lilypond-module)
  (for-each (lambda (base-name)
              (let ((scm-file-name (string-append base-name ".scm"))
                    (go-file-name  (string-append base-name ".go")))
                (compile-file scm-file-name
                              #:output-file go-file-name
                              #:env (current-module)
                              #:opts %auto-compilation-options)
                (load scm-file-name)))
            <LIST-OF-BASE-NAMES>)

BTW, if I wanted to take a look to try to get it working myself, what
branch of the lilypond git repo should I look at?

    Thanks,
      Mark

Re: Autocompilation/LilyPond

[David Kastrup]

Mark H Weaver <mhw@netris.org> writes:

> David Kastrup <dak@gnu.org> writes:
>
>>> In the long run, I think this is probably your best way forward, but
>>> admittedly it would require more work to make this transition.
>>
>> The main problem is that it requires such a large reorganisation of the
>> LilyPond sources that the attempts to do it in that manner tended to
>> consist of outdated nontrivially rearranged parts before getting through
>> peer review successfully.  LilyPond is quite a moving target.
>>
>> Ian Hulin has mostly worked on Guilev2 migration in that manner, and it
>> has caused him to do lots of futile work, a major cause of frustration
>> for him and of worry for others because the large reorganisations made
>> the work hard to verify.
>>
>> So while this might be the "if everything was written from scratch, it
>> would make most sense to do it this way" approach, it has proven less
>> than fabulous as a migration strategy [...]
>
> Okay, understood.  The other alternatives are workable, with the build
> script written in Scheme probably being the most future-proof of the
> remaining options.

Sounds somewhat like it.

> It occurs to me that if the "lilypond module" has not already been set
> up,

I think it will be.

> then it will be completely empty and not suitable for compiling
> anything, so in that case you'll want to do something closer to this:
>
>   (define lilypond-module (make-fresh-user-module))
>   (module-use! lilypond-module (resolve-interface '(guile)))
>   (set-module-name! lilypond-module '(LILYPOND MODULE NAME))
>   (set-current-module lilypond-module)
>   (for-each (lambda (base-name)
>               (let ((scm-file-name (string-append base-name ".scm"))
>                     (go-file-name  (string-append base-name ".go")))
>                 (compile-file scm-file-name
>                               #:output-file go-file-name
>                               #:env (current-module)
>                               #:opts %auto-compilation-options)
>                 (load scm-file-name)))
>             <LIST-OF-BASE-NAMES>)
>
> BTW, if I wanted to take a look to try to get it working myself, what
> branch of the lilypond git repo should I look at?

Believe it or not, master.  There is nothing in the public repository
that would have more progress regarding Guilev2 migration (it might make
sense to ask Ian whether you can pull something from him).  master is
always kept in a state where it compiles, passes regtests, and builds
the documentation, and should not trail the "staging" branch (where new
changes get pushed) for more than a day.  Unless "staging" does not pass
the tests, in which case the automated push does not proceed.

You can look for open issues regarding Guilev2 migration in
<URL:http://code.google.com/p/lilypond/issues/list?can=2&q=guile>

Various issue carry patches.  I think the one where Ian ran out of steam
on last had been
<URL:http://code.google.com/p/lilypond/issues/detail?id=2026>

Note that the problems occuring here are not all particularly related to
Guilev2 or its directory layout.  But whenever there was non-trivial
delay, Ian had to start reorganizing the source tree from scratch or try
tracking recent changes back into his version.  So each of the
comparatively small problems in migrating meant that he had to start
over.  And it meant that it was very hard to verify his work since there
was a large reorganisation each time.

So while it may all seem like a huge bunch of unnecessary stupidity, the
"proper" way to do this has, in practice, shown itself to result in
motivationally devastating effects.  We need a migration strategy with
more incremental psychological rewards and less fear (namely, something
that is better verifiable than a large reorganisation of the source).

We mostly have to rely on working with humans.

-- 
David Kastrup

Re: Autocompilation/LilyPond

[Ian Hulin]

Hi David, Mark,
I am still around, I've not had much time for hacking lately as I've
been getting sick again, and the meds tend to sap the higher brain
functions.


On 09/03/12 19:27, David Kastrup wrote:
> Mark H Weaver <mhw@netris.org> writes:
> 
>> David Kastrup <dak@gnu.org> writes:
>> 
>>>> In the long run, I think this is probably your best way
>>>> forward, but admittedly it would require more work to make
>>>> this transition.
>>> 
>>> The main problem is that it requires such a large
>>> reorganisation of the LilyPond sources that the attempts to do
>>> it in that manner tended to consist of outdated nontrivially
>>> rearranged parts before getting through peer review
>>> successfully.  LilyPond is quite a moving target.
>>> 
>>> Ian Hulin has mostly worked on Guilev2 migration in that
>>> manner, and it has caused him to do lots of futile work, a
>>> major cause of frustration for him and of worry for others
>>> because the large reorganisations made the work hard to
>>> verify.
>>> 
>>> So while this might be the "if everything was written from
>>> scratch, it would make most sense to do it this way" approach,
>>> it has proven less than fabulous as a migration strategy [...]
>> 
>> Okay, understood.  The other alternatives are workable, with the
>> build script written in Scheme probably being the most
>> future-proof of the remaining options.
> 
> Sounds somewhat like it.
> 
>> It occurs to me that if the "lilypond module" has not already
>> been set up,
> 
The module is '(lily). It's set up in the Lily initialization code in
C++ thus:

1. main routine calls scm_boot_guile with callback to C++ routine
main_with_guile.

2. main_with_guile
2.1 sets up scheme LOAD_PATH to prepend our local LilyPond root
directory and the /scm subdirectory (this is so the modules declared
in our kit's scheme files (like scm/display-lily.scm get autoloaded
correctly).
2.2 calls ly_c_init_guile to define the '(lily) module in code and
specify ly_init_ly_module as a callback.

2.2.1 ly_init_ly_callback
2.2.1.1 sets up some internal initialization functions,
2.2.1.2 dumps out some trace code, if required, about what it's doing
2.2.1.3 does the equivalent of  (primitive_load_path "lily.scm") from
code.
2.2.2 (now we're back in ly_c_init_guile). scm_c_define_module has
returned a handle to the new module, step 2.2.1.3 has loaded and
evaluated all the stuff in lily.scm - including the load loop with all
the component scheme file held in scm/*.scm with their problematic
interactions.
2.2.3 do equivalent of (use_modules '(lily))
2.3 (now we're back in main_with_guile).  Set up local C++ constructors
2.4 set up fonts stuff
2.5 set up for multi-file compilation
2.5 process command-line options
2.6 set up handling to operate in server (jail) mode
2.7 look up and call the scheme entry-point in the '(lily) module to
actually do the LilyPond compilations - there is no normal exit point
from here, exiting the image is handled by scheme (exit) calls in the
'(lily) code.

So the "lilypond module" *is* normally set up when running with Guile
1.8, providing nothing breaks when lily.scm is loaded.

> I think it will be.
> 
>> then it will be completely empty and not suitable for compiling 
>> anything, so in that case you'll want to do something closer to
>> this:
>> 
>> (define lilypond-module (make-fresh-user-module)) (module-use!
>> lilypond-module (resolve-interface '(guile))) (set-module-name!
>> lilypond-module '(LILYPOND MODULE NAME)) (set-current-module
>> lilypond-module) (for-each (lambda (base-name) (let
>> ((scm-file-name (string-append base-name ".scm")) (go-file-name
>> (string-append base-name ".go"))) (compile-file scm-file-name 
>> #:output-file go-file-name #:env (current-module) #:opts
>> %auto-compilation-options) (load scm-file-name))) 
>> <LIST-OF-BASE-NAMES>)
>> 
>> BTW, if I wanted to take a look to try to get it working myself,
>> what branch of the lilypond git repo should I look at?
> 
> Believe it or not, master.  There is nothing in the public
> repository that would have more progress regarding Guilev2
> migration (it might make sense to ask Ian whether you can pull
> something from him).  master is always kept in a state where it
> compiles, passes regtests, and builds the documentation, and should
> not trail the "staging" branch (where new changes get pushed) for
> more than a day.  Unless "staging" does not pass the tests, in
> which case the automated push does not proceed.
> 
> You can look for open issues regarding Guilev2 migration in 
> <URL:http://code.google.com/p/lilypond/issues/list?can=2&q=guile>
> 
> Various issue carry patches.  I think the one where Ian ran out of
> steam on last had been 
> <URL:http://code.google.com/p/lilypond/issues/detail?id=2026>
> 
Sorry David, this may be a bit of a Red Herring.
<URL:http://code.google.com/p/lilypond/issues/detail?id=1686> was what
I was working on. 2026 involved trying to package the markup subsystem
within the '(lily) module code so we could load it as a module.
Problems are this is
a) subject to the flakiness of the dynamic load order within lily.scm,
*and*
b) as it defines an internal interpreter, the definition order within
the code defining the markup interpreter macros
(scm/markup-macros.scm)is crucial, especially when some of them
actually assume they can forward-reference macro definitions within
the module. This breaks badly when you try to use byte-compiled code
for this file.
c) there are internal design issues with the LilyPond markup code re
validation within the interpreter and its interactions with
declarations in lily.scm.
I decided I needed to park work on 2026 and re-try implementing 1686
using compile-one-file approach.
> Note that the problems occuring here are not all particularly
> related to Guilev2 or its directory layout.  But whenever there was
> non-trivial delay, Ian had to start reorganizing the source tree
> from scratch or try tracking recent changes back into his version.
> So each of the comparatively small problems in migrating meant that
> he had to start over.  And it meant that it was very hard to verify
> his work since there was a large reorganisation each time.
> 
> So while it may all seem like a huge bunch of unnecessary
> stupidity, the "proper" way to do this has, in practice, shown
> itself to result in motivationally devastating effects.  We need a
> migration strategy with more incremental psychological rewards and
> less fear (namely, something that is better verifiable than a large
> reorganisation of the source).
> 
> We mostly have to rely on working with humans.
> 
I had got some way with 1686.  I've got the changes working in our
main.cc (the stuff I walked through above mostly working, and was
trying this strategy.

For Guile 2, lily.scm is re-written as lily-guile2.scm.  The guile 1
code lily.scm is renamed to lily-guile1.scm.
lily-guile2.scm is uses

I added a shim in guile-init.cc (the one containing ly_c_init_guile
and ly_init_ly_module) to load lily-guile2.scm if running Guile 2, and
lily-guile1.scm if running with Guile 1.8.

I also tried compiling lily-guile2.scm  if running Guile2 and if
lily-guile2.go didn't exist before doing calling load-from-path (this
way I could be sure of loading the .go file if at all possible).

This is where I got stuck.  When compiling the new lily-guile2.scm, it
includes a file called music-functions.scm, which autoloads a module
(scm display-lily), which is defined in scm/display-lily.scm, which in
turn uses a scheme hash table.  The hash table is defined in code
using LY_DEFINE (our version of SCM_DEFINE) in our C++ code, and the
current code contains a (use-modules '(lily)) to actually reference
it. The compilation stage currently dies with a scheme
unbound-variable throw.

The next I was going to try before illness intervened was supplying a
new shim lily.scm to do

(cond-expand
	(guile-2
	  (load-from-path "lily-guile2.scm"))
	(guile-1
	  (load-from-path "lily-guile1.scm")))
So the autoload from music-functions.scm found something to load and
maybe gave me a more sensible message to see if it *really* couldn't
find the internally declared scheme definition.

Guys, thanks for looking at this, hope this info is useful.  Let me
know if you want any of the code I've changed thus far.

Cheers,

Ian Hulin
	

Re: Autocompilation/LilyPond

[David Kastrup]

Ian Hulin <ian@hulin.org.uk> writes:

> Hi David, Mark,
> I am still around, I've not had much time for hacking lately as I've
> been getting sick again, and the meds tend to sap the higher brain
> functions.

I'll be taking a closer look in several days (my schedule does not
really permit me doing much before the end of next week when I'll hold a
talk about LilyPond).

And I certainly wish you all the best for getting well.  But the main
points of what I wrote are not rendered invalid: you were put back
repeatedly to square one by delays (whatever might have caused them),
and if you had managed to do everything at once, the change would not
have been easily reviewable, and would have made our change history very
difficult to navigate, and would have rendered it very hard to track
whether already made changes were lost.

Just recently, we had a mishap with git and merging diverging branches
that caused about two versions worth of work to disappear.  I figured
out a cure (probably not the optimal one), and about 35 already verified
issues needed to get rechecked, by looking whether the changes had been
properly reintroduced by my fix.  This verification was achieved in
several days due to the bug squad working overtime.  And the patches
that were checked only had to be checked to be present in the same place
as before.

If we would need to verify a large _reorganisation_ of the source to be
up to date, this would be much much more intractable.  We really need to
be able to do this in smaller steps.  And it is not your health that is
causing a holdback, but rather the health of the project.

I am glad for the information you gave here, and I'll try picking up on
it after the conference.

I wish you the best regarding your health, and with some luck, by the
time you are interested in getting to work on LilyPond again, this rock
of Sisyphos will already be on the other side, making room for more
rewarding things.

All the best

-- 
David Kastrup

Re: Autocompilation/LilyPond

[Ludovic Courtès]

Hi David,

Sorry for the late reply.

David Kastrup <dak@gnu.org> skribis:

> Previous attempts have mostly exploded around the problem that we have
> something like
>
> (for-each ly:load init-scheme-files)
>
> in our lily.scm file, and the auto-compiler attempts to compile all of
> those files independently as far as I understand.  Unfortunately, some
> of them contain macro definitions that other files rely on.

The order in which files get compiled does not matter; the semantics of
programs do not depend on whether code is being bytecode-interpreted or
just interpreted by (ice-9 eval).

The only reason you might want to compile files in topological order is
performance.

Does that answer your question?

Thanks,
Ludo’.

Re: Autocompilation/LilyPond

[David Kastrup]

ludo@gnu.org (Ludovic Courtès) writes:

> Hi David,
>
> Sorry for the late reply.
>
> David Kastrup <dak@gnu.org> skribis:
>
>> Previous attempts have mostly exploded around the problem that we have
>> something like
>>
>> (for-each ly:load init-scheme-files)
>>
>> in our lily.scm file, and the auto-compiler attempts to compile all of
>> those files independently as far as I understand.  Unfortunately, some
>> of them contain macro definitions that other files rely on.
>
> The order in which files get compiled does not matter; the semantics of
> programs do not depend on whether code is being bytecode-interpreted or
> just interpreted by (ice-9 eval).

Little things like

(define-public fancy-format
  format)

(define-public (ergonomic-simple-format dest . rest)
  "Like ice-9's @code{format}, but without the memory consumption."
  (if (string? dest)
      (apply simple-format (cons #f (cons dest rest)))
      (apply simple-format (cons dest rest))))

(define format
  ergonomic-simple-format)

tend to make quite a difference depending on whether they are loaded or
not before compiling.

That one actually caused a lot of wasted effort on
<URL:http://code.google.com/p/lilypond/issues/detail?id=1780>

> The only reason you might want to compile files in topological order
> is performance.

And macros.  And redefinitions.  And module hierarchy.

> Does that answer your question?

Sure.  Unfortunately, we were already hit by this answer being wrong.

-- 
David Kastrup

Re: Autocompilation/LilyPond

[Ludovic Courtès]

Hi David,

David Kastrup <dak@gnu.org> skribis:

> ludo@gnu.org (Ludovic Courtès) writes:

[...]

>> The order in which files get compiled does not matter; the semantics of
>> programs do not depend on whether code is being bytecode-interpreted or
>> just interpreted by (ice-9 eval).
>
> Little things like
>
> (define-public fancy-format
>   format)
>
> (define-public (ergonomic-simple-format dest . rest)
>   "Like ice-9's @code{format}, but without the memory consumption."
>   (if (string? dest)
>       (apply simple-format (cons #f (cons dest rest)))
>       (apply simple-format (cons dest rest))))
>
> (define format
>   ergonomic-simple-format)
>
> tend to make quite a difference depending on whether they are loaded or
> not before compiling.
>
> That one actually caused a lot of wasted effort on
> <URL:http://code.google.com/p/lilypond/issues/detail?id=1780>

The ticket is a bit too dense for me.  What’s the operational difference
between compiling and evaluating the above code?

>> The only reason you might want to compile files in topological order
>> is performance.
>
> And macros.  And redefinitions.  And module hierarchy.

I think you’re talking about what should be *re*compiled when a module
has changed, right?

What I was mentioning above holds for one-shot compilation of a set of
modules, but you’re right that things are trickier when it comes to
incrementally changing part of that module set.

Thanks,
Ludo’.