* Autocompilation/LilyPond @ 2012-03-05 10:27 David Kastrup 2012-03-05 13:45 ` Autocompilation/LilyPond Mark H Weaver 2012-03-10 22:41 ` Autocompilation/LilyPond Ludovic Courtès 0 siblings, 2 replies; 16+ messages in thread From: David Kastrup @ 2012-03-05 10:27 UTC (permalink / raw) To: guile-devel Hi, with the stable release 2.16 of LilyPond looming around the corner, it will become imminent soon to think about supporting Guile 2.0. Previous attempts have mostly exploded around the problem that we have something like (for-each ly:load init-scheme-files) in our lily.scm file, and the auto-compiler attempts to compile all of those files independently as far as I understand. Unfortunately, some of them contain macro definitions that other files rely on. Personally, I think it would make sense if we could get the autocompiler to treat the whole blob of files as _one_ unit, and recompile the unit if it gets out of date. That would save us from trying to factor out macro dependencies into separate files (and since our markup system defines a macro for every markup function, and since the macros are needed when building markups in Scheme, this is actually rather hard to do). You might say that it is LilyPond's own fault that it has reverted a bit more to macro programming than feasible for its own good. I would not actually say that you are wrong. However, there is the problem of lifting a whole bunch of working code base under active development into the Guilev2 era, and if we could tackle design or maldesign questions mostly independently and in bite-sized chunks rather than humongous patches moving material around, this would help a lot in getting Guilev2 support on track. Suggestions? -- David Kastrup ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Autocompilation/LilyPond 2012-03-05 10:27 Autocompilation/LilyPond David Kastrup @ 2012-03-05 13:45 ` Mark H Weaver 2012-03-05 14:39 ` Autocompilation/LilyPond David Kastrup 2012-03-10 22:41 ` Autocompilation/LilyPond Ludovic Courtès 1 sibling, 1 reply; 16+ messages in thread From: Mark H Weaver @ 2012-03-05 13:45 UTC (permalink / raw) To: David Kastrup; +Cc: guile-devel David Kastrup <dak@gnu.org> writes: > with the stable release 2.16 of LilyPond looming around the corner, it > will become imminent soon to think about supporting Guile 2.0. > > Previous attempts have mostly exploded around the problem that we have > something like > > (for-each ly:load init-scheme-files) > > in our lily.scm file, and the auto-compiler attempts to compile all of > those files independently as far as I understand. Unfortunately, some > of them contain macro definitions that other files rely on. > > Personally, I think it would make sense if we could get the autocompiler > to treat the whole blob of files as _one_ unit, and recompile the unit > if it gets out of date. I'm not sure that would help much. There's a deeper problem that you should be aware of. In Guile 1.x, macro uses within procedures are not expanded until the procedure is evaluated. In Guile 2, macros are expanded as soon as the procedure is defined, even if compilation is turned off. This means that functions can only use macros that were previously defined. For example, the following works in Guile 1.8 but not in Guile 2: (define (foo x) (bar x)) (define-macro (bar x) `(quote ,x)) In Guile 2, you must put the definition of 'bar' before 'foo'. So you'll need to load your code in the right order so that macros always come before their uses. One clarification is in order. You might conclude from this that it is not possible to define mutually-recursive macros, but that's not true. For example, the following is fine: (define-macro (beep x) `(boop ,x)) (define-macro (boop x) `(quote ,x)) That's because 'beep' doesn't use 'boop', it merely produces a use of 'boop' in its expansion. 'boop' is merely part of a quoted literal within 'beep'. Regards, Mark ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Autocompilation/LilyPond 2012-03-05 13:45 ` Autocompilation/LilyPond Mark H Weaver @ 2012-03-05 14:39 ` David Kastrup 2012-03-05 17:57 ` Autocompilation/LilyPond Mark H Weaver 0 siblings, 1 reply; 16+ messages in thread From: David Kastrup @ 2012-03-05 14:39 UTC (permalink / raw) To: Mark H Weaver; +Cc: guile-devel Mark H Weaver <mhw@netris.org> writes: > David Kastrup <dak@gnu.org> writes: > >> with the stable release 2.16 of LilyPond looming around the corner, it >> will become imminent soon to think about supporting Guile 2.0. >> >> Previous attempts have mostly exploded around the problem that we have >> something like >> >> (for-each ly:load init-scheme-files) >> >> in our lily.scm file, and the auto-compiler attempts to compile all >> of those files independently as far as I understand. Unfortunately, >> some of them contain macro definitions that other files rely on. >> >> Personally, I think it would make sense if we could get the >> autocompiler to treat the whole blob of files as _one_ unit, and >> recompile the unit if it gets out of date. > > I'm not sure that would help much. There's a deeper problem that you > should be aware of. In Guile 1.x, macro uses within procedures are > not expanded until the procedure is evaluated. In Guile 2, macros are > expanded as soon as the procedure is defined, even if compilation is > turned off. This means that functions can only use macros that were > previously defined. I don't think that making this condition hold would be really hard. LilyPond has a rather carefully selected load order in several stages, so use-before-definition, whether in the context of macros or not, should be more the exception than the rule, and only require smaller rearrangements. > One clarification is in order. You might conclude from this that it > is not possible to define mutually-recursive macros, but that's not > true. I don't think we use them, anyway. Most problems are more due to small oversights rather than maliciously clever overdesign. -- David Kastrup ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Autocompilation/LilyPond 2012-03-05 14:39 ` Autocompilation/LilyPond David Kastrup @ 2012-03-05 17:57 ` Mark H Weaver 2012-03-06 1:03 ` Autocompilation/LilyPond David Kastrup 0 siblings, 1 reply; 16+ messages in thread From: Mark H Weaver @ 2012-03-05 17:57 UTC (permalink / raw) To: David Kastrup; +Cc: guile-devel David Kastrup <dak@gnu.org> writes: > Mark H Weaver <mhw@netris.org> writes: > >> David Kastrup <dak@gnu.org> writes: >> >>> with the stable release 2.16 of LilyPond looming around the corner, it >>> will become imminent soon to think about supporting Guile 2.0. >>> >>> Previous attempts have mostly exploded around the problem that we have >>> something like >>> >>> (for-each ly:load init-scheme-files) >>> >>> in our lily.scm file, and the auto-compiler attempts to compile all >>> of those files independently as far as I understand. Unfortunately, >>> some of them contain macro definitions that other files rely on. >>> >>> Personally, I think it would make sense if we could get the >>> autocompiler to treat the whole blob of files as _one_ unit, and >>> recompile the unit if it gets out of date. >> >> I'm not sure that would help much. There's a deeper problem that you >> should be aware of. In Guile 1.x, macro uses within procedures are >> not expanded until the procedure is evaluated. In Guile 2, macros are >> expanded as soon as the procedure is defined, even if compilation is >> turned off. This means that functions can only use macros that were >> previously defined. > > I don't think that making this condition hold would be really hard. > LilyPond has a rather carefully selected load order in several stages, > so use-before-definition, whether in the context of macros or not, > should be more the exception than the rule, and only require smaller > rearrangements. Excellent! As long as you load everything in the right order, such that macros are defined before they are used, I don't see why there should be any other problems related to macros and compilation. Mark ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Autocompilation/LilyPond 2012-03-05 17:57 ` Autocompilation/LilyPond Mark H Weaver @ 2012-03-06 1:03 ` David Kastrup 2012-03-06 2:36 ` Autocompilation/LilyPond Mark H Weaver 0 siblings, 1 reply; 16+ messages in thread From: David Kastrup @ 2012-03-06 1:03 UTC (permalink / raw) To: Mark H Weaver; +Cc: guile-devel Mark H Weaver <mhw@netris.org> writes: > David Kastrup <dak@gnu.org> writes: > >> Mark H Weaver <mhw@netris.org> writes: >> >>> David Kastrup <dak@gnu.org> writes: >>> >>>> with the stable release 2.16 of LilyPond looming around the corner, it >>>> will become imminent soon to think about supporting Guile 2.0. >>>> >>>> Previous attempts have mostly exploded around the problem that we have >>>> something like >>>> >>>> (for-each ly:load init-scheme-files) >>>> >>>> in our lily.scm file, and the auto-compiler attempts to compile all >>>> of those files independently as far as I understand. Unfortunately, >>>> some of them contain macro definitions that other files rely on. >>>> >>>> Personally, I think it would make sense if we could get the >>>> autocompiler to treat the whole blob of files as _one_ unit, and >>>> recompile the unit if it gets out of date. >>> >>> I'm not sure that would help much. There's a deeper problem that you >>> should be aware of. In Guile 1.x, macro uses within procedures are >>> not expanded until the procedure is evaluated. In Guile 2, macros are >>> expanded as soon as the procedure is defined, even if compilation is >>> turned off. This means that functions can only use macros that were >>> previously defined. >> >> I don't think that making this condition hold would be really hard. >> LilyPond has a rather carefully selected load order in several stages, >> so use-before-definition, whether in the context of macros or not, >> should be more the exception than the rule, and only require smaller >> rearrangements. > > Excellent! As long as you load everything in the right order, such that > macros are defined before they are used, I don't see why there should be > any other problems related to macros and compilation. Because the individual files are not independent from one another? That's why I wrote: Personally, I think it would make sense if we could get the autocompiler to treat the whole blob of files as _one_ unit, and recompile the unit if it gets out of date. -- David Kastrup ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Autocompilation/LilyPond 2012-03-06 1:03 ` Autocompilation/LilyPond David Kastrup @ 2012-03-06 2:36 ` Mark H Weaver 2012-03-07 23:51 ` Autocompilation/LilyPond David Kastrup 0 siblings, 1 reply; 16+ messages in thread From: Mark H Weaver @ 2012-03-06 2:36 UTC (permalink / raw) To: David Kastrup; +Cc: guile-devel David Kastrup <dak@gnu.org> writes: > Mark H Weaver <mhw@netris.org> writes: > >> Excellent! As long as you load everything in the right order, such that >> macros are defined before they are used, I don't see why there should be >> any other problems related to macros and compilation. > > Because the individual files are not independent from one another? > That's why I wrote: > > Personally, I think it would make sense if we could get the > autocompiler to treat the whole blob of files as _one_ unit, and > recompile the unit if it gets out of date. There's no problem with them being dependent on one another. When you load a file, even with auto-compilation, the macro expander will make use of whatever macros are already bound in the current module. The rest of the compiler sees only the output of the macro expander. Try the following experiment: put "(define-macro (bar x) `(quote ,x))" into "foo1.scm", and "(define (foo x) (bar x))" into "foo2.scm", and then within a REPL type: (load "foo1.scm") (load "foo2.scm") and observe that everything works as it should. If you really want everything to be compiled as one unit, you can use 'include' (which acts essentially the same as #include in C), though beware that Guile is not yet smart enough to auto-recompile when one of the included files gets updated. I don't see any compelling benefit to compiling everything as one unit, but do as you wish :) Regards, Mark ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Autocompilation/LilyPond 2012-03-06 2:36 ` Autocompilation/LilyPond Mark H Weaver @ 2012-03-07 23:51 ` David Kastrup 2012-03-09 16:19 ` Autocompilation/LilyPond Mark H Weaver 0 siblings, 1 reply; 16+ messages in thread From: David Kastrup @ 2012-03-07 23:51 UTC (permalink / raw) To: guile-devel Mark H Weaver <mhw@netris.org> writes: > David Kastrup <dak@gnu.org> writes: > >> Mark H Weaver <mhw@netris.org> writes: >> >>> Excellent! As long as you load everything in the right order, such that >>> macros are defined before they are used, I don't see why there should be >>> any other problems related to macros and compilation. >> >> Because the individual files are not independent from one another? >> That's why I wrote: >> >> Personally, I think it would make sense if we could get the >> autocompiler to treat the whole blob of files as _one_ unit, and >> recompile the unit if it gets out of date. > > There's no problem with them being dependent on one another. When you > load a file, even with auto-compilation, the macro expander will make > use of whatever macros are already bound in the current module. The > rest of the compiler sees only the output of the macro expander. > > Try the following experiment: put "(define-macro (bar x) `(quote ,x))" > into "foo1.scm", and "(define (foo x) (bar x))" into "foo2.scm", and > then within a REPL type: (load "foo1.scm") (load "foo2.scm") and observe > that everything works as it should. > > If you really want everything to be compiled as one unit, you can use > 'include' (which acts essentially the same as #include in C), though > beware that Guile is not yet smart enough to auto-recompile when one of > the included files gets updated. > > I don't see any compelling benefit to compiling everything as one unit, > but do as you wish :) How is this supposed to work for compiling and installing a package? Basically, make all sudo make install The usual case will be that the user calling lilypond will not have write permission in the installed directories (and even if he did, like when calling lilypond as root, lilypond should not stomp over the installed files). So what would make all do to generate one or more .go files? -- David Kastrup ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Autocompilation/LilyPond 2012-03-07 23:51 ` Autocompilation/LilyPond David Kastrup @ 2012-03-09 16:19 ` Mark H Weaver 2012-03-09 16:55 ` Autocompilation/LilyPond David Kastrup 0 siblings, 1 reply; 16+ messages in thread From: Mark H Weaver @ 2012-03-09 16:19 UTC (permalink / raw) To: David Kastrup; +Cc: guile-devel David Kastrup <dak@gnu.org> writes: > How is this supposed to work for compiling and installing a package? > Basically, > > make all > sudo make install > > The usual case will be that the user calling lilypond will not have > write permission in the installed directories (and even if he did, like > when calling lilypond as root, lilypond should not stomp over the > installed files). > > So what would make all do to generate one or more .go files? I'm not very familiar with the build system, so it would be great if Ludovic or Andy could chime in here, but as I understand it, the way it's meant to work is as follows: Ideally, each file contains a single module, whose name matches its own pathname relative to an element of the GUILE_LOAD_PATH, with a (define-module ...) header at the top declaring its dependencies on other modules (using #:use-module) and its exports (using #:export and #:export-syntax). See guile-2.0.5/modules/* for many examples, and section 6.19.3 (Creating Guile Modules) for reference. Then, when you boot Lilypond, instead of using 'load' to load these files, you'd instead use 'use-modules', which would both load the .go files and import all of their exported definitions into the main Lilypond module. If you do this, then you don't have to worry about what order you use to compile or load things, and you can use the 'guild compile' command to compile each file to a .go file. See section 6.17.5 (Compiling Scheme Code) in the manual for more details. For example: $ guild compile -o foo.go foo.scm wrote `foo.go' In the long run, I think this is probably your best way forward, but admittedly it would require more work to make this transition. So now I will outline a couple of other options that require much less work. You could write a little Scheme script that gets run by the Lilypond build system to create the .go files. This script would first set the current module to whatever it will be when the Lilypond scheme files are loaded at runtime, and then compile and load the .scm files in the appropriate order, using something like this (untested) code: (set-current-module (resolve-module '(LILYPOND MODULE NAME))) (for-each (lambda (base-name) (let ((scm-file-name (string-append base-name ".scm")) (go-file-name (string-append base-name ".go"))) (compile-file scm-file-name #:output-file go-file-name #:env (current-module) #:opts %auto-compilation-options) (load scm-file-name))) <LIST-OF-BASE-NAMES>) By compiling and loading each file in sequence, the macro definitions of the earlier files will be available to the later files. Alternatively, you could simply run Lilypond itself during the build process, with the XDG_CACHE_HOME environment set to something in the build directory so that the auto-compiled .go files will end up there. However, this solution seems a bit less robust to me, as I could imagine some day changing our policy of where the auto-compiled files are placed. Again, this is not my area of expertise, so hopefully Ludovic or Andy could take a look at what I've written here and let us know if I made any mistakes. Best, Mark ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Autocompilation/LilyPond 2012-03-09 16:19 ` Autocompilation/LilyPond Mark H Weaver @ 2012-03-09 16:55 ` David Kastrup 2012-03-09 18:57 ` Autocompilation/LilyPond Mark H Weaver 0 siblings, 1 reply; 16+ messages in thread From: David Kastrup @ 2012-03-09 16:55 UTC (permalink / raw) To: Mark H Weaver; +Cc: guile-devel Mark H Weaver <mhw@netris.org> writes: > David Kastrup <dak@gnu.org> writes: > >> How is this supposed to work for compiling and installing a package? >> Basically, >> >> make all >> sudo make install >> >> The usual case will be that the user calling lilypond will not have >> write permission in the installed directories (and even if he did, like >> when calling lilypond as root, lilypond should not stomp over the >> installed files). >> >> So what would make all do to generate one or more .go files? > > I'm not very familiar with the build system, so it would be great if > Ludovic or Andy could chime in here, but as I understand it, the way > it's meant to work is as follows: > > Ideally, each file contains a single module, whose name matches its own > pathname relative to an element of the GUILE_LOAD_PATH, with a > (define-module ...) header at the top declaring its dependencies on > other modules (using #:use-module) and its exports (using #:export and > #:export-syntax). See guile-2.0.5/modules/* for many examples, and > section 6.19.3 (Creating Guile Modules) for reference. That's not really all that feasible: many of the dependencies are implied in the load order, and it would be rather awkward to put an explicit #:use-module for every part previous in the order. > If you do this, then you don't have to worry about what order you use > to compile or load things, and you can use the 'guild compile' command > to compile each file to a .go file. See section 6.17.5 (Compiling > Scheme Code) in the manual for more details. For example: > > $ guild compile -o foo.go foo.scm > wrote `foo.go' > > In the long run, I think this is probably your best way forward, but > admittedly it would require more work to make this transition. The main problem is that it requires such a large reorganisation of the LilyPond sources that the attempts to do it in that manner tended to consist of outdated nontrivially rearranged parts before getting through peer review successfully. LilyPond is quite a moving target. Ian Hulin has mostly worked on Guilev2 migration in that manner, and it has caused him to do lots of futile work, a major cause of frustration for him and of worry for others because the large reorganisations made the work hard to verify. So while this might be the "if everything was written from scratch, it would make most sense to do it this way" approach, it has proven less than fabulous as a migration strategy and is probably the major cause that Ian and myself probably don't have the most happy thoughts when thinking about each other. So an approach that does not require manual declaration of dependencies would certainly help. > So now I will outline a couple of other options that require much less > work. I'll have to look at them. -- David Kastrup ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Autocompilation/LilyPond 2012-03-09 16:55 ` Autocompilation/LilyPond David Kastrup @ 2012-03-09 18:57 ` Mark H Weaver 2012-03-09 19:27 ` Autocompilation/LilyPond David Kastrup 0 siblings, 1 reply; 16+ messages in thread From: Mark H Weaver @ 2012-03-09 18:57 UTC (permalink / raw) To: David Kastrup; +Cc: guile-devel David Kastrup <dak@gnu.org> writes: >> In the long run, I think this is probably your best way forward, but >> admittedly it would require more work to make this transition. > > The main problem is that it requires such a large reorganisation of the > LilyPond sources that the attempts to do it in that manner tended to > consist of outdated nontrivially rearranged parts before getting through > peer review successfully. LilyPond is quite a moving target. > > Ian Hulin has mostly worked on Guilev2 migration in that manner, and it > has caused him to do lots of futile work, a major cause of frustration > for him and of worry for others because the large reorganisations made > the work hard to verify. > > So while this might be the "if everything was written from scratch, it > would make most sense to do it this way" approach, it has proven less > than fabulous as a migration strategy [...] Okay, understood. The other alternatives are workable, with the build script written in Scheme probably being the most future-proof of the remaining options. It occurs to me that if the "lilypond module" has not already been set up, then it will be completely empty and not suitable for compiling anything, so in that case you'll want to do something closer to this: (define lilypond-module (make-fresh-user-module)) (module-use! lilypond-module (resolve-interface '(guile))) (set-module-name! lilypond-module '(LILYPOND MODULE NAME)) (set-current-module lilypond-module) (for-each (lambda (base-name) (let ((scm-file-name (string-append base-name ".scm")) (go-file-name (string-append base-name ".go"))) (compile-file scm-file-name #:output-file go-file-name #:env (current-module) #:opts %auto-compilation-options) (load scm-file-name))) <LIST-OF-BASE-NAMES>) BTW, if I wanted to take a look to try to get it working myself, what branch of the lilypond git repo should I look at? Thanks, Mark ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Autocompilation/LilyPond 2012-03-09 18:57 ` Autocompilation/LilyPond Mark H Weaver @ 2012-03-09 19:27 ` David Kastrup 2012-03-10 12:24 ` Autocompilation/LilyPond Ian Hulin 0 siblings, 1 reply; 16+ messages in thread From: David Kastrup @ 2012-03-09 19:27 UTC (permalink / raw) To: Mark H Weaver; +Cc: guile-devel Mark H Weaver <mhw@netris.org> writes: > David Kastrup <dak@gnu.org> writes: > >>> In the long run, I think this is probably your best way forward, but >>> admittedly it would require more work to make this transition. >> >> The main problem is that it requires such a large reorganisation of the >> LilyPond sources that the attempts to do it in that manner tended to >> consist of outdated nontrivially rearranged parts before getting through >> peer review successfully. LilyPond is quite a moving target. >> >> Ian Hulin has mostly worked on Guilev2 migration in that manner, and it >> has caused him to do lots of futile work, a major cause of frustration >> for him and of worry for others because the large reorganisations made >> the work hard to verify. >> >> So while this might be the "if everything was written from scratch, it >> would make most sense to do it this way" approach, it has proven less >> than fabulous as a migration strategy [...] > > Okay, understood. The other alternatives are workable, with the build > script written in Scheme probably being the most future-proof of the > remaining options. Sounds somewhat like it. > It occurs to me that if the "lilypond module" has not already been set > up, I think it will be. > then it will be completely empty and not suitable for compiling > anything, so in that case you'll want to do something closer to this: > > (define lilypond-module (make-fresh-user-module)) > (module-use! lilypond-module (resolve-interface '(guile))) > (set-module-name! lilypond-module '(LILYPOND MODULE NAME)) > (set-current-module lilypond-module) > (for-each (lambda (base-name) > (let ((scm-file-name (string-append base-name ".scm")) > (go-file-name (string-append base-name ".go"))) > (compile-file scm-file-name > #:output-file go-file-name > #:env (current-module) > #:opts %auto-compilation-options) > (load scm-file-name))) > <LIST-OF-BASE-NAMES>) > > BTW, if I wanted to take a look to try to get it working myself, what > branch of the lilypond git repo should I look at? Believe it or not, master. There is nothing in the public repository that would have more progress regarding Guilev2 migration (it might make sense to ask Ian whether you can pull something from him). master is always kept in a state where it compiles, passes regtests, and builds the documentation, and should not trail the "staging" branch (where new changes get pushed) for more than a day. Unless "staging" does not pass the tests, in which case the automated push does not proceed. You can look for open issues regarding Guilev2 migration in <URL:http://code.google.com/p/lilypond/issues/list?can=2&q=guile> Various issue carry patches. I think the one where Ian ran out of steam on last had been <URL:http://code.google.com/p/lilypond/issues/detail?id=2026> Note that the problems occuring here are not all particularly related to Guilev2 or its directory layout. But whenever there was non-trivial delay, Ian had to start reorganizing the source tree from scratch or try tracking recent changes back into his version. So each of the comparatively small problems in migrating meant that he had to start over. And it meant that it was very hard to verify his work since there was a large reorganisation each time. So while it may all seem like a huge bunch of unnecessary stupidity, the "proper" way to do this has, in practice, shown itself to result in motivationally devastating effects. We need a migration strategy with more incremental psychological rewards and less fear (namely, something that is better verifiable than a large reorganisation of the source). We mostly have to rely on working with humans. -- David Kastrup ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Autocompilation/LilyPond 2012-03-09 19:27 ` Autocompilation/LilyPond David Kastrup @ 2012-03-10 12:24 ` Ian Hulin 2012-03-10 13:05 ` Autocompilation/LilyPond David Kastrup 0 siblings, 1 reply; 16+ messages in thread From: Ian Hulin @ 2012-03-10 12:24 UTC (permalink / raw) To: guile-devel; +Cc: Mark H Weaver Hi David, Mark, I am still around, I've not had much time for hacking lately as I've been getting sick again, and the meds tend to sap the higher brain functions. On 09/03/12 19:27, David Kastrup wrote: > Mark H Weaver <mhw@netris.org> writes: > >> David Kastrup <dak@gnu.org> writes: >> >>>> In the long run, I think this is probably your best way >>>> forward, but admittedly it would require more work to make >>>> this transition. >>> >>> The main problem is that it requires such a large >>> reorganisation of the LilyPond sources that the attempts to do >>> it in that manner tended to consist of outdated nontrivially >>> rearranged parts before getting through peer review >>> successfully. LilyPond is quite a moving target. >>> >>> Ian Hulin has mostly worked on Guilev2 migration in that >>> manner, and it has caused him to do lots of futile work, a >>> major cause of frustration for him and of worry for others >>> because the large reorganisations made the work hard to >>> verify. >>> >>> So while this might be the "if everything was written from >>> scratch, it would make most sense to do it this way" approach, >>> it has proven less than fabulous as a migration strategy [...] >> >> Okay, understood. The other alternatives are workable, with the >> build script written in Scheme probably being the most >> future-proof of the remaining options. > > Sounds somewhat like it. > >> It occurs to me that if the "lilypond module" has not already >> been set up, > The module is '(lily). It's set up in the Lily initialization code in C++ thus: 1. main routine calls scm_boot_guile with callback to C++ routine main_with_guile. 2. main_with_guile 2.1 sets up scheme LOAD_PATH to prepend our local LilyPond root directory and the /scm subdirectory (this is so the modules declared in our kit's scheme files (like scm/display-lily.scm get autoloaded correctly). 2.2 calls ly_c_init_guile to define the '(lily) module in code and specify ly_init_ly_module as a callback. 2.2.1 ly_init_ly_callback 2.2.1.1 sets up some internal initialization functions, 2.2.1.2 dumps out some trace code, if required, about what it's doing 2.2.1.3 does the equivalent of (primitive_load_path "lily.scm") from code. 2.2.2 (now we're back in ly_c_init_guile). scm_c_define_module has returned a handle to the new module, step 2.2.1.3 has loaded and evaluated all the stuff in lily.scm - including the load loop with all the component scheme file held in scm/*.scm with their problematic interactions. 2.2.3 do equivalent of (use_modules '(lily)) 2.3 (now we're back in main_with_guile). Set up local C++ constructors 2.4 set up fonts stuff 2.5 set up for multi-file compilation 2.5 process command-line options 2.6 set up handling to operate in server (jail) mode 2.7 look up and call the scheme entry-point in the '(lily) module to actually do the LilyPond compilations - there is no normal exit point from here, exiting the image is handled by scheme (exit) calls in the '(lily) code. So the "lilypond module" *is* normally set up when running with Guile 1.8, providing nothing breaks when lily.scm is loaded. > I think it will be. > >> then it will be completely empty and not suitable for compiling >> anything, so in that case you'll want to do something closer to >> this: >> >> (define lilypond-module (make-fresh-user-module)) (module-use! >> lilypond-module (resolve-interface '(guile))) (set-module-name! >> lilypond-module '(LILYPOND MODULE NAME)) (set-current-module >> lilypond-module) (for-each (lambda (base-name) (let >> ((scm-file-name (string-append base-name ".scm")) (go-file-name >> (string-append base-name ".go"))) (compile-file scm-file-name >> #:output-file go-file-name #:env (current-module) #:opts >> %auto-compilation-options) (load scm-file-name))) >> <LIST-OF-BASE-NAMES>) >> >> BTW, if I wanted to take a look to try to get it working myself, >> what branch of the lilypond git repo should I look at? > > Believe it or not, master. There is nothing in the public > repository that would have more progress regarding Guilev2 > migration (it might make sense to ask Ian whether you can pull > something from him). master is always kept in a state where it > compiles, passes regtests, and builds the documentation, and should > not trail the "staging" branch (where new changes get pushed) for > more than a day. Unless "staging" does not pass the tests, in > which case the automated push does not proceed. > > You can look for open issues regarding Guilev2 migration in > <URL:http://code.google.com/p/lilypond/issues/list?can=2&q=guile> > > Various issue carry patches. I think the one where Ian ran out of > steam on last had been > <URL:http://code.google.com/p/lilypond/issues/detail?id=2026> > Sorry David, this may be a bit of a Red Herring. <URL:http://code.google.com/p/lilypond/issues/detail?id=1686> was what I was working on. 2026 involved trying to package the markup subsystem within the '(lily) module code so we could load it as a module. Problems are this is a) subject to the flakiness of the dynamic load order within lily.scm, *and* b) as it defines an internal interpreter, the definition order within the code defining the markup interpreter macros (scm/markup-macros.scm)is crucial, especially when some of them actually assume they can forward-reference macro definitions within the module. This breaks badly when you try to use byte-compiled code for this file. c) there are internal design issues with the LilyPond markup code re validation within the interpreter and its interactions with declarations in lily.scm. I decided I needed to park work on 2026 and re-try implementing 1686 using compile-one-file approach. > Note that the problems occuring here are not all particularly > related to Guilev2 or its directory layout. But whenever there was > non-trivial delay, Ian had to start reorganizing the source tree > from scratch or try tracking recent changes back into his version. > So each of the comparatively small problems in migrating meant that > he had to start over. And it meant that it was very hard to verify > his work since there was a large reorganisation each time. > > So while it may all seem like a huge bunch of unnecessary > stupidity, the "proper" way to do this has, in practice, shown > itself to result in motivationally devastating effects. We need a > migration strategy with more incremental psychological rewards and > less fear (namely, something that is better verifiable than a large > reorganisation of the source). > > We mostly have to rely on working with humans. > I had got some way with 1686. I've got the changes working in our main.cc (the stuff I walked through above mostly working, and was trying this strategy. For Guile 2, lily.scm is re-written as lily-guile2.scm. The guile 1 code lily.scm is renamed to lily-guile1.scm. lily-guile2.scm is uses I added a shim in guile-init.cc (the one containing ly_c_init_guile and ly_init_ly_module) to load lily-guile2.scm if running Guile 2, and lily-guile1.scm if running with Guile 1.8. I also tried compiling lily-guile2.scm if running Guile2 and if lily-guile2.go didn't exist before doing calling load-from-path (this way I could be sure of loading the .go file if at all possible). This is where I got stuck. When compiling the new lily-guile2.scm, it includes a file called music-functions.scm, which autoloads a module (scm display-lily), which is defined in scm/display-lily.scm, which in turn uses a scheme hash table. The hash table is defined in code using LY_DEFINE (our version of SCM_DEFINE) in our C++ code, and the current code contains a (use-modules '(lily)) to actually reference it. The compilation stage currently dies with a scheme unbound-variable throw. The next I was going to try before illness intervened was supplying a new shim lily.scm to do (cond-expand (guile-2 (load-from-path "lily-guile2.scm")) (guile-1 (load-from-path "lily-guile1.scm"))) So the autoload from music-functions.scm found something to load and maybe gave me a more sensible message to see if it *really* couldn't find the internally declared scheme definition. Guys, thanks for looking at this, hope this info is useful. Let me know if you want any of the code I've changed thus far. Cheers, Ian Hulin ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Autocompilation/LilyPond 2012-03-10 12:24 ` Autocompilation/LilyPond Ian Hulin @ 2012-03-10 13:05 ` David Kastrup 0 siblings, 0 replies; 16+ messages in thread From: David Kastrup @ 2012-03-10 13:05 UTC (permalink / raw) To: guile-devel Ian Hulin <ian@hulin.org.uk> writes: > Hi David, Mark, > I am still around, I've not had much time for hacking lately as I've > been getting sick again, and the meds tend to sap the higher brain > functions. I'll be taking a closer look in several days (my schedule does not really permit me doing much before the end of next week when I'll hold a talk about LilyPond). And I certainly wish you all the best for getting well. But the main points of what I wrote are not rendered invalid: you were put back repeatedly to square one by delays (whatever might have caused them), and if you had managed to do everything at once, the change would not have been easily reviewable, and would have made our change history very difficult to navigate, and would have rendered it very hard to track whether already made changes were lost. Just recently, we had a mishap with git and merging diverging branches that caused about two versions worth of work to disappear. I figured out a cure (probably not the optimal one), and about 35 already verified issues needed to get rechecked, by looking whether the changes had been properly reintroduced by my fix. This verification was achieved in several days due to the bug squad working overtime. And the patches that were checked only had to be checked to be present in the same place as before. If we would need to verify a large _reorganisation_ of the source to be up to date, this would be much much more intractable. We really need to be able to do this in smaller steps. And it is not your health that is causing a holdback, but rather the health of the project. I am glad for the information you gave here, and I'll try picking up on it after the conference. I wish you the best regarding your health, and with some luck, by the time you are interested in getting to work on LilyPond again, this rock of Sisyphos will already be on the other side, making room for more rewarding things. All the best -- David Kastrup ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Autocompilation/LilyPond 2012-03-05 10:27 Autocompilation/LilyPond David Kastrup 2012-03-05 13:45 ` Autocompilation/LilyPond Mark H Weaver @ 2012-03-10 22:41 ` Ludovic Courtès 2012-03-10 23:15 ` Autocompilation/LilyPond David Kastrup 1 sibling, 1 reply; 16+ messages in thread From: Ludovic Courtès @ 2012-03-10 22:41 UTC (permalink / raw) To: guile-devel Hi David, Sorry for the late reply. David Kastrup <dak@gnu.org> skribis: > Previous attempts have mostly exploded around the problem that we have > something like > > (for-each ly:load init-scheme-files) > > in our lily.scm file, and the auto-compiler attempts to compile all of > those files independently as far as I understand. Unfortunately, some > of them contain macro definitions that other files rely on. The order in which files get compiled does not matter; the semantics of programs do not depend on whether code is being bytecode-interpreted or just interpreted by (ice-9 eval). The only reason you might want to compile files in topological order is performance. Does that answer your question? Thanks, Ludo’. ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Autocompilation/LilyPond 2012-03-10 22:41 ` Autocompilation/LilyPond Ludovic Courtès @ 2012-03-10 23:15 ` David Kastrup 2012-03-16 11:12 ` Autocompilation/LilyPond Ludovic Courtès 0 siblings, 1 reply; 16+ messages in thread From: David Kastrup @ 2012-03-10 23:15 UTC (permalink / raw) To: guile-devel ludo@gnu.org (Ludovic Courtès) writes: > Hi David, > > Sorry for the late reply. > > David Kastrup <dak@gnu.org> skribis: > >> Previous attempts have mostly exploded around the problem that we have >> something like >> >> (for-each ly:load init-scheme-files) >> >> in our lily.scm file, and the auto-compiler attempts to compile all of >> those files independently as far as I understand. Unfortunately, some >> of them contain macro definitions that other files rely on. > > The order in which files get compiled does not matter; the semantics of > programs do not depend on whether code is being bytecode-interpreted or > just interpreted by (ice-9 eval). Little things like (define-public fancy-format format) (define-public (ergonomic-simple-format dest . rest) "Like ice-9's @code{format}, but without the memory consumption." (if (string? dest) (apply simple-format (cons #f (cons dest rest))) (apply simple-format (cons dest rest)))) (define format ergonomic-simple-format) tend to make quite a difference depending on whether they are loaded or not before compiling. That one actually caused a lot of wasted effort on <URL:http://code.google.com/p/lilypond/issues/detail?id=1780> > The only reason you might want to compile files in topological order > is performance. And macros. And redefinitions. And module hierarchy. > Does that answer your question? Sure. Unfortunately, we were already hit by this answer being wrong. -- David Kastrup ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Autocompilation/LilyPond 2012-03-10 23:15 ` Autocompilation/LilyPond David Kastrup @ 2012-03-16 11:12 ` Ludovic Courtès 0 siblings, 0 replies; 16+ messages in thread From: Ludovic Courtès @ 2012-03-16 11:12 UTC (permalink / raw) To: guile-devel Hi David, David Kastrup <dak@gnu.org> skribis: > ludo@gnu.org (Ludovic Courtès) writes: [...] >> The order in which files get compiled does not matter; the semantics of >> programs do not depend on whether code is being bytecode-interpreted or >> just interpreted by (ice-9 eval). > > Little things like > > (define-public fancy-format > format) > > (define-public (ergonomic-simple-format dest . rest) > "Like ice-9's @code{format}, but without the memory consumption." > (if (string? dest) > (apply simple-format (cons #f (cons dest rest))) > (apply simple-format (cons dest rest)))) > > (define format > ergonomic-simple-format) > > tend to make quite a difference depending on whether they are loaded or > not before compiling. > > That one actually caused a lot of wasted effort on > <URL:http://code.google.com/p/lilypond/issues/detail?id=1780> The ticket is a bit too dense for me. What’s the operational difference between compiling and evaluating the above code? >> The only reason you might want to compile files in topological order >> is performance. > > And macros. And redefinitions. And module hierarchy. I think you’re talking about what should be *re*compiled when a module has changed, right? What I was mentioning above holds for one-shot compilation of a set of modules, but you’re right that things are trickier when it comes to incrementally changing part of that module set. Thanks, Ludo’. ^ permalink raw reply [flat|nested] 16+ messages in thread
end of thread, other threads:[~2012-03-16 11:12 UTC | newest] Thread overview: 16+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2012-03-05 10:27 Autocompilation/LilyPond David Kastrup 2012-03-05 13:45 ` Autocompilation/LilyPond Mark H Weaver 2012-03-05 14:39 ` Autocompilation/LilyPond David Kastrup 2012-03-05 17:57 ` Autocompilation/LilyPond Mark H Weaver 2012-03-06 1:03 ` Autocompilation/LilyPond David Kastrup 2012-03-06 2:36 ` Autocompilation/LilyPond Mark H Weaver 2012-03-07 23:51 ` Autocompilation/LilyPond David Kastrup 2012-03-09 16:19 ` Autocompilation/LilyPond Mark H Weaver 2012-03-09 16:55 ` Autocompilation/LilyPond David Kastrup 2012-03-09 18:57 ` Autocompilation/LilyPond Mark H Weaver 2012-03-09 19:27 ` Autocompilation/LilyPond David Kastrup 2012-03-10 12:24 ` Autocompilation/LilyPond Ian Hulin 2012-03-10 13:05 ` Autocompilation/LilyPond David Kastrup 2012-03-10 22:41 ` Autocompilation/LilyPond Ludovic Courtès 2012-03-10 23:15 ` Autocompilation/LilyPond David Kastrup 2012-03-16 11:12 ` Autocompilation/LilyPond Ludovic Courtès
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).