FWD: Test reader speed of Guile 3.0.6 in Lilypond

FWD: Test reader speed of Guile 3.0.6 in Lilypond

[Dr. Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 3061 bytes --]

Hi,

due to the reader-rewrite, I asked in the lilypond-list for testing,
because they are likely the strongest power-user of the reader.

Han-Wen Nienhuys checked performance in the benchmark suite:

> $ cat ../bench.ly
> #(define (microseconds)
>   (let* ((t (gettimeofday))
> (us (/ (cdr t) 1000000.0)))
>    (+ (car t) us)))
>
> #(define start (microseconds))
>
> % \include "bench-largeexp.ly"
> \include "bench-manysmall.ly"
>
> #(newline)
> #(display (- (microseconds) start))
>
> Parsing & evaluating '(1 2 3) 200 times.
> - guile 1.8: 1.25ms
> - guile 2.2: 3.2ms
> - guile 3.0.6: 2.08ms
>
> Parsing & evaluating the giant expression in define-grobs.scm once:
>
> - guile 1.8: 10.6ms
> - guile 2.2: 166ms
> - guile 3.0.6: 71ms
>
> Parsing & evaluating the giant expression in define-grobs.scm once
> (but quoted, ie. not real evaluation):
>
> - guile 1.8: 10.0ms
> - guile 2.2: 13ms
> - guile 3.0.6: 12.8ms
>
> In summary, the read speed itself for large expressions is on the same
> order as 1.8, but for many small expressions (which is a much more
> common use-case) there is still a 60% slowdown.

>> asking here because I want to avoid surprising and avoidable changes
>> that block Lilypond. I consider Lilypond to be the most important
>> flagship project of Guile, and I want to do what I can to prevent
>> unnecessary friction.

> I appreciate the heads up you gave here today, but from our side, it
> doesn't seem like the Guile project is much concerned with our needs.
> The evaluation speed of GUILE 3.x is still pretty poor. Having fast,
> JIT'ed code seems interesting in theory, but the way it's implemented
> in Guile 3.x is a giant headache: the separate byte compilation is
> extremely slow, and it is hard to manage (where should the .go files
> be stored/installed, how/when are they generated etc.). It also
> doesn't match our use case, because a lot of the code that we have
> comes from .ly files, so it cannot be precompiled.

Are there ways to improve this? For example sticking to the baseline
compiler (as described in the wingolog:
https://wingolog.org/archives/2020/06/03/a-baseline-compiler-for-guile )

>> The article linked above shows that setting -O1 as optimization of the
>> code could help (if you’re not already doing that).

> The article gives a pointer to the code, which is
> module/language/tree-il/compile-bytecode.scm. I ran
> 
> for c in  $(git log module/language/tree-il/compile-bytecode.scm|grep
> commit|awk '{print $2;}'); do git show $c ; done | grep -i doc
> 
> to look for documentation, but couldn't find it. The module has one
> exported symbol, which is compile-bytecode.
> 
> Could you give some practical tips on how we'd use this?

Can someone help with this?

It would be pretty cool if there were a simple way to speed-up lilypond
like changing optimization level that just got lost in communication (or
missing docs).

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein
ohne es zu merken

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 1125 bytes --]

Re: FWD: Test reader speed of Guile 3.0.6 in Lilypond

[Dr. Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 4335 bytes --]

More tests were done by Thomas Morley, which also give a perspective on
the initial very negative reaction by the lilypond developers, as it
shows the abysmal speed that initial versions of Guile2-Lilypond
integration had.

Thomas Morley <thomasmorley65@gmail.com> writes:

> Am Fr., 12. März 2021 um 10:45 Uhr schrieb Thomas Morley
> <thomasmorley65@gmail.com>:
>>
>> Am Do., 11. März 2021 um 22:47 Uhr schrieb Dr. Arne Babenhauserheide
>
>> > Do you already have results?
>>
>> Currently my regular job eats up all my time.
>> Probably I'll find some free time at upcoming weekend, not sure though...
>> If so, to which guile-version should I compare, guile-1.8 (which is
>> still LilyPond-default) or a certain guile-2/3-version?
>
> Found some time today...
>
> I tested a huge ly-score, once sent by Jean-Charles for testing
> different guile-versions:
> https://lists.gnu.org/archive/html/lilypond-user/2016-11/msg00948.html
> Thanks again!
> I made it compilable with recent ly-master out of
> 55ca32d951dd17556df8d0332b861f0cd189379a
> (though I didn't fix all warnings and programming errors)
>
> Tested with these guile versions:
> guile-1.8.8 from tarball
> guile-2-2-6 out of a69b567d97f7c9193924c775e1dd86e43a35b8bd
> guile-3.0.6 from master, i.e. out of 85433fc2b122dc78342c3c83941949d1d9318399
> (although (version) says: 3.0.5.116-85433)
>
> There are some guile-warnings, multiple instances of:
> WARNING: (#{ g478}#): `smallCaps-markup' imported from both (lily) and
> (#{ g171}#)
> WARNING: (scm framework-ps): imported module (lily) overrides core binding `_'
> The latter one is about gettext and PITA.
>
>
> I compiled Urtext.ly from the tarball linked above (giving a 158 pages pdf):
>
> %%%%%%%%%%%%%%%%%%%%
> guile-1.8.8
>
> real    4m37,304s
> user    4m14,144s
> sys    0m7,717s
>
> %%%%%%%%%%%%%%%%%%%%
> guile-2.2.6, (version) says: 2.2.6.7-9a11b
>
> real    7m18,995s
> user    6m41,808s
> sys    0m11,960s
>
> eating much more memory than guile-1.8.8
>
> %%%%%%%%%%%%%%%%%%%%
> guile-3.0.6, (version) says: 3.0.5.116-85433
>
> real    5m49,587s
> user    5m11,033s
> sys    0m9,488s
>
> eating a little more memory than with guile-1.8.8 but far less than guile-2.2.6
…
> I tested _one_ huge file.
> In it there's some custom-code to deal with accented letters like in
> "éèêëáàâäíìîïóòôöúùûüçœæ", which guile-1 was not able to cope with in
> many cases.
> I think guile-2/3 does a better job in this regard.
> Btw, this is my strongest argument to switch to guile-2/3
> I could eliminate that code for guile-2/3 runs and take timing values again.
> Maybe it has positive impact.
> Interested? Has to wait for the weekend, though.
> 
> Otoh, adjusting small things in a small file like finding the needed value for
> \override Grob.X-offset = FINDME
> is a pain with guile-2/3 because of the already mentioned startup delay.
> The strongest argument against switching to guile-2/3, imho.
…
> %%%%%%%%%%%%%%%%%%%%
> %%%%%%%%%%%%%%%%%%%%
>
> Interestingly I found some older timing values from former tests:
>
> %%%%%%%%%%%%%%%%%%%%
> released lilypond-2.19.52 with guile-1.8.7
>
> real    8m16.191s
> user    6m39.864s
> sys    0m10.860s
>
> %%%%%%%%%%%%%%%%%%%%
> with guile-versions from 2.0.14 and 2.1.7 I noted:
>
> Best result was:
> real    25m17.082s
> user    31m1.668s
> sys    0m5.480s
>
> Worst result was:
> real    77m44.348s
> user    100m40.324s
> sys    0m18.932s
>
> Though, my noted former results are a little messy: I can't say
> anymore which guile-version with which lilypond-patches was
> best/worst.
>
> Anyway, both, LilyPond and guile seem to have improved a lot.

> A: Did you do these tests on the same machine?

> Yes, it was all on the same machine.
> Though, guile-2.0.12 was the first version with a _chance_ to compile
> LilyPond, iirc
> These early attempts used patches which nowadays are superseeded by
> far better ones.
> So while the machine is the same, LilyPond has become far better to
> deal with guile-2/3.
> I think those scary values are documenting where we started and are of
> historic interest, nothing more.

> Cheers,
>   Harm


-- 
Unpolitisch sein
heißt politisch sein
ohne es zu merken

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 1125 bytes --]

Fwd: Re: Test reader speed of Guile 3.0.6

[Dr. Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 481 bytes --]


Knut Petersen:

>> I wonder what the Carver MSDM score would look like. Searching the archives, it's seemingly become an informal standard for large-project stress tests.
>
> Carver MDSM compiled with guile master and lilypond master: 19.7 seconds
> Carver MDSM compiled with guile 1.8 + lilypond master: 14.6 seconds
> For the Carver MDSM we see a performance penalty of only about 35% for guile master.

-- 
Unpolitisch sein
heißt politisch sein
ohne es zu merken

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 1125 bytes --]

Re: FWD: Test reader speed of Guile 3.0.6 in Lilypond

[Dr. Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 1223 bytes --]

Hi,

I’d like to ask again if anyone can help the Lilypond folks with setting
optimization settings for code they compile from lilypond-files:

> Are there ways to improve this? For example sticking to the baseline
> compiler (as described in the wingolog:
> https://wingolog.org/archives/2020/06/03/a-baseline-compiler-for-guile )
>
>>> The article linked above shows that setting -O1 as optimization of the
>>> code could help (if you’re not already doing that).
>
>> The article gives a pointer to the code, which is
>> module/language/tree-il/compile-bytecode.scm. I ran
>> 
>> for c in  $(git log module/language/tree-il/compile-bytecode.scm|grep
>> commit|awk '{print $2;}'); do git show $c ; done | grep -i doc
>> 
>> to look for documentation, but couldn't find it. The module has one
>> exported symbol, which is compile-bytecode.
>> 
>> Could you give some practical tips on how we'd use this?
>
> Can someone help with this?
>
> It would be pretty cool if there were a simple way to speed-up lilypond
> like changing optimization level that just got lost in communication (or
> missing docs).

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein
ohne es zu merken

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 1125 bytes --]