Results of tests of guile-2.9

Results of tests of guile-2.9

[Stefan Israelsson Tampe]

[-- Attachment #1: Type: text/plain, Size: 862 bytes --]

Hi,

I've taken 2.9 on a ride with my active code bases, guile-log,
guile-syntax-parse and python-on -guile.

Generelly it's a pleasure as always to work with guile. I can compile all
code and especially the
clpfd code in the prolog part works out nicely (A huge file that takes now
6minutes to compile
a 10MB go file).

What's was surprising was that guile-syntax-parse code is much much quicker
than before. Especially the compilation of the racket match.scm file has
improved quite a lot. Prolog code
compiles more quick as well which is very much appriciated.

A few hickups.

1. compiling with O0 fails for me because guile does not translate all
letrec forms down to lover level forms.

2. If you put a lambda as a syntax value and try to execute that code you
get an unhelpful message down in the compiler tower, (types.scm).

Else super work and thanks!

[-- Attachment #2: Type: text/html, Size: 1098 bytes --]

Re: Results of tests of guile-2.9

[Mikael Djurfeldt]

[-- Attachment #1: Type: text/plain, Size: 1117 bytes --]

That sounds great! Can you say something about how much quicker 2.9.1 is
compared to 2.2?

Den sön 11 nov. 2018 21:53 skrev Stefan Israelsson Tampe <
stefan.itampe@gmail.com>:

> Hi,
>
> I've taken 2.9 on a ride with my active code bases, guile-log,
> guile-syntax-parse and python-on -guile.
>
> Generelly it's a pleasure as always to work with guile. I can compile all
> code and especially the
> clpfd code in the prolog part works out nicely (A huge file that takes now
> 6minutes to compile
> a 10MB go file).
>
> What's was surprising was that guile-syntax-parse code is much much
> quicker than before. Especially the compilation of the racket match.scm
> file has improved quite a lot. Prolog code
> compiles more quick as well which is very much appriciated.
>
> A few hickups.
>
> 1. compiling with O0 fails for me because guile does not translate all
> letrec forms down to lover level forms.
>
> 2. If you put a lambda as a syntax value and try to execute that code you
> get an unhelpful message down in the compiler tower, (types.scm).
>
> Else super work and thanks!
>

[-- Attachment #2: Type: text/html, Size: 1521 bytes --]

Re: Results of tests of guile-2.9

[Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 3058 bytes --]


Mikael Djurfeldt <mikael@djurfeldt.com> writes:

> That sounds great! Can you say something about how much quicker 2.9.1 is
> compared to 2.2?

You can find that by looking at the benchmarks by ecraven:

https://ecraven.github.io/r7rs-benchmarks/

Comparing 2.9 and 2.2 there, you see a 30% reduction in execution time
(averaged by geometric mean).

In other words: Guile 2.9 is about 50% faster than Guile 2.2.

And looking at details, Guile 2.9 is faster than Guile 2.2 in almost
every test, and it did not experience significant slowdown in any test.


I also did a summary of all the results which shows that Guile is
closing in on MIT-scheme and chicken (but you’ll notice that the
ordering is very different from the one on the site, which shows nicely
that your result depends on what you look at, and how you look — and
you can see that they have very different performance characteristics):


The first number is the geometric mean of the slowdown against the
fastest implementation in each test. The number in parens is the
number of successful tests.

1.9881572085609192 (38 / 38) stalin-unknown
2.1500822915753734 (57 / 57) chez-9.5.1-m64
2.738525957787122 (55 / 55) gambitc-v4.9.0
2.7694443820075634 (55 / 55) gerbil-v0.14-DEV
4.205151966183653 (50 / 50) bigloo-4.3a
5.442681840154815 (57 / 57) larceny-1.3
5.707385688762197 (57 / 57) racket-7.0/r7rs
8.679978781946975 (50 / 50) chicken-4.13.0
9.248983537329178 (51 / 51) mit-9.2.1
10.587408686012083 (55 / 55) guile-2.9.1.3-1f678
10.615583087968362 (41 / 41) bones-unknown
11.524752498102057 (56 / 56) cyclone-0.9.3
14.448014458884698 (57 / 57) petite-9.5.1-m64
15.089971411932236 (56 / 56) guile-2.2.4
18.035143748368437 (45 / 45) ypsilon-unknown
19.005148516339332 (44 / 44) femtolisp-unknown
19.139543005333042 (56 / 56) gauche-0.9.6
27.645742735331833 (57 / 57) sagittarius-0.9.2
31.157381722908422 (36 / 36) rscheme-unknown
34.153836451059746 (39 / 39) scheme48-unknown
36.48670680531284 (41 / 41) picrin-unknown
38.99165232121692 (48 / 48) kawa-3.0
47.53962620985255 (28 / 28) rhizome-unknown
55.19456628888817 (11 / 11) s9fes-unknown
64.80503623166697 (35 / 35) SISC-1.16.6
86.0140998934114 (48 / 48) chibi-unknown
109.67746150832924 (35 / 35) chickencsi-4.13.0
180.3672988266313 (17 / 17) foment-0.4

created with

`for i in bigloo-4.3a bones-unknown chez-9.5.1-m64 chibi-unknown chicken-4.13.0 chickencsi-4.13.0 cyclone-0.9.3 femtolisp-unknown foment-0.4 gambitc-v4.9.0 gauche-0.9.6 gerbil-v0.14-DEV guile-2.2.4 guile-2.9.1.3-1f678 ironscheme kawa-3.0 larceny-1.3 mit-9.2.1 petite-9.5.1-m64 picrin-unknown racket-7.0/r7rs rhizome-unknown rscheme-unknown s9fes-unknown sagittarius-0.9.2 scheme48-unknown SISC-1.16.6 stalin-unknown tinyscheme ypsilon-unknown; do echo $(./evaluate-r7rs-benchmark.w /tmp/all.csv $i | tail -n 1) $i; done | sort -g`

using https://bitbucket.org/ArneBab/wisp/src/tip/examples/evaluate-r7rs-benchmark.w

Best wishes,
Arne
--
Unpolitisch sein
heißt politisch sein
ohne es zu merken

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 1076 bytes --]

Re: Results of tests of guile-2.9

[Mikael Djurfeldt]

[-- Attachment #1: Type: text/plain, Size: 3309 bytes --]

Thanks, Arne!

Den mån 12 nov. 2018 01:04 skrev Arne Babenhauserheide <arne_bab@web.de>:

>
> Mikael Djurfeldt <mikael@djurfeldt.com> writes:
>
> > That sounds great! Can you say something about how much quicker 2.9.1 is
> > compared to 2.2?
>
> You can find that by looking at the benchmarks by ecraven:
>
> https://ecraven.github.io/r7rs-benchmarks/
>
> Comparing 2.9 and 2.2 there, you see a 30% reduction in execution time
> (averaged by geometric mean).
>
> In other words: Guile 2.9 is about 50% faster than Guile 2.2.
>
> And looking at details, Guile 2.9 is faster than Guile 2.2 in almost
> every test, and it did not experience significant slowdown in any test.
>
>
> I also did a summary of all the results which shows that Guile is
> closing in on MIT-scheme and chicken (but you’ll notice that the
> ordering is very different from the one on the site, which shows nicely
> that your result depends on what you look at, and how you look — and
> you can see that they have very different performance characteristics):
>
>
> The first number is the geometric mean of the slowdown against the
> fastest implementation in each test. The number in parens is the
> number of successful tests.
>
> 1.9881572085609192 (38 / 38) stalin-unknown
> 2.1500822915753734 (57 / 57) chez-9.5.1-m64
> 2.738525957787122 (55 / 55) gambitc-v4.9.0
> 2.7694443820075634 (55 / 55) gerbil-v0.14-DEV
> 4.205151966183653 (50 / 50) bigloo-4.3a
> 5.442681840154815 (57 / 57) larceny-1.3
> 5.707385688762197 (57 / 57) racket-7.0/r7rs
> 8.679978781946975 (50 / 50) chicken-4.13.0
> 9.248983537329178 (51 / 51) mit-9.2.1
> 10.587408686012083 (55 / 55) guile-2.9.1.3-1f678
> 10.615583087968362 (41 / 41) bones-unknown
> 11.524752498102057 (56 / 56) cyclone-0.9.3
> 14.448014458884698 (57 / 57) petite-9.5.1-m64
> 15.089971411932236 (56 / 56) guile-2.2.4
> 18.035143748368437 (45 / 45) ypsilon-unknown
> 19.005148516339332 (44 / 44) femtolisp-unknown
> 19.139543005333042 (56 / 56) gauche-0.9.6
> 27.645742735331833 (57 / 57) sagittarius-0.9.2
> 31.157381722908422 (36 / 36) rscheme-unknown
> 34.153836451059746 (39 / 39) scheme48-unknown
> 36.48670680531284 (41 / 41) picrin-unknown
> 38.99165232121692 (48 / 48) kawa-3.0
> 47.53962620985255 (28 / 28) rhizome-unknown
> 55.19456628888817 (11 / 11) s9fes-unknown
> 64.80503623166697 (35 / 35) SISC-1.16.6
> 86.0140998934114 (48 / 48) chibi-unknown
> 109.67746150832924 (35 / 35) chickencsi-4.13.0
> 180.3672988266313 (17 / 17) foment-0.4
>
> created with
>
> `for i in bigloo-4.3a bones-unknown chez-9.5.1-m64 chibi-unknown
> chicken-4.13.0 chickencsi-4.13.0 cyclone-0.9.3 femtolisp-unknown foment-0.4
> gambitc-v4.9.0 gauche-0.9.6 gerbil-v0.14-DEV guile-2.2.4
> guile-2.9.1.3-1f678 ironscheme kawa-3.0 larceny-1.3 mit-9.2.1
> petite-9.5.1-m64 picrin-unknown racket-7.0/r7rs rhizome-unknown
> rscheme-unknown s9fes-unknown sagittarius-0.9.2 scheme48-unknown
> SISC-1.16.6 stalin-unknown tinyscheme ypsilon-unknown; do echo
> $(./evaluate-r7rs-benchmark.w /tmp/all.csv $i | tail -n 1) $i; done | sort
> -g`
>
> using
> https://bitbucket.org/ArneBab/wisp/src/tip/examples/evaluate-r7rs-benchmark.w
>
> Best wishes,
> Arne
> --
> Unpolitisch sein
> heißt politisch sein
> ohne es zu merken
>

[-- Attachment #2: Type: text/html, Size: 3995 bytes --]

Re: Results of tests of guile-2.9

[Mikael Djurfeldt]

[-- Attachment #1: Type: text/plain, Size: 3670 bytes --]

It would be nice to have guile-1.8 in that list since some users stayed at
that version due to 2.0 being slower.

Maybe, in time, we can get everyone back to the most recent release. :-))

Den tis 13 nov. 2018 00:48 skrev Mikael Djurfeldt <mikael@djurfeldt.com>:

> Thanks, Arne!
>
> Den mån 12 nov. 2018 01:04 skrev Arne Babenhauserheide <arne_bab@web.de>:
>
>>
>> Mikael Djurfeldt <mikael@djurfeldt.com> writes:
>>
>> > That sounds great! Can you say something about how much quicker 2.9.1 is
>> > compared to 2.2?
>>
>> You can find that by looking at the benchmarks by ecraven:
>>
>> https://ecraven.github.io/r7rs-benchmarks/
>>
>> Comparing 2.9 and 2.2 there, you see a 30% reduction in execution time
>> (averaged by geometric mean).
>>
>> In other words: Guile 2.9 is about 50% faster than Guile 2.2.
>>
>> And looking at details, Guile 2.9 is faster than Guile 2.2 in almost
>> every test, and it did not experience significant slowdown in any test.
>>
>>
>> I also did a summary of all the results which shows that Guile is
>> closing in on MIT-scheme and chicken (but you’ll notice that the
>> ordering is very different from the one on the site, which shows nicely
>> that your result depends on what you look at, and how you look — and
>> you can see that they have very different performance characteristics):
>>
>>
>> The first number is the geometric mean of the slowdown against the
>> fastest implementation in each test. The number in parens is the
>> number of successful tests.
>>
>> 1.9881572085609192 (38 / 38) stalin-unknown
>> 2.1500822915753734 (57 / 57) chez-9.5.1-m64
>> 2.738525957787122 (55 / 55) gambitc-v4.9.0
>> 2.7694443820075634 (55 / 55) gerbil-v0.14-DEV
>> 4.205151966183653 (50 / 50) bigloo-4.3a
>> 5.442681840154815 (57 / 57) larceny-1.3
>> 5.707385688762197 (57 / 57) racket-7.0/r7rs
>> 8.679978781946975 (50 / 50) chicken-4.13.0
>> 9.248983537329178 (51 / 51) mit-9.2.1
>> 10.587408686012083 (55 / 55) guile-2.9.1.3-1f678
>> 10.615583087968362 (41 / 41) bones-unknown
>> 11.524752498102057 (56 / 56) cyclone-0.9.3
>> 14.448014458884698 (57 / 57) petite-9.5.1-m64
>> 15.089971411932236 (56 / 56) guile-2.2.4
>> 18.035143748368437 (45 / 45) ypsilon-unknown
>> 19.005148516339332 (44 / 44) femtolisp-unknown
>> 19.139543005333042 (56 / 56) gauche-0.9.6
>> 27.645742735331833 (57 / 57) sagittarius-0.9.2
>> 31.157381722908422 (36 / 36) rscheme-unknown
>> 34.153836451059746 (39 / 39) scheme48-unknown
>> 36.48670680531284 (41 / 41) picrin-unknown
>> 38.99165232121692 (48 / 48) kawa-3.0
>> 47.53962620985255 (28 / 28) rhizome-unknown
>> 55.19456628888817 (11 / 11) s9fes-unknown
>> 64.80503623166697 (35 / 35) SISC-1.16.6
>> 86.0140998934114 (48 / 48) chibi-unknown
>> 109.67746150832924 (35 / 35) chickencsi-4.13.0
>> 180.3672988266313 (17 / 17) foment-0.4
>>
>> created with
>>
>> `for i in bigloo-4.3a bones-unknown chez-9.5.1-m64 chibi-unknown
>> chicken-4.13.0 chickencsi-4.13.0 cyclone-0.9.3 femtolisp-unknown foment-0.4
>> gambitc-v4.9.0 gauche-0.9.6 gerbil-v0.14-DEV guile-2.2.4
>> guile-2.9.1.3-1f678 ironscheme kawa-3.0 larceny-1.3 mit-9.2.1
>> petite-9.5.1-m64 picrin-unknown racket-7.0/r7rs rhizome-unknown
>> rscheme-unknown s9fes-unknown sagittarius-0.9.2 scheme48-unknown
>> SISC-1.16.6 stalin-unknown tinyscheme ypsilon-unknown; do echo
>> $(./evaluate-r7rs-benchmark.w /tmp/all.csv $i | tail -n 1) $i; done | sort
>> -g`
>>
>> using
>> https://bitbucket.org/ArneBab/wisp/src/tip/examples/evaluate-r7rs-benchmark.w
>>
>> Best wishes,
>> Arne
>> --
>> Unpolitisch sein
>> heißt politisch sein
>> ohne es zu merken
>>
>

[-- Attachment #2: Type: text/html, Size: 4614 bytes --]

Re: Results of tests of guile-2.9

[Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 489 bytes --]


Mikael Djurfeldt <mikael@djurfeldt.com> writes:

> It would be nice to have guile-1.8 in that list since some users stayed at
> that version due to 2.0 being slower.

I did a first step towards that: making the benchmarking suite work with
Guile 1.8.8: https://github.com/ecraven/r7rs-benchmarks/pull/38

I hope ecraven can pick up from that. But you could run your own tests
with it, too.

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein
ohne es zu merken

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 1076 bytes --]

Re: Results of tests of guile-2.9

[Thomas Morley]

Am Di., 13. Nov. 2018 um 01:12 Uhr schrieb Mikael Djurfeldt
<mikael@djurfeldt.com>:
>
> It would be nice to have guile-1.8 in that list since some users stayed at that version due to 2.0 being slower.

As does LilyPond...

> Maybe, in time, we can get everyone back to the most recent release. :-))

I wrote some own test-code.
To be absolutely clear, this code is silly and terrible and
accumulates expensive procedures done again and again..
If I would see something like that I'd usually shout: "Man, are you crazy??"

Nevertheless, I post it here (and the results) to expose the
string-problem, which LilyPond still fights with (among others).

(begin
  (use-modules (srfi srfi-1))

  (let ((ls '()))
    (for-each
      (lambda (x)
        (set! ls
          (cons
            (string-append "'" (object->string (eval-string "'(a b c)")))
            ls
            )))
        (iota 10000))
    (set! ls (delete-duplicates ls))
    (write (version))))

Invoked with
$ time guile-whatever-version path/to/file

"1.8.8"
real    0m0,471s
user    0m0,446s
sys    0m0,018s

"2.0.14"
real    0m3,141s
user    0m3,595s
sys    0m0,111s

"2.2.4.9-71f536"
real    0m1,406s
user    0m1,633s
sys    0m0,063s


"2.9.1"
real    0m1,385s
user    0m1,615s
sys    0m0,033s


For this code guilev1 beats all others, although guile-2.9.1 is much
faster than 2.0.14

As said, it's a worse test, exposing a guilev2 weakness.
And ofcourse not the only problem LilyPond is facing with guilev2.
I do have a working guile-2.9.1/LilyPond-setup and results for
real-world ly-files are usually not that dramatically, but still big.


Otoh, I observe a steady improvement from 2.0.14 to 2.2.4 and the most
recent 2.9.1

So I'm eager to see how the final 3.0 will work.

Thanks,
  Harm

Re: Results of tests of guile-2.9

[Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 2150 bytes --]


Thomas Morley <thomasmorley65@gmail.com> writes:

> Am Di., 13. Nov. 2018 um 01:12 Uhr schrieb Mikael Djurfeldt
> <mikael@djurfeldt.com>:
>>
>> It would be nice to have guile-1.8 in that list since some users stayed at that version due to 2.0 being slower.
>
> As does LilyPond...

Yes — and that’s one of the things which I see as very important to change.

>> Maybe, in time, we can get everyone back to the most recent release. :-))
>
> I wrote some own test-code.

This is what I see in this test, though with an added byte-compilation step:

cat /tmp/string-append.scm; GUILE=~/Quell/Programme/guile/meta/guile ~/Quell/Programme/guile/meta/guild compile /tmp/string-append.scm ; time ~/Quell/Programme/guile/meta/guile /tmp/string-append.scm; guild compile /tmp/string-append.scm; time guile /tmp/string-append.scm; time ~/Quell/Programme/guile-1.8.8/guile-1.8.8/pre-inst-guile /tmp/string-append.scm
(begin
  (use-modules (srfi srfi-1))

  (let ((ls '()))
    (for-each
      (lambda (x)
        (set! ls
          (cons
            (string-append "'" (object->string (eval-string "'(a b c)")))
            ls
            )))
        (iota 10000))
    (set! ls (delete-duplicates ls))
    (write (version))))
wrote `/home/arne/Quell/Programme/guile/cache/guile/ccache/3.0-LE-8-4.1/tmp/string-append.scm.go'
"UNKNOWN"
real    0m0,660s
user    0m0,599s
sys     0m0,024s
wrote `/home/arne/.cache/guile/ccache/2.2-LE-8-3.A/tmp/string-append.scm.go'
"2.2.2"
real    0m0,529s
user    0m0,498s
sys     0m0,014s
"1.8.8"
real    0m0,530s
user    0m0,504s
sys     0m0,010s

UNKNOWN is v2.9.1

> As said, it's a worse test, exposing a guilev2 weakness.
> And ofcourse not the only problem LilyPond is facing with guilev2.
> I do have a working guile-2.9.1/LilyPond-setup

Very cool!

> and results for
> real-world ly-files are usually not that dramatically, but still big.

> Otoh, I observe a steady improvement from 2.0.14 to 2.2.4 and the most
> recent 2.9.1

Sounds good! Thank you for testing!

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein
ohne es zu merken

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 1076 bytes --]

Re: Results of tests of guile-2.9

[Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 2402 bytes --]


Thomas Morley <thomasmorley65@gmail.com> writes:

> Am Di., 13. Nov. 2018 um 01:12 Uhr schrieb Mikael Djurfeldt
> <mikael@djurfeldt.com>:
>>
>> It would be nice to have guile-1.8 in that list since some users stayed at that version due to 2.0 being slower.
>
> As does LilyPond...

Initial results:

~/wisp/examples $ ./evaluate-r7rs-benchmark.w ~/Quell/Programme/r7rs-benchmarks/all.csv guile-1.8.8

=== Best times ===

(("browse:2000" . 28.757008189)
 ("deriv:10000000" . 41.676202123)
 ("destruc:600:50:4000" . 22.127862312)
 ("diviter:1000:1000000" . 28.267739825)
 ("divrec:1000:1000000" . 44.214388322)
 ("ntakl:40:20:12:1" . 48.965012737)
 ("puzzle:1000" . 43.052130699)
 ("tak:40:20:11:1" . 23.331631568)
 ("takl:40:20:12:1" . 49.317768584)
 ("triangl:22:1:50" . 24.260957572))

=== Guile-1.8.8 times ===

(("browse:2000" "426.84")
 ("deriv:10000000" "331.99")
 ("diviter:1000:1000000" "426.45")
 ("puzzle:1000" "617.5")
 ("tak:40:20:11:1" "799.41")
 ("takl:40:20:12:1" "2154.59")
 ("triangl:22:1:50" "528.44"))

=== Guile-1.8.8 slowdown ===

(43.68790522892811
 34.26292746266462
 21.781498048118348
 14.343076404679387
 15.086101776798138
 7.965936987736784
 14.842990522333713)

=== Guile-1.8.8 Geometric Mean slowdown (successful tests / total tests) ===

18.813276420052787 (7 / 7)

missing results for 1.8.8 are those which did not work for 1.8.8, but
did for 2.9.1

For these first tests Guile 1.8.8 is a LOT slower than 2.9.1. I would
expect later tests to have more varied results.


Data in all.csv created with

r7rs-benchmarks $ for i in browse deriv destruc diviter divrec puzzle triangl tak takl ntakl cpstak ctak fib fibc fibfp sum sumfp fft mbrot mbrotZ nucleic pi pnpoly ray simplex ack array1 string sum1 cat tail wc read1 compiler conform dynamic earley graphs lattice matrix maze mazefun nqueens paraffins parsing peval primes quicksort scheme slatex chudnovsky nboyer sboyer gcbench mperm equal bv2string; do CPU_LIMIT=3600 GUILE=~/Quell/Programme/guile/meta/guile GUILD=~/Quell/Programme/guile/meta/guild bash -x ./bench guile $i; CPU_LIMIT=3600 GUILE=~/Quell/Programme/guile-1.8.8/guile-1.8.8/pre-inst-guile GUILD=echo bash -x ./bench guile $i; done

grep -a -h '+!CSVLINE' results.* | sed 's/+!CSVLINE!+//' > all.csv


Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein
ohne es zu merken

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 1076 bytes --]