* elisp-benchmarks
@ 2022-02-09 22:19 Stefan Monnier
2022-02-10 6:50 ` elisp-benchmarks Lars Ingebrigtsen
` (2 more replies)
0 siblings, 3 replies; 9+ messages in thread
From: Stefan Monnier @ 2022-02-09 22:19 UTC (permalink / raw)
To: emacs-devel
Hi,
I just ran the elisp-benchmarks (with a repeat count of 10) on some
little-used server we have here and figured others might like to see the
result:
Before sympos (i.e. commit f57f28935a1f0c46776976dd497322a07d248f5f)
| test | non-gc avg (s) | gc avg (s) | gcs avg | tot avg (s) | tot avg err (s) |
|--------------------+----------------+------------+---------+-------------+-----------------|
| bubble | 6.75 | 8.75 | 433 | 15.50 | 0.03 |
| bubble-no-cons | 16.26 | 0.06 | 3 | 16.32 | 0.00 |
| bytecomp | 3.24 | 2.53 | 123 | 5.77 | 0.16 |
| dhrystone | 15.54 | 0.00 | 0 | 15.54 | 0.00 |
| eieio | 5.57 | 3.38 | 166 | 8.95 | 0.02 |
| fibn | 5.45 | 0.00 | 0 | 5.45 | 0.00 |
| fibn-named-let | 6.00 | 0.00 | 0 | 6.00 | 0.01 |
| fibn-rec | 11.11 | 0.00 | 0 | 11.11 | 0.01 |
| fibn-tc | 9.24 | 0.00 | 0 | 9.24 | 0.00 |
| flet | 18.50 | 0.00 | 0 | 18.50 | 0.01 |
| inclist | 18.11 | 0.00 | 0 | 18.11 | 0.01 |
| inclist-type-hints | 18.11 | 0.00 | 0 | 18.11 | 0.01 |
| listlen-tc | 9.33 | 0.00 | 0 | 9.33 | 0.02 |
| map-closure | 14.49 | 0.00 | 0 | 14.49 | 0.01 |
| nbody | 5.98 | 16.34 | 791 | 22.32 | 0.06 |
| pack-unpack | 1.07 | 0.87 | 43 | 1.94 | 0.00 |
| pack-unpack-old | 3.29 | 1.76 | 87 | 5.06 | 0.01 |
| pcase | 20.28 | 0.00 | 0 | 20.28 | 0.00 |
| pidigits | 11.96 | 10.17 | 401 | 22.13 | 0.18 |
| scroll | 1.34 | 0.12 | 6 | 1.46 | 0.00 |
| smie | 3.06 | 0.95 | 46 | 4.00 | 0.01 |
|--------------------+----------------+------------+---------+-------------+-----------------|
| total | 204.67 | 44.94 | 2102 | 249.61 | 0.26 |
Right after sympos (i.e. commit df49e3a3ab4cddf1e3c0f5482c7fdd809d8a8884)
| test | non-gc avg (s) | gc avg (s) | gcs avg | tot avg (s) | tot avg err (s) |
|--------------------+----------------+------------+---------+-------------+-----------------|
| bubble | 6.39 | 9.09 | 449 | 15.48 | 0.04 |
| bubble-no-cons | 16.50 | 0.06 | 3 | 16.56 | 0.00 |
| bytecomp | 3.43 | 2.69 | 130 | 6.12 | 0.16 |
| dhrystone | 16.07 | 0.00 | 0 | 16.07 | 0.01 |
| eieio | 5.50 | 3.38 | 166 | 8.88 | 0.02 |
| fibn | 5.42 | 0.00 | 0 | 5.42 | 0.00 |
| fibn-named-let | 5.87 | 0.00 | 0 | 5.87 | 0.00 |
| fibn-rec | 11.27 | 0.00 | 0 | 11.27 | 0.01 |
| fibn-tc | 9.42 | 0.00 | 0 | 9.42 | 0.00 |
| flet | 19.65 | 0.00 | 0 | 19.65 | 0.01 |
| inclist | 19.43 | 0.00 | 0 | 19.43 | 0.00 |
| inclist-type-hints | 19.43 | 0.00 | 0 | 19.43 | 0.00 |
| listlen-tc | 9.35 | 0.00 | 0 | 9.35 | 0.01 |
| map-closure | 14.83 | 0.00 | 0 | 14.83 | 0.05 |
| nbody | 5.78 | 16.38 | 789 | 22.15 | 0.07 |
| pack-unpack | 1.06 | 0.87 | 43 | 1.93 | 0.00 |
| pack-unpack-old | 3.32 | 1.77 | 86 | 5.09 | 0.01 |
| pcase | 17.65 | 0.00 | 0 | 17.65 | 0.00 |
| pidigits | 11.15 | 10.06 | 400 | 21.21 | 0.18 |
| scroll | 1.30 | 0.12 | 6 | 1.42 | 0.00 |
| smie | 3.11 | 0.95 | 46 | 4.06 | 0.00 |
|--------------------+----------------+------------+---------+-------------+-----------------|
| total | 205.92 | 45.37 | 2122 | 251.29 | 0.26 |
From master (i.e. commit d41a5e7e33067eb38b147ee2f8a1615f6faed7a4)
| test | non-gc avg (s) | gc avg (s) | gcs avg | tot avg (s) | tot avg err (s) |
|--------------------+----------------+------------+---------+-------------+-----------------|
| bubble | 5.00 | 9.15 | 433 | 14.15 | 0.07 |
| bubble-no-cons | 13.43 | 0.06 | 3 | 13.49 | 0.00 |
| bytecomp | 3.22 | 2.65 | 123 | 5.87 | 0.16 |
| dhrystone | 12.37 | 0.00 | 0 | 12.37 | 0.03 |
| eieio | 4.65 | 3.40 | 160 | 8.05 | 0.06 |
| fibn | 3.74 | 0.00 | 0 | 3.74 | 0.00 |
| fibn-named-let | 4.10 | 0.00 | 0 | 4.10 | 0.00 |
| fibn-rec | 7.33 | 0.00 | 0 | 7.33 | 0.00 |
| fibn-tc | 5.58 | 0.00 | 0 | 5.58 | 0.00 |
| flet | 12.55 | 0.00 | 0 | 12.55 | 0.01 |
| inclist | 17.67 | 0.00 | 0 | 17.67 | 0.01 |
| inclist-type-hints | 17.67 | 0.00 | 0 | 17.67 | 0.01 |
| listlen-tc | 6.84 | 0.00 | 0 | 6.84 | 0.01 |
| map-closure | 10.06 | 0.00 | 0 | 10.06 | 0.00 |
| nbody | 5.08 | 16.51 | 762 | 21.59 | 0.04 |
| pack-unpack | 0.86 | 0.88 | 41 | 1.74 | 0.01 |
| pack-unpack-old | 2.99 | 1.78 | 83 | 4.77 | 0.02 |
| pcase | 17.36 | 0.00 | 0 | 17.36 | 0.00 |
| pidigits | 10.32 | 9.97 | 386 | 20.28 | 0.34 |
| scroll | 1.39 | 0.13 | 6 | 1.52 | 0.00 |
| smie | 3.10 | 0.95 | 44 | 4.06 | 0.02 |
|--------------------+----------------+------------+---------+-------------+-----------------|
| total | 165.30 | 45.49 | 2044 | 210.78 | 0.39 |
`bytecomp`, `scroll`, and `smie` are benchmarks I added yesterday which
are not microbenchmarks and thus hopefully reflect "real use".
This was all with the byte-compiler and not the native compiler.
These result suggest that the symbol-with-pos changes don't introduce
a significant slowdown, even for the bytecompiler.
And we see that Matthias's recent improvements to the bytecode
interpreter do make a quite significant difference on several of those
microbenchmarks ;-), and also on the bytecompiler benchmark (offsetting
the extra work needed for the symbol-with-positions) tho they don't make
much of a difference when it comes to scrolling(with-jit-lock) or when
it comes to reindenting code with SMIE :-)
The >10% slowdown recently seen on the test suite is still a mystery
waiting for someone to figure out what's going on.
BTW, I think one thing is clear when I look at those benchmarks:
Emacs's GC is not good enough.
Stefan
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: elisp-benchmarks
2022-02-09 22:19 elisp-benchmarks Stefan Monnier
@ 2022-02-10 6:50 ` Lars Ingebrigtsen
2022-02-10 7:52 ` elisp-benchmarks Eli Zaretskii
2022-02-10 12:12 ` elisp-benchmarks Mattias Engdegård
2022-02-10 14:18 ` elisp-benchmarks Stefan Monnier
2 siblings, 1 reply; 9+ messages in thread
From: Lars Ingebrigtsen @ 2022-02-10 6:50 UTC (permalink / raw)
To: Stefan Monnier; +Cc: emacs-devel
Stefan Monnier <monnier@iro.umontreal.ca> writes:
> These result suggest that the symbol-with-pos changes don't introduce
> a significant slowdown, even for the bytecompiler.
Well, that's good news. 😀
> And we see that Matthias's recent improvements to the bytecode
> interpreter do make a quite significant difference on several of those
> microbenchmarks ;-), and also on the bytecompiler benchmark (offsetting
> the extra work needed for the symbol-with-positions)
Looks like Matthias's work has made a significant positive impact on
virtually all the tests. Great work, Matthias!
> tho they don't make much of a difference when it comes to
> scrolling(with-jit-lock) or when it comes to reindenting code with
> SMIE :-)
Darn.
> The >10% slowdown recently seen on the test suite is still a mystery
> waiting for someone to figure out what's going on.
Yes, but it sounded like Alan was chasing down oddities in
byte-compiling the ert tests, so hopefully we'll have an answer soonish.
--
(domestic pets only, the antidote for overdose, milk.)
bloggy blog: http://lars.ingebrigtsen.no
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: elisp-benchmarks
2022-02-10 6:50 ` elisp-benchmarks Lars Ingebrigtsen
@ 2022-02-10 7:52 ` Eli Zaretskii
0 siblings, 0 replies; 9+ messages in thread
From: Eli Zaretskii @ 2022-02-10 7:52 UTC (permalink / raw)
To: Lars Ingebrigtsen; +Cc: monnier, emacs-devel
> From: Lars Ingebrigtsen <larsi@gnus.org>
> Date: Thu, 10 Feb 2022 07:50:34 +0100
> Cc: emacs-devel@gnu.org
>
> > And we see that Matthias's recent improvements to the bytecode
> > interpreter do make a quite significant difference on several of those
> > microbenchmarks ;-), and also on the bytecompiler benchmark (offsetting
> > the extra work needed for the symbol-with-positions)
>
> Looks like Matthias's work has made a significant positive impact on
> virtually all the tests. Great work, Matthias!
I'm also pleased with the speedup of bytecode, but I'm worried by the
potentially negative effect of some of those changes on memory
fragmentation: we now prevent the bytecode strings from being moved by
GC (as part of compacting strings), which could prevent Emacs from
being able to return memory to the OS, and in general increase
fragmentation. It would be good to measure this effect somehow. We
had in the recent past some reports about people who have the memory
footprint of their Emacs session on the order of 5 to 10 GB, allegedly
due to large blocks of memory being trapped in a way that we cannot
give them back to the OS. I hope these changes won't make the
situation worse.
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: elisp-benchmarks
2022-02-09 22:19 elisp-benchmarks Stefan Monnier
2022-02-10 6:50 ` elisp-benchmarks Lars Ingebrigtsen
@ 2022-02-10 12:12 ` Mattias Engdegård
2022-02-10 14:13 ` elisp-benchmarks Stefan Monnier
2022-02-10 14:18 ` elisp-benchmarks Stefan Monnier
2 siblings, 1 reply; 9+ messages in thread
From: Mattias Engdegård @ 2022-02-10 12:12 UTC (permalink / raw)
To: Stefan Monnier; +Cc: Emacs-Devel
9 feb. 2022 kl. 23.19 skrev Stefan Monnier <monnier@iro.umontreal.ca>:
> And we see that Matthias's recent improvements to the bytecode
> interpreter do make a quite significant difference on several of those
> microbenchmarks ;-), and also on the bytecompiler benchmark (offsetting
> the extra work needed for the symbol-with-positions) tho they don't make
> much of a difference when it comes to scrolling(with-jit-lock) or when
> it comes to reindenting code with SMIE :-)
You are much too kind; my own macro-benchmarks indicate that we haven't quite reached status quo ante yet.
However I would like to caution against using the 'total' line of these benchmarks and in fact suggest that it be removed entirely since it can be very misleading: it is in effect tantamount to a completely arbitrary weighting of the individual benchmarks.
If we want an aggregate number that weighs all benchmark components equally, one way is to first establish a baseline, use relative changes to that baseline, and take the geometric average of those.
But that only makes sense if we value each component equally and there is no reason to do that -- many of the benchmarks measure essentially the same thing, and the mixture cannot in any way be defended as representative of any kind of practical Emacs use.
Individual benchmarks can of course be of interest: a proper presentation would be in a table where they each can be compared across different changes. Rearranging your date and normalising to before sympos, separately for timings that exclude and include GC, gives:
| | ex gc | inc gc |
| test | sympos | master | sympos | master |
|--------------------+--------+--------+--------+--------|
| bubble | 0.95 | 0.74 | 1.00 | 0.91 |
| bubble-no-cons | 1.01 | 0.83 | 1.01 | 0.83 |
| bytecomp | 1.06 | 0.99 | 1.06 | 1.02 |
| dhrystone | 1.03 | 0.80 | 1.03 | 0.80 |
| eieio | 0.99 | 0.83 | 0.99 | 0.90 |
| fibn | 0.99 | 0.69 | 0.99 | 0.69 |
| fibn-named-let | 0.98 | 0.68 | 0.98 | 0.68 |
| fibn-rec | 1.01 | 0.66 | 1.01 | 0.66 |
| fibn-tc | 1.02 | 0.60 | 1.02 | 0.60 |
| flet | 1.06 | 0.68 | 1.06 | 0.68 |
| inclist | 1.07 | 0.98 | 1.07 | 0.98 |
| inclist-type-hints | 1.07 | 0.98 | 1.07 | 0.98 |
| listlen-tc | 1.00 | 0.73 | 1.00 | 0.73 |
| map-closure | 1.02 | 0.69 | 1.02 | 0.69 |
| nbody | 0.97 | 0.85 | 0.99 | 0.97 |
| pack-unpack | 0.99 | 0.80 | 0.99 | 0.90 |
| pack-unpack-old | 1.01 | 0.91 | 1.01 | 0.94 |
| pcase | 0.87 | 0.86 | 0.87 | 0.86 |
| pidigits | 0.93 | 0.86 | 0.96 | 0.92 |
| scroll | 0.97 | 1.04 | 0.97 | 1.04 |
| smie | 1.02 | 1.01 | 1.01 | 1.01 |
which is a bit more informative (at least if you know what the benchmarks do, but otherwise it's all nonsense numbers anyway).
For the record, my own Relint benchmark (we all have our pets!) is at about 1.03 from the same baseline which is notable because it exercises a wide variety of operations, many of which weren't affected by any of the changes.
> The >10% slowdown recently seen on the test suite is still a mystery
> waiting for someone to figure out what's going on.
A config option to switch off sympos would be handy.
> BTW, I think one thing is clear when I look at those benchmarks:
> Emacs's GC is not good enough.
Indeed it's the elephant in the room. In addition, designing meaningful benchmarks that aren't GC-dominated can be tricky.
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: elisp-benchmarks
2022-02-10 12:12 ` elisp-benchmarks Mattias Engdegård
@ 2022-02-10 14:13 ` Stefan Monnier
0 siblings, 0 replies; 9+ messages in thread
From: Stefan Monnier @ 2022-02-10 14:13 UTC (permalink / raw)
To: Mattias Engdegård; +Cc: Emacs-Devel
> However I would like to caution against using the 'total' line of these
> benchmarks and in fact suggest that it be removed entirely since it can be
> very misleading: it is in effect tantamount to a completely arbitrary
> weighting of the individual benchmarks.
Definitely: I should have excised it when copy&pasting, sorry.
Those benchmarks only make sense individually and you need to know what
they do to interpret the results.
> For the record, my own Relint benchmark (we all have our pets!) is at about
> 1.03 from the same baseline which is notable because it exercises a wide
> variety of operations, many of which weren't affected by any of the changes.
I encourage you to contribute it to elisp-benchmarks.
"All it takes" is a file which defines an `elb-relint-entry` function
whose execution takes a reasonable amount of time (I'd say more than 1s
but not too much more than 10s).
Of course, with Relint there's the extra issue that it involves
a package not included in Emacs. For the SMIE test, I used `sm-c-mode`
by copying a snapshot of it into the `elisp-benchmark`, because the
intention was not to measure the performance of `sm-c-mode` but the
performance of SMIE.
Stefan
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: elisp-benchmarks
2022-02-09 22:19 elisp-benchmarks Stefan Monnier
2022-02-10 6:50 ` elisp-benchmarks Lars Ingebrigtsen
2022-02-10 12:12 ` elisp-benchmarks Mattias Engdegård
@ 2022-02-10 14:18 ` Stefan Monnier
2022-02-10 16:51 ` elisp-benchmarks Stefan Monnier
2 siblings, 1 reply; 9+ messages in thread
From: Stefan Monnier @ 2022-02-10 14:18 UTC (permalink / raw)
To: emacs-devel
> I just ran the elisp-benchmarks (with a repeat count of 10) on some
> little-used server we have here and figured others might like to see the
> result:
Further results:
`emacs-28` (i.e. commit 82e74e4559b8becd44f3e7ac0134e2baddd69921):
| test | non-gc avg (s) | gc avg (s) | gcs avg | tot avg (s) | tot avg err (s) |
|--------------------+----------------+------------+---------+-------------+-----------------|
| bubble | 5.91 | 8.70 | 439 | 14.61 | 0.03 |
| bubble-no-cons | 15.90 | 0.06 | 3 | 15.96 | 0.00 |
| bytecomp | 3.04 | 2.49 | 124 | 5.53 | 0.06 |
| dhrystone | 15.20 | 0.00 | 0 | 15.20 | 0.01 |
| eieio | 5.07 | 3.33 | 167 | 8.41 | 0.02 |
| fibn | 5.44 | 0.00 | 0 | 5.44 | 0.00 |
| fibn-named-let | 5.83 | 0.00 | 0 | 5.83 | 0.00 |
| fibn-rec | 10.86 | 0.00 | 0 | 10.86 | 0.03 |
| fibn-tc | 9.01 | 0.00 | 0 | 9.01 | 0.01 |
| flet | 17.98 | 0.00 | 0 | 17.98 | 0.03 |
| inclist | 18.16 | 0.00 | 0 | 18.16 | 0.00 |
| inclist-type-hints | 18.16 | 0.00 | 0 | 18.16 | 0.00 |
| listlen-tc | 9.06 | 0.00 | 0 | 9.06 | 0.02 |
| map-closure | 14.55 | 0.00 | 0 | 14.55 | 0.02 |
| nbody | 5.86 | 16.11 | 795 | 21.97 | 0.05 |
| pack-unpack | 1.03 | 0.86 | 43 | 1.89 | 0.01 |
| pack-unpack-old | 3.15 | 1.74 | 87 | 4.89 | 0.01 |
| pcase | 19.70 | 0.00 | 0 | 19.70 | 0.00 |
| pidigits | 10.97 | 9.88 | 403 | 20.85 | 0.13 |
| scroll | 1.28 | 0.08 | 4 | 1.37 | 0.00 |
| smie | 3.07 | 0.94 | 46 | 4.00 | 0.01 |
|--------------------+----------------+------------+---------+-------------+-----------------|
`emacs-27` (i.e. commit d7f4cc0974645cc6a295740afe85c6e21d956119)
| test | non-gc avg (s) | gc avg (s) | gcs avg | tot avg (s) | tot avg err (s) |
|--------------------+----------------+------------+---------+-------------+-----------------|
| bubble | 8.85 | 8.98 | 503 | 17.84 | 0.02 |
| bubble-no-cons | 18.36 | 0.07 | 4 | 18.43 | 0.00 |
| bytecomp | 2.87 | 2.45 | 134 | 5.32 | 0.00 |
| dhrystone | 17.04 | 0.00 | 0 | 17.04 | 0.00 |
| eieio | 5.25 | 3.67 | 204 | 8.92 | 0.01 |
| fibn | 6.26 | 0.00 | 0 | 6.26 | 0.00 |
| fibn-rec | 12.41 | 0.00 | 0 | 12.41 | 0.01 |
| fibn-tc | 9.84 | 0.00 | 0 | 9.84 | 0.01 |
| flet | 25.14 | 0.00 | 0 | 25.14 | 0.01 |
| inclist | 31.55 | 0.02 | 1 | 31.57 | 0.05 |
| inclist-type-hints | 42.77 | 0.02 | 1 | 42.79 | 0.01 |
| listlen-tc | 9.92 | 0.00 | 0 | 9.92 | 0.03 |
| map-closure | 15.90 | 0.00 | 0 | 15.90 | 0.03 |
| nbody | 6.74 | 15.23 | 839 | 21.96 | 0.04 |
| pack-unpack-old | 5.65 | 1.65 | 92 | 7.30 | 0.01 |
| pcase | 23.63 | 0.00 | 0 | 23.63 | 0.00 |
| pidigits | 10.39 | 10.10 | 457 | 20.49 | 0.19 |
| smie | 3.39 | 0.98 | 54 | 4.37 | 0.00 |
|--------------------+----------------+------------+---------+-------------+-----------------|
-- Stefan
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: elisp-benchmarks
2022-02-10 14:18 ` elisp-benchmarks Stefan Monnier
@ 2022-02-10 16:51 ` Stefan Monnier
2022-02-10 21:53 ` elisp-benchmarks Mattias Engdegård
0 siblings, 1 reply; 9+ messages in thread
From: Stefan Monnier @ 2022-02-10 16:51 UTC (permalink / raw)
To: emacs-devel
Stefan Monnier [2022-02-10 09:18:12] wrote:
>> I just ran the elisp-benchmarks (with a repeat count of 10) on some
>> little-used server we have here and figured others might like to see the
>> result:
>
> Further results:
>
> `emacs-28` (i.e. commit 82e74e4559b8becd44f3e7ac0134e2baddd69921):
>
> | test | non-gc avg (s) | gc avg (s) | gcs avg | tot avg (s) | tot avg err (s) |
> |--------------------+----------------+------------+---------+-------------+-----------------|
> | bubble | 5.91 | 8.70 | 439 | 14.61 | 0.03 |
> | bubble-no-cons | 15.90 | 0.06 | 3 | 15.96 | 0.00 |
> | bytecomp | 3.04 | 2.49 | 124 | 5.53 | 0.06 |
> | dhrystone | 15.20 | 0.00 | 0 | 15.20 | 0.01 |
> | eieio | 5.07 | 3.33 | 167 | 8.41 | 0.02 |
> | fibn | 5.44 | 0.00 | 0 | 5.44 | 0.00 |
> | fibn-named-let | 5.83 | 0.00 | 0 | 5.83 | 0.00 |
> | fibn-rec | 10.86 | 0.00 | 0 | 10.86 | 0.03 |
> | fibn-tc | 9.01 | 0.00 | 0 | 9.01 | 0.01 |
> | flet | 17.98 | 0.00 | 0 | 17.98 | 0.03 |
> | inclist | 18.16 | 0.00 | 0 | 18.16 | 0.00 |
> | inclist-type-hints | 18.16 | 0.00 | 0 | 18.16 | 0.00 |
> | listlen-tc | 9.06 | 0.00 | 0 | 9.06 | 0.02 |
> | map-closure | 14.55 | 0.00 | 0 | 14.55 | 0.02 |
> | nbody | 5.86 | 16.11 | 795 | 21.97 | 0.05 |
> | pack-unpack | 1.03 | 0.86 | 43 | 1.89 | 0.01 |
> | pack-unpack-old | 3.15 | 1.74 | 87 | 4.89 | 0.01 |
> | pcase | 19.70 | 0.00 | 0 | 19.70 | 0.00 |
> | pidigits | 10.97 | 9.88 | 403 | 20.85 | 0.13 |
> | scroll | 1.28 | 0.08 | 4 | 1.37 | 0.00 |
> | smie | 3.07 | 0.94 | 46 | 4.00 | 0.01 |
> |--------------------+----------------+------------+---------+-------------+-----------------|
>
> `emacs-27` (i.e. commit d7f4cc0974645cc6a295740afe85c6e21d956119)
>
> | test | non-gc avg (s) | gc avg (s) | gcs avg | tot avg (s) | tot avg err (s) |
> |--------------------+----------------+------------+---------+-------------+-----------------|
> | bubble | 8.85 | 8.98 | 503 | 17.84 | 0.02 |
> | bubble-no-cons | 18.36 | 0.07 | 4 | 18.43 | 0.00 |
> | bytecomp | 2.87 | 2.45 | 134 | 5.32 | 0.00 |
> | dhrystone | 17.04 | 0.00 | 0 | 17.04 | 0.00 |
> | eieio | 5.25 | 3.67 | 204 | 8.92 | 0.01 |
> | fibn | 6.26 | 0.00 | 0 | 6.26 | 0.00 |
> | fibn-rec | 12.41 | 0.00 | 0 | 12.41 | 0.01 |
> | fibn-tc | 9.84 | 0.00 | 0 | 9.84 | 0.01 |
> | flet | 25.14 | 0.00 | 0 | 25.14 | 0.01 |
> | inclist | 31.55 | 0.02 | 1 | 31.57 | 0.05 |
> | inclist-type-hints | 42.77 | 0.02 | 1 | 42.79 | 0.01 |
> | listlen-tc | 9.92 | 0.00 | 0 | 9.92 | 0.03 |
> | map-closure | 15.90 | 0.00 | 0 | 15.90 | 0.03 |
> | nbody | 6.74 | 15.23 | 839 | 21.96 | 0.04 |
> | pack-unpack-old | 5.65 | 1.65 | 92 | 7.30 | 0.01 |
> | pcase | 23.63 | 0.00 | 0 | 23.63 | 0.00 |
> | pidigits | 10.39 | 10.10 | 457 | 20.49 | 0.19 |
> | smie | 3.39 | 0.98 | 54 | 4.37 | 0.00 |
> |--------------------+----------------+------------+---------+-------------+-----------------|
>
>
> -- Stefan
And here's `emacs-26` (i.e. commit 7497ee44b471f69ce59d131a6dece261e871534f)
| test | non-gc avg (s) | gc avg (s) | gcs avg | tot avg (s) | tot avg err (s) |
|--------------------+----------------+------------+---------+-------------+-----------------|
| bubble | 10.93 | 4.56 | 319 | 15.49 | 0.05 |
| bubble-no-cons | 21.20 | 0.03 | 2 | 21.23 | 0.00 |
| bytecomp | 3.04 | 1.24 | 84 | 4.28 | 0.02 |
| dhrystone | 18.65 | 0.00 | 0 | 18.65 | 0.02 |
| eieio | 5.35 | 1.89 | 129 | 7.24 | 0.03 |
| fibn | 7.26 | 0.00 | 0 | 7.26 | 0.00 |
| fibn-rec | 10.85 | 0.00 | 0 | 10.85 | 0.00 |
| fibn-tc | 9.38 | 0.00 | 0 | 9.38 | 0.00 |
| flet | 20.40 | 0.00 | 0 | 20.40 | 0.00 |
| inclist | 23.13 | 0.00 | 0 | 23.13 | 0.01 |
| inclist-type-hints | 30.49 | 0.00 | 0 | 30.49 | 0.00 |
| listlen-tc | 10.21 | 0.00 | 0 | 10.21 | 0.03 |
| map-closure | 14.00 | 0.00 | 0 | 14.00 | 0.00 |
| nbody | 7.11 | 7.56 | 531 | 14.67 | 0.04 |
| pack-unpack-old | 6.38 | 0.83 | 58 | 7.21 | 0.00 |
| pcase | 27.48 | 0.00 | 0 | 27.48 | 0.00 |
| pidigits | 0.98 | 0.17 | 12 | 1.15 | 0.00 |
| smie | 3.96 | 0.46 | 32 | 4.42 | 0.01 |
|--------------------+----------------+------------+---------+-------------+-----------------|
[ I'm having trouble building `emacs-25` ("cannot open
`m4/count-leading-zeros.m4'" and things like that), so I think I'll stop here. ]
Stefan
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: elisp-benchmarks
2022-02-10 16:51 ` elisp-benchmarks Stefan Monnier
@ 2022-02-10 21:53 ` Mattias Engdegård
2022-02-10 22:31 ` elisp-benchmarks Stefan Monnier
0 siblings, 1 reply; 9+ messages in thread
From: Mattias Engdegård @ 2022-02-10 21:53 UTC (permalink / raw)
To: Stefan Monnier; +Cc: emacs-devel
10 feb. 2022 kl. 17.51 skrev Stefan Monnier <monnier@iro.umontreal.ca>:
> And here's `emacs-26` (i.e. commit 7497ee44b471f69ce59d131a6dece261e871534f)
Thank you! Normalised to the first version where each benchmark result appears, we have (excluding GC):
| | 26 | 27 | 28 | pre sympos | sympos | master |
|--------------------+------+------+------+------------+--------+--------|
| bubble | 1.00 | 0.81 | 0.54 | 0.62 | 0.58 | 0.46 |
| bubble-no-cons | 1.00 | 0.87 | 0.75 | 0.77 | 0.78 | 0.63 |
| bytecomp | 1.00 | 0.94 | 1.00 | 1.07 | 1.13 | 1.06 |
| dhrystone | 1.00 | 0.91 | 0.82 | 0.83 | 0.86 | 0.66 |
| eieio | 1.00 | 0.98 | 0.95 | 1.04 | 1.03 | 0.87 |
| fibn | 1.00 | 0.86 | 0.75 | 0.75 | 0.75 | 0.52 |
| fibn-named-let | | | 1.00 | 1.03 | 1.01 | 0.70 |
| fibn-rec | 1.00 | 1.14 | 1.00 | 1.02 | 1.04 | 0.68 |
| fibn-tc | 1.00 | 1.05 | 0.96 | 0.99 | 1.00 | 0.59 |
| flet | 1.00 | 1.23 | 0.88 | 0.91 | 0.96 | 0.62 |
| inclist | 1.00 | 1.36 | 0.79 | 0.78 | 0.84 | 0.76 |
| inclist-type-hints | 1.00 | 1.40 | 0.60 | 0.59 | 0.64 | 0.58 |
| listlen-tc | 1.00 | 0.97 | 0.89 | 0.91 | 0.92 | 0.67 |
| map-closure | 1.00 | 1.14 | 1.04 | 1.04 | 1.06 | 0.72 |
| nbody | 1.00 | 0.95 | 0.82 | 0.84 | 0.81 | 0.71 |
| pack-unpack | | | 1.00 | 1.04 | 1.03 | 0.83 |
| pack-unpack-old | 1.00 | 0.89 | 0.49 | 0.52 | 0.52 | 0.47 |
| pcase | 1.00 | 0.86 | 0.72 | 0.74 | 0.64 | 0.63 |
| pidigits | | 1.00 | 1.06 | 1.15 | 1.07 | 0.99 |
| scroll | | | 1.00 | 1.05 | 1.02 | 1.09 |
| smie | 1.00 | 0.86 | 0.78 | 0.77 | 0.79 | 0.78 |
(The pidigits result was omitted from Emacs 26 because it didn't make much sense.)
The bytecomp benchmark is mobile -- it does more work (and generates better code) now than in earlier versions.
Not sure how comparable scroll is, but it's probably a difficult thing to measure.
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: elisp-benchmarks
2022-02-10 21:53 ` elisp-benchmarks Mattias Engdegård
@ 2022-02-10 22:31 ` Stefan Monnier
0 siblings, 0 replies; 9+ messages in thread
From: Stefan Monnier @ 2022-02-10 22:31 UTC (permalink / raw)
To: Mattias Engdegård; +Cc: emacs-devel
> Thank you! Normalised to the first version where each benchmark result
> appears, we have (excluding GC):
>
> | | 26 | 27 | 28 | pre sympos | sympos | master |
> |--------------------+------+------+------+------------+--------+--------|
> | bubble | 1.00 | 0.81 | 0.54 | 0.62 | 0.58 | 0.46 |
> | bubble-no-cons | 1.00 | 0.87 | 0.75 | 0.77 | 0.78 | 0.63 |
> | bytecomp | 1.00 | 0.94 | 1.00 | 1.07 | 1.13 | 1.06 |
> | dhrystone | 1.00 | 0.91 | 0.82 | 0.83 | 0.86 | 0.66 |
> | eieio | 1.00 | 0.98 | 0.95 | 1.04 | 1.03 | 0.87 |
> | fibn | 1.00 | 0.86 | 0.75 | 0.75 | 0.75 | 0.52 |
> | fibn-named-let | | | 1.00 | 1.03 | 1.01 | 0.70 |
> | fibn-rec | 1.00 | 1.14 | 1.00 | 1.02 | 1.04 | 0.68 |
> | fibn-tc | 1.00 | 1.05 | 0.96 | 0.99 | 1.00 | 0.59 |
> | flet | 1.00 | 1.23 | 0.88 | 0.91 | 0.96 | 0.62 |
> | inclist | 1.00 | 1.36 | 0.79 | 0.78 | 0.84 | 0.76 |
> | inclist-type-hints | 1.00 | 1.40 | 0.60 | 0.59 | 0.64 | 0.58 |
> | listlen-tc | 1.00 | 0.97 | 0.89 | 0.91 | 0.92 | 0.67 |
> | map-closure | 1.00 | 1.14 | 1.04 | 1.04 | 1.06 | 0.72 |
> | nbody | 1.00 | 0.95 | 0.82 | 0.84 | 0.81 | 0.71 |
> | pack-unpack | | | 1.00 | 1.04 | 1.03 | 0.83 |
> | pack-unpack-old | 1.00 | 0.89 | 0.49 | 0.52 | 0.52 | 0.47 |
> | pcase | 1.00 | 0.86 | 0.72 | 0.74 | 0.64 | 0.63 |
> | pidigits | | 1.00 | 1.06 | 1.15 | 1.07 | 0.99 |
> | scroll | | | 1.00 | 1.05 | 1.02 | 1.09 |
> | smie | 1.00 | 0.86 | 0.78 | 0.77 | 0.79 | 0.78 |
Thanks.
> (The pidigits result was omitted from Emacs 26 because it didn't make much sense.)
Good catch, yes, `pidigits` needs bignums to make sense, otherwise it
computes (quickly) a bogus result.
> The bytecomp benchmark is mobile -- it does more work (and generates better
> code) now than in earlier versions.
Yes, we don't want `bytecomp` to become unbearably so, it's OK if it
gets a bit slower if we get better code or better error messages out of it.
> Not sure how comparable scroll is, but it's probably a difficult thing to measure.
I don't know how to interpret the evolution of that benchmark, indeed.
It paradoxically seems to be made faster by sympos and slower by your
subsequent bytecode improvements. I'd tend to assume that the "faster
with sympos" is true-or-negligible, and but the "slower on master" is
a bit more perplexing. Could it be that there have been other changes that
happened on master recently that would cause that?
My intuition is that neither sympos nor your bytecode work should make
very much difference to the `scroll` benchmark since it should spend
most of its time in other parts of the C code (e.g. text-property
manipulations, regexp matching, redisplay, string extraction from the
buffer, ...).
Stefan
^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2022-02-10 22:31 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-02-09 22:19 elisp-benchmarks Stefan Monnier
2022-02-10 6:50 ` elisp-benchmarks Lars Ingebrigtsen
2022-02-10 7:52 ` elisp-benchmarks Eli Zaretskii
2022-02-10 12:12 ` elisp-benchmarks Mattias Engdegård
2022-02-10 14:13 ` elisp-benchmarks Stefan Monnier
2022-02-10 14:18 ` elisp-benchmarks Stefan Monnier
2022-02-10 16:51 ` elisp-benchmarks Stefan Monnier
2022-02-10 21:53 ` elisp-benchmarks Mattias Engdegård
2022-02-10 22:31 ` elisp-benchmarks Stefan Monnier
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/emacs.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).