Emacs benchmark workload to run and time instead of hunch performance

Emacs benchmark workload to run and time instead of hunch performance

[Emanuel Berg]

When I just did something with emacs -Q I noticed
startup was instantaneous compared to my regular. I
expected this and this is cool since I only start Emacs
once every time I start the God-damned computer (it is
even automatic), and even then, startup time is hardly
bad. However, when I did some things with the -Q'd
Emacs, I had a feeling typing, bringing up the M-x, and
so on, everything was just a tiny microscopic bit
faster... Perhaps imagination, but anyway I thought, is
there a bunch of code that resembles normal usage for
some interval, whatever that is (normal emaxing), so it
(the benchmark code) can be used on two configs, and
then the wall-clock times can be compared? If one
indeed is faster than the other, is this some stupid
configuration on my part or is it normal if you bring
in a lot of stuff? If I populate alists, hooks, you
name it, with lots of stuff (by the way, where do all
the defuns go? memory? some data structure?) - yeah,
that should slow it down, but will it really to the
degree you'd actually notice it?

-- 
underground experts united

Re: Emacs benchmark workload to run and time instead of hunch performance

[Emanuel Berg]

Grant Rettke <gcr@wisdomandwonder.com> writes:

> https://github.com/jschaf/esup

I suspect this was meant for gnu.emacs.help - OK, will
check that out.

-- 
underground experts united

Re: Emacs benchmark workload to run and time instead of hunch performance

[Emanuel Berg]

Emanuel Berg <embe8573@student.uu.se> didn't write:

> https://github.com/jschaf/esup
>
> I suspect this was meant for gnu.emacs.help - OK,
> will check that out.

What I can see from looking a the screenshot and
skimming the README.md this measures Emacs startup
time (esup) - but if you read my post, I said there is
no reason (for me at least) to do that, as I have the
OS automatically start Emacs after booting, so for me
it is part of the boot process. After that I never
close Emacs until I'm done for the time being, closing
Emacs as well as shutting down the computer. And that
is the recommended usage. It can of course be
interesting to measure startup time but for my
practical situation it wouldn't hurt (really) if Emacs
took 10 minutes to start! No, I asked for some
benchmark workload to run that would emulate or to a
degree actually perform what could be considered normal
emaxing - that way, the exact same workload could be
run with and without (emacs -Q) extensive configuration
and the require and load of modules, and then the times
could be compared. In C, there is a way of hammering
the DRAM and the memory bus (both of which might be
shared on a multiprocessor architecture) with memory
accesses that will crash through the core-local caches:
it is called pointer chasing and is basically creating
lots of pointers and then assigning values and
dereferencing the pointers... Pretty simple and
effective. Because Emacs is much more complicated, the
method perhaps must be, too (?); but actually any Elisp
could be executed and timed this way to give some
estimate.

-- 
underground experts united

Re: Emacs benchmark workload to run and time instead of hunch performance

[Emanuel Berg]

Emanuel Berg <embe8573@student.uu.se> did write:

> What I can see from looking a the screenshot and
> skimming the README.md this measures Emacs startup
> time (esup) - but if you read my post, I said there
> is no reason (for me at least) to do that, as I have
> the OS automatically start Emacs after booting, so
> for me it is part of the boot process. After that I
> never close Emacs until I'm done for the time being,
> closing Emacs as well as shutting down the
> computer. And that is the recommended usage. It can
> of course be interesting to measure startup time but
> for my practical situation it wouldn't hurt (really)
> if Emacs took 10 minutes to start! No, I asked for
> some benchmark workload to run that would emulate or
> to a degree actually perform what could be considered
> normal emaxing - that way, the exact same workload
> could be run with and without (emacs -Q) extensive
> configuration and the require and load of modules,
> and then the times could be compared. In C, there is
> a way of hammering the DRAM and the memory bus (both
> of which might be shared on a multiprocessor
> architecture) with memory accesses that will crash
> through the core-local caches: it is called pointer
> chasing and is basically creating lots of pointers
> and then assigning values and dereferencing the
> pointers... Pretty simple and effective. Because
> Emacs is much more complicated, the method perhaps
> must be, too (?); but actually any Elisp could be
> executed and timed this way to give some estimate.

It is as I feared. Blinded by success, the bright side
has grown complacent, allowing the dark side to regroup
unhindered. Only the super-senses of the
sleep-and-food-deprived Jedi Knights alerted us to this
grave danger!

Not-very-ambitious workload but still:

(defun bench ()
  (interactive)
  (define-abbrev global-abbrev-table "rsb" "rec.sport.boxing")
  (insert "rsb")
  (expand-abbrev)

  (insert "  3")
  (backward-char 3)
  (delete-char 2)

  (buffer-menu)

  (man "ls")

  (beginning-of-buffer) )

(progn
  (elp-instrument-function 'bench)
  (bench)
  (elp-results) )

Execution times for first invocation:

emacs -Q:

bench          1           0.038667778   0.038667778

emacs, with tons of configuration:

bench          1           0.05604648    0.05604648

Keep calling it I guess it ends up in a cache because
it gets much faster - but the emacs -Q is still faster.

So the next question is just what configs slows it down
- if there is indeed a difference there...

-- 
underground experts united

Re: Emacs benchmark workload to run and time instead of hunch performance

[Robert Thorpe]

Emanuel Berg <embe8573@student.uu.se> writes:

> Emanuel Berg <embe8573@student.uu.se> did write:
...
> Not-very-ambitious workload but still:
>
> (defun bench ()
>   (interactive)
>   (define-abbrev global-abbrev-table "rsb" "rec.sport.boxing")
>   (insert "rsb")
>   (expand-abbrev)
>
>   (insert "  3")
>   (backward-char 3)
>   (delete-char 2)
>
>   (buffer-menu)
>
>   (man "ls")
>
>   (beginning-of-buffer) )
>
> (progn
>   (elp-instrument-function 'bench)
>   (bench)
>   (elp-results) )
>
> Execution times for first invocation:
>
> emacs -Q:
>
> bench          1           0.038667778   0.038667778
>
> emacs, with tons of configuration:
>
> bench          1           0.05604648    0.05604648

I get ~0.028 for emacs -Q and ~0.03 using my init file.

> Keep calling it I guess it ends up in a cache because
> it gets much faster - but the emacs -Q is still faster.

That's not the reason.

Do (elp-instrument-function 'man).  It'll tell you that for the
single-run case much of the time is spent in (man "ls").  The Emacs
"man" command works by calling the system man command which typesets the
man page from it's troff source then pipes it to Emacs.  The Emacs man
command first checks if the man page is already open in a buffer, if so
it just switches to that buffer.  So, on the first interation of your
benchmark you have the system man program performing typesetting, then
on each later iteration it does nothing.  To make the benchmark fair you
have to kill the man buffer before each run.

Of course cache effects still make the benchmark quicker for repeated
runs.  That's partly because of the processor's cache, but also because
Linux's file-system cache will store the man page's source file.  (It
goes below 0.01 for me).

> So the next question is just what configs slows it down
> - if there is indeed a difference there...

I don't know.  Try dividing you init file into two halves.  See if the
first half or the second half causes the slowdown.  When you've found
which half it is divide that into halves too, and so on until you've
found the culprit.

BR,
Robert Thorpe

Re: Emacs benchmark workload to run and time instead of hunch performance

[Emanuel Berg]

Robert Thorpe <rt@robertthorpeconsulting.com> writes:

> I get ~0.028 for emacs -Q and ~0.03 using my init file.

From the Emacs man page:

   -q, --no-init-file
           Do not load an init file.

   --no-site-file
           Do not load the site-wide startup file.

   [...]

   -Q, --quick
           Similar  to  "-q  --no-site-file   --no-splash".
           Also, avoid processing X resources.

So yes, it seems this method is correct: emacs and
'emacs -Q' should be identical if in the emacs case,
there aren't any init, startup, or X resources files.

>> Keep calling it I guess it ends up in a cache
>> because it gets much faster - but the emacs -Q is
>> still faster.
>
> That's not the reason.
>
> Do (elp-instrument-function 'man).  It'll tell you
> that for the single-run case much of the time is
> spent in (man "ls").  The Emacs "man" command works
> by calling the system man command which typesets the
> man page from it's troff source then pipes it to
> Emacs.  The Emacs man command first checks if the man
> page is already open in a buffer, if so it just
> switches to that buffer.  So, on the first interation
> of your benchmark you have the system man program
> performing typesetting, then on each later iteration
> it does nothing.  To make the benchmark fair you have
> to kill the man buffer before each run.

OK. Didn't anyone ever attempt to write a serious
workload that would take into account all things like
that? It is a very basic idea so I would be very
surprised if no one did it before, larger in scope and
correcting such details as the one you mentioned.

> Of course cache effects still make the benchmark
> quicker for repeated runs.  That's partly because of
> the processor's cache, but also because Linux's
> file-system cache will store the man page's source
> file.  (It goes below 0.01 for me).

OK.

> I don't know.  Try dividing you init file into two
> halves.  See if the first half or the second half
> causes the slowdown.  When you've found which half it
> is divide that into halves too, and so on until
> you've found the culprit.

Yes. But I'll hold my breath for a while to find out if
no one did this before. I don't want to do organized
testing with that workload. Also, my
modules-loaded-and-configured Emacs is absolutely not
slow - it is just that 'emacs -Q' seems (and is, as
seen) just a tiny bit snappier.

-- 
underground experts united

Re: Emacs benchmark workload to run and time instead of hunch performance

[Stefan Monnier]

> testing with that workload. Also, my
> modules-loaded-and-configured Emacs is absolutely not
> slow - it is just that 'emacs -Q' seems (and is, as
> seen) just a tiny bit snappier.

It's to be expected that every loaded module might slow down
some operations.  Loading a major mode should usually not make
a difference, and neither should minor modes that just add
key-bindings.  But many minor modes use various hooks to do work
before/after some/every commands, or before/during redisplay.
So these tend to slow down the interactive feel while they'll make
little/no difference in typical benchmarks that are done as a single
command and don't involve redisplay.


        Stefan

Re: Emacs benchmark workload to run and time instead of hunch performance

[Emanuel Berg]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> It's to be expected that every loaded module might
> slow down some operations.  Loading a major mode
> should usually not make a difference, and neither
> should minor modes that just add key-bindings.  But
> many minor modes use various hooks to do work
> before/after some/every commands, or before/during
> redisplay.  So these tend to slow down the
> interactive feel while they'll make little/no
> difference in typical benchmarks that are done as a
> single command and don't involve redisplay.

Any suggestions how to make it more "real"? Does it
help to call everything interactively? Or perhaps feed
a keyboard macro to do all those things?

-- 
underground experts united