Performance tracking

Performance tracking

[Neil Jerram]

Here's an idea for performance tracking.  Please let me know what you
think.

- I think the main people interested in performance tracking are the
  core Guile developers, so this proposal is primarily for them (aka
  us).

- Because we all have different machines, we need separate tracking.

- So, each developer who wants to track performance over time creates a
  performance-data/<unique-name> file in their tree, and commits it to
  the repository.

- <unique-name> needs to be associated with the particular machine that
  a particular developer uses, over time, to run performance tests.  For
  example, I might create `performance-data/neil-arudy'.

- The file content is created and updated by running `benchmark-guile >
  performance-data/<unique-name>'.

- You don't have to do this every time you commit any change, but we
  need the convention that whenever a performance-data file _is_
  updated, the data in it is generated by the tree of that commit.

- We can write tools to show changes in performance data over time (or
  over commits).  This should be easy, as the output of benchmark-guile
  is Scheme-readable.

Thanks,
        Neil

Re: Performance tracking

[Ludovic Courtès]

Hi,

This sounds like one possibility, but I think it might be inconvenient
to use that data to view how performance evolved over time.

I was thinking we could have a dedicated machine running benchmarks,
say, everyday, and publishing plots somewhere.

The machine could be one from the GCC Compile Farm; we’d need to arrange
so that nothing runs at the same time as the benchmarks.  We’d also need
to find a box that can host the plots.

Thanks,
Ludo’.

Re: Performance tracking

[Ken Raeburn]

On Sep 17, 2009, at 17:53, Ludovic Courtès wrote:
> I was thinking we could have a dedicated machine running benchmarks,
> say, everyday, and publishing plots somewhere.

I'd suggest multiple machines, if possible.  Different operating  
systems (for example, I've seen that mutex performance differs quite a  
bit between Linux and Mac OS X), maybe different processor families.   
If some of the benchmarks test multithreaded performance, perhaps  
multiprocessor vs uniprocessor systems too.

Testing on a single machine is better than not testing at all, but I  
don't think we want to fine-tune it for one system at the expense of  
performance on lots of others.

Ken

Re: Performance tracking

[Neil Jerram]

ludo@gnu.org (Ludovic Courtès) writes:

> Hi,
>
> This sounds like one possibility, but I think it might be inconvenient
> to use that data to view how performance evolved over time.

Well let's try it and see.  If it really is difficult or not useful, we
can just delete the data file(s) again.

> I was thinking we could have a dedicated machine running benchmarks,
> say, everyday, and publishing plots somewhere.

Yes, that would indeed be good too.

>  We’d also need
> to find a box that can host the plots.

www.gnu.org/software/guile/...?  Or savannah?  Perhaps whatever we do
for Guile could be generalized for projects in general?

     Neil

Performance tracking

[Ludovic Courtès]

Hello!

I was looking at the “history chart” at
<http://hydra.nixos.org/build/2517280#tabs-history>, which shows graphs
of the build time and installed Guile size vs. commits.  Timings must be
taken with a grain of salt, because of variability on the build machines.

Still, a couple of worthwhile observations:

  • commit 1af6d2a (“Minimize size of embedded syntax objects in
    psyntax-pp.scm”) reduced the installed size from ~14.6 MiB to
    ~13.3 MiB;

  • CSE led to a build time increase from 28m at
    <http://hydra.nixos.org/build/2413477> to 43m
    <http://hydra.nixos.org/build/2478518>.

Thanks,
Ludo’.

Re: Performance tracking

[Noah Lavine]

It appears to me (anecdotally) that most of the build time is spent
compiling Scheme code, rather than C code.

One idea I had been toying with is whether Guile could compile faster
if it had another copy of Guile already around, so it could skip the
portion of compile-time where the interpreter is running the compiler.
This is how most compilers do it - you want another C compiler around
to compile GCC, etc. I was afraid that this would result in a
too-complicated build system, but maybe not. Does anyone have
experience with implementing something like this?

Noah

On Fri, May 4, 2012 at 7:30 PM, Ludovic Courtès <ludo@gnu.org> wrote:
> Hello!
>
> I was looking at the “history chart” at
> <http://hydra.nixos.org/build/2517280#tabs-history>, which shows graphs
> of the build time and installed Guile size vs. commits.  Timings must be
> taken with a grain of salt, because of variability on the build machines.
>
> Still, a couple of worthwhile observations:
>
>  • commit 1af6d2a (“Minimize size of embedded syntax objects in
>    psyntax-pp.scm”) reduced the installed size from ~14.6 MiB to
>    ~13.3 MiB;
>
>  • CSE led to a build time increase from 28m at
>    <http://hydra.nixos.org/build/2413477> to 43m
>    <http://hydra.nixos.org/build/2478518>.
>
> Thanks,
> Ludo’.
>
>

Re: Performance tracking

[David Kastrup]

Noah Lavine <noah.b.lavine@gmail.com> writes:

> It appears to me (anecdotally) that most of the build time is spent
> compiling Scheme code, rather than C code.
>
> One idea I had been toying with is whether Guile could compile faster
> if it had another copy of Guile already around, so it could skip the
> portion of compile-time where the interpreter is running the compiler.

Bootstrap hell.  If the "copy of Guile already around" happens to
produce different code, you have lots of fun ahead.

-- 
David Kastrup

Re: Performance tracking

[Noah Lavine]

Oh, I was unclear. I meant that the existing copy of Guile would run
the compiler from the new copy of Guile.

In the worst case you'd have to bootstrap, but that's what we do now,
every time.

Noah

On Sat, May 5, 2012 at 12:42 PM, David Kastrup <dak@gnu.org> wrote:
> Noah Lavine <noah.b.lavine@gmail.com> writes:
>
>> It appears to me (anecdotally) that most of the build time is spent
>> compiling Scheme code, rather than C code.
>>
>> One idea I had been toying with is whether Guile could compile faster
>> if it had another copy of Guile already around, so it could skip the
>> portion of compile-time where the interpreter is running the compiler.
>
> Bootstrap hell.  If the "copy of Guile already around" happens to
> produce different code, you have lots of fun ahead.
>
> --
> David Kastrup
>
>

Re: Performance tracking

[David Kastrup]

Noah Lavine <noah.b.lavine@gmail.com> writes:

>> On Sat, May 5, 2012 at 12:42 PM, David Kastrup <dak@gnu.org> wrote:
>>> Noah Lavine <noah.b.lavine@gmail.com> writes:
>>>
>>>> It appears to me (anecdotally) that most of the build time is spent
>>>> compiling Scheme code, rather than C code.
>>>>
>>>> One idea I had been toying with is whether Guile could compile faster
>>>> if it had another copy of Guile already around, so it could skip the
>>>> portion of compile-time where the interpreter is running the compiler.
>>>
>>> Bootstrap hell.  If the "copy of Guile already around" happens to
>>> produce different code, you have lots of fun ahead.
>
>
> Oh, I was unclear. I meant that the existing copy of Guile would run
> the compiler from the new copy of Guile.

That assumes that it runs it producing a correct result.

> In the worst case you'd have to bootstrap, but that's what we do now,
> every time.

If the "copy of Guile already around" is malicious, bootstrapping from
the produced compilation can't get rid of the malice.  See Ken
Thompson's malicious compiler experiment.

-- 
David Kastrup

Re: Performance tracking

[Ludovic Courtès]

Hi Noah,

Noah Lavine <noah.b.lavine@gmail.com> skribis:

> One idea I had been toying with is whether Guile could compile faster
> if it had another copy of Guile already around, so it could skip the
> portion of compile-time where the interpreter is running the compiler.
> This is how most compilers do it - you want another C compiler around
> to compile GCC, etc. I was afraid that this would result in a
> too-complicated build system, but maybe not. Does anyone have
> experience with implementing something like this?

$GUILE_FOR_BUILD is used when cross-compiling Guile itself–i.e., a
native Guile is needed to cross-build Guile.

And according to <http://hydra.nixos.org/build/2517410>, Guile is
cross-built in ~4m, whereas the native Guile itself is cross-built in ~40m.

Now, using $GUILE_FOR_BUILD to build the native Guile itself would be
tricky, because there’s no distinction between the run-time load path
and the compiler’s search path (see ‘GUILE_CHECK_GUILE_FOR_BUILD’ in
acinclude.m4.)

Thanks,
Ludo’.

Re: Performance tracking

[dsmich]

---- Noah Lavine <noah.b.lavine@gmail.com> wrote: 
> It appears to me (anecdotally) that most of the build time is spent
> compiling Scheme code, rather than C code.
> 
> One idea I had been toying with is whether Guile could compile faster
> if it had another copy of Guile already around, so it could skip the
> portion of compile-time where the interpreter is running the compiler.

This is what the GUILE_FOR_BUILD env var is for.

-Dale

Re: Performance tracking

[Ludovic Courtès]

Hi!

<dsmich@roadrunner.com> skribis:

> ---- Noah Lavine <noah.b.lavine@gmail.com> wrote: 
>> It appears to me (anecdotally) that most of the build time is spent
>> compiling Scheme code, rather than C code.
>> 
>> One idea I had been toying with is whether Guile could compile faster
>> if it had another copy of Guile already around, so it could skip the
>> portion of compile-time where the interpreter is running the compiler.
>
> This is what the GUILE_FOR_BUILD env var is for.

It’s actually only used when cross-compiling.

Ludo’.

Re: Performance tracking

[dsmich]

---- "Ludovic Courtès" <ludo@gnu.org> wrote: 
> Hi!
> 
> <dsmich@roadrunner.com> skribis:
> 
> > ---- Noah Lavine <noah.b.lavine@gmail.com> wrote: 
> >> It appears to me (anecdotally) that most of the build time is spent
> >> compiling Scheme code, rather than C code.
> >> 
> >> One idea I had been toying with is whether Guile could compile faster
> >> if it had another copy of Guile already around, so it could skip the
> >> portion of compile-time where the interpreter is running the compiler.
> >
> > This is what the GUILE_FOR_BUILD env var is for.
> 
> It’s actually only used when cross-compiling.
> 
> Ludo’.

Ahh.  That explains why I've never seen much speed improvement when using it. ;^)

Thanks.

-Dale