Re: [ELPA] New package: elisp-benchmarks

[Eric Abrahamsen <eric@ericabrahamsen.net>]

Andrea Corallo <akrl@sdf.org> writes:

> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> Andrea Corallo <akrl@sdf.org> writes:
>>
>>> Hi, I attach the patch git formatted.  I've also fixed two nits.
>>
>> Thanks for this package, and I'm looking forward to trying it out! I
>> have one question at this point, which is that it seems somewhat
>> limiting to require benchmarks to be placed into this package's own
>> benchmarks/ directory. It seems like it would be nice to let them live
>> anywhere, even if that means selecting them manually, rather than
>> forcing a single location, as the package currently seems to do (unless
>> I'm missing something?). For example, it might be nice to ship other
>> packages with their own benchmark libraries, which would then be
>> selectable somehow. What do you think?
>>
>> Thanks!
>> Eric
>
> Hi Eric,
> that's a very good point thanks for commenting.
>
> Actually with the current setup you could depose a benchmark in the
> benchmarks folder that calls say some gnus function and define a test in
> that way.  Everything would work except the 'recompile'.

Is the expectation that we'd actually be committing benchmarks to the
benchmarks package? Or just dropping files into the installation
directory? Both approaches seem a bit awkward -- for the former, you'd
have to update the benchmark package whenever any other package updated
its benchmarks. For the latter, when we update the benchmarks package,
we'll lose any files we stuck in there manually.

> I also like the idea of doing the other way around (defining benchmarks
> and have them shipped with the package itself).  This would be
> potentially more elegant but I guess we should then have the
> infrastructure to do that in Emacs and not just in ELPA.  Is this
> correct?
>
> Maybe you have in mind a different mechanism for this?

I suppose the fanciest approach would be to do what ert does: provide a
`elb-defbenchmark' macro that registers the benchmark in some central
location, and allows the user to select which of the registered
benchmarks to run. That might take more work than you'd had in mind,
though. A simpler approach might be just to make the "working directory"
argument of the interactive commands selectable with a prefix arg. Right
now the prefix arg controls the selector -- maybe that could be a
two-part selection? First directory, then string? Or something like
that.

At the very least, `elb-bench-directory' could be let-bound in a custom
function. But making it a defconst really makes it feel like we
shouldn't be doing that.

Hope that's useful,
Eric