bug#75359: [scratch/elisp-benchmarks] test dependencies

bug#75359: [scratch/elisp-benchmarks] test dependencies

[Pip Cet via Bug reports for GNU Emacs, the Swiss army knife of text editors]

There are dependencies between emacs-lisp-benchmarks tests, meaning we
can't always modify one test without affecting others.  The ones I'm
aware of are:

- elb-bytecomp.el uses elb-smie.el as compilation input (in this
  context, elb-smie.el is a "resource", not source code).  If we
  modify elb-smie.el, elb-bytecomp.el results will change and become
  uncomparable.

- elb-scroll.el uses elb-smie.el as a prerequisite.  If we modify
  elb-smie.el, elb-scroll.el results will change and become
  uncomparable.  Furthermore, elb-scroll.el relies on elb-smie.el
  being compiled in the appropriate fashion by some external
  mechanism.  (I think the best solution here is to merge elb-smie.el
  and elb-scroll.el into a single file declaring several tests, using
  the established ERT test framework)

- bubble.el and bubble-no-cons.el both "provide" the same feature.
  I'm not sure whether this causes any acute problems, but it
  demonstrates that the inconsistent use of (provide) in the
  benchmarks is problematic.  If we ever load tests using Frequire, we
  will load the wrong files.  If we don't, provide shouldn't be
  necessary.

- xmenu.c is used as a resource; this is fine.  But we must make sure
  we never copy the "current" xmenu.c to the benchmarks directory
  without clearly indicating that the benchmark results will become
  uncomparable.

bug#75359: [scratch/elisp-benchmarks] test dependencies

[Andrea Corallo]

Pip Cet via "Bug reports for GNU Emacs, the Swiss army knife of text
editors" <bug-gnu-emacs@gnu.org> writes:

> There are dependencies between emacs-lisp-benchmarks tests, meaning we
> can't always modify one test without affecting others.  The ones I'm
> aware of are:
>
> - elb-bytecomp.el uses elb-smie.el as compilation input (in this
>   context, elb-smie.el is a "resource", not source code).  If we
>   modify elb-smie.el, elb-bytecomp.el results will change and become
>   uncomparable.

Shouldn't we just copy of elb-smie.el to resources?

> - elb-scroll.el uses elb-smie.el as a prerequisite.  If we modify
>   elb-smie.el, elb-scroll.el results will change and become
>   uncomparable.  Furthermore, elb-scroll.el relies on elb-smie.el
>   being compiled in the appropriate fashion by some external
>   mechanism.  (I think the best solution here is to merge elb-smie.el
>   and elb-scroll.el into a single file declaring several tests, using
>   the established ERT test framework)

I'm not sure these are real/serious bugs, different versions of
elisp-benchmarks don't guarantee to give comparable results (like every
benchmark).  Anyway adding Stefan who added this code.

> - bubble.el and bubble-no-cons.el both "provide" the same feature.
>   I'm not sure whether this causes any acute problems, but it
>   demonstrates that the inconsistent use of (provide) in the
>   benchmarks is problematic.  If we ever load tests using Frequire, we
>   will load the wrong files.  If we don't, provide shouldn't be
>   necessary.

AFAIU the provide is not used by elisp-benchmarks, anyway the typo is
now fixed.

> - xmenu.c is used as a resource; this is fine.  But we must make sure
>   we never copy the "current" xmenu.c to the benchmarks directory
>   without clearly indicating that the benchmark results will become
>   uncomparable.

Again, changes to the benchmark pertubating results should be tagged as
a new version, so will be fine to update xmenu.c if necessary as long as
we update elisp-benchmarks version.

bug#75359: [scratch/elisp-benchmarks] test dependencies

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> There are dependencies between emacs-lisp-benchmarks tests, meaning we
>> can't always modify one test without affecting others.  The ones I'm
>> aware of are:
>>
>> - elb-bytecomp.el uses elb-smie.el as compilation input (in this
>>   context, elb-smie.el is a "resource", not source code).  If we
>>   modify elb-smie.el, elb-bytecomp.el results will change and become
>>   uncomparable.
>
> Shouldn't we just copy of elb-smie.el to resources?

If so, we could just as well use some other file.  I used `elb-smie.el`
because it was a large enough file and it was already part of
`elisp-benchmarks`.

>> - elb-scroll.el uses elb-smie.el as a prerequisite.  If we modify
>>   elb-smie.el, elb-scroll.el results will change and become
>>   uncomparable.  Furthermore, elb-scroll.el relies on elb-smie.el
>>   being compiled in the appropriate fashion by some external
>>   mechanism.  (I think the best solution here is to merge elb-smie.el
>>   and elb-scroll.el into a single file declaring several tests, using
>>   the established ERT test framework)
>
> I'm not sure these are real/serious bugs, different versions of
> elisp-benchmarks don't guarantee to give comparable results (like every
> benchmark).

Agreed.
This said, merging `elb-smie.el` and `elb-scroll.el` would be fine.
Can't remember why I didn't do that from the start.

[ Not sure what ERT has to do with it, BTW.  ]

>> - bubble.el and bubble-no-cons.el both "provide" the same feature.
>>   I'm not sure whether this causes any acute problems, but it
>>   demonstrates that the inconsistent use of (provide) in the
>>   benchmarks is problematic.  If we ever load tests using Frequire, we
>>   will load the wrong files.  If we don't, provide shouldn't be
>>   necessary.
>
> AFAIU the provide is not used by elisp-benchmarks, anyway the typo is
> now fixed.

FWIW, these two files date back to the time where the benchmarks were
defined by their file (i.e. one file per benchmark), whereas nowadays
the benchmarks are defined by the `elb-*-entry` functions, so we should
probably merge those two files into one.

>> - xmenu.c is used as a resource; this is fine.  But we must make sure
>>   we never copy the "current" xmenu.c to the benchmarks directory
>>   without clearly indicating that the benchmark results will become
>>   uncomparable.
> Again, changes to the benchmark pertubating results should be tagged as
> a new version, so will be fine to update xmenu.c if necessary as long as
> we update elisp-benchmarks version.

Also I can't think of any reason why we'd want to "update" that file.
At least not until its use of the C language is considered too
antiquated/deprecated.


        Stefan

bug#75359: [scratch/elisp-benchmarks] test dependencies

[Pip Cet via Bug reports for GNU Emacs, the Swiss army knife of text editors]

"Stefan Monnier" <monnier@iro.umontreal.ca> writes:

> This said, merging `elb-smie.el` and `elb-scroll.el` would be fine.
> Can't remember why I didn't do that from the start.

Did that on my branch :-)

> [ Not sure what ERT has to do with it, BTW.  ]

See the "make benchmark" discussion, if you really want to :-)

Context: I have extended ERT so it has some minimal benchmarking
functionality, after investigating (twice) whether elisp-benchmarks.el
can't be made to do the job.  My conclusion was that that would take too
much time, we should use ERT and ditch elisp-benchmarks.el, modifying
the benchmarks instead.

Andrea asked for bug reports, I provided some, but my opinion is that
spending further time on elisp-benchmarks.el isn't productive, so I
won't.

Pip

bug#75359: [scratch/elisp-benchmarks] test dependencies

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> Context: I have extended ERT so it has some minimal benchmarking
> functionality, after investigating (twice) whether elisp-benchmarks.el
> can't be made to do the job.  My conclusion was that that would take too
> much time, we should use ERT and ditch elisp-benchmarks.el, modifying
> the benchmarks instead.

I don't have any opinion on that.  The `elisp-benchmark.el` code itself
is very small/trivial, so rewriting it should be quite easy.

[ Then again, I can't think of any reason why any of it would take "much
  time", whether modifying `elisp-benchmarks.el`, or extending ERT, or
  writing it all from scratch.  So I'm probably missing something.  ]


        Stefan