Splitting some erc tests

Splitting some erc tests

[Lars Ingebrigtsen]

Some of the erc test files have a lot more tests than others, or have a
lot of tests that take a bit of time.  For instance, doing comp-tests
takes 40 seconds on my build machine, and tramp takes 30 seconds.  But
here's the thing: All the other tests, in total, clock in at under 20
seconds, because the machine has a lot of CPUs, and those other bits run
in parallel.

So I wondered -- has anybody given any thought to "internal" parallelism
in (some) test files?  We obviously don't want to rearrange the test
files themselves, but perhaps there's something simple we could do here.

Like...  have a SELECTOR that says "run the Xth 20% of the tests",
perhaps?  And then we can fire off 5 Emacsen to do 20% each.

The tests are supposed to be independent of each other, but are they
always in practice?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Splitting some erc tests

[Stefan Monnier]

> The tests are supposed to be independent of each other, but are they
> always in practice?

My bet is that they aren't.  But IIRC the tests aren't run in the order
in which they appear in the file, so I suspect that in *most* cases they
are indeed sufficiently independent for your parallelism use-case.
And those that aren't should be fixed accordingly.


        Stefan

Re: Splitting some erc tests

[Lars Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> My bet is that they aren't.  But IIRC the tests aren't run in the order
> in which they appear in the file, so I suspect that in *most* cases they
> are indeed sufficiently independent for your parallelism use-case.
> And those that aren't should be fixed accordingly.

I've done some slight testing, and it seems to work OK.  I've extended
ert-run-tests-batch-and-exit to take a "modulo" parameter to select the
nth test, and then you can pass that in from the Makefile.

However, I'm a bit lost as to how to express this stuff in the Makefile
itself.  If we're running without "-j<large-num>", then we don't want to
do this, but that information doesn't seem to be available without
something like this?

  https://blog.jgc.org/2015/03/gnu-make-insanity-finding-value-of-j.html

We'd also just want this for a select number of the tests, and the tests
are run as such:

%.log: %.elc
	$(AM_V_GEN)${MKDIR_P} $(dir $@)
	$(AM_V_at)HOME=$(TEST_HOME) $(emacs) \
	  -l ert ${ert_opts} -l $(testloadfile) \
	  $(TEST_RUN_ERT)

What would be the most convenient way to have a different rule for a set
of named tests?  I was first thinking of the moral equivalent of 

%.log: %.elc
       for i in `seq $jobs`; do
         $(emacs) ... (ert-run-tests-batch-and-exit ... $i $jobs) &
       done > $*.log

but that goes against the Make grain -- Make should control the
parallelism, of course.  So a number of rules like:

%.log-1: %.elc
       $(emacs) ... (ert-run-tests-batch-and-exit ... $* $jobs)

or something...  but...  this is way out of my Makefile comfort zone.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Splitting some erc tests

[Stefan Monnier]

> but that goes against the Make grain -- Make should control the
> parallelism, of course.  So a number of rules like:
>
> %.log-1: %.elc
>        $(emacs) ... (ert-run-tests-batch-and-exit ... $* $jobs)
>
> or something...  but...  this is way out of my Makefile comfort zone.

Something like that, yes.

My intuition tells me to first give those tests some unique name
pattern, so that we can use a different rule like

    %-par2.log: %-par2.elc %-2.1-part.log %-2.2-part.log
        cat $*-2.1-part.log $*-2.2-part.log >$@

Generalizing this to `%-parN.log` and writing the rules for
`%-N.M-part.log` is left as an exercise for the reader.

BTW I don't think we need to know the `-j` arg passed to `make` to
decide how to split the files: just split them into coarse enough grains
that splitting them doesn't slow down the -j1 case significantly.


        Stefan

Re: Splitting some erc tests

[Lars Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> My intuition tells me to first give those tests some unique name
> pattern, so that we can use a different rule like
>
>     %-par2.log: %-par2.elc %-2.1-part.log %-2.2-part.log
>         cat $*-2.1-part.log $*-2.2-part.log >$@

Hm...

> Generalizing this to `%-parN.log` and writing the rules for
> `%-N.M-part.log` is left as an exercise for the reader.

Darn!

> BTW I don't think we need to know the `-j` arg passed to `make` to
> decide how to split the files: just split them into coarse enough grains
> that splitting them doesn't slow down the -j1 case significantly.

I was hoping for a solution where -j1 didn't split anything -- it'd make
things easier to debug (and debugging ert failures is already hard
enough).

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no