all messages for Emacs-related lists mirrored at yhetil.org
 help / color / mirror / code / Atom feed
* Splitting some erc tests
@ 2022-05-24 19:11 Lars Ingebrigtsen
  2022-05-24 19:59 ` Stefan Monnier
  0 siblings, 1 reply; 5+ messages in thread
From: Lars Ingebrigtsen @ 2022-05-24 19:11 UTC (permalink / raw)
  To: emacs-devel@gnu.org

Some of the erc test files have a lot more tests than others, or have a
lot of tests that take a bit of time.  For instance, doing comp-tests
takes 40 seconds on my build machine, and tramp takes 30 seconds.  But
here's the thing: All the other tests, in total, clock in at under 20
seconds, because the machine has a lot of CPUs, and those other bits run
in parallel.

So I wondered -- has anybody given any thought to "internal" parallelism
in (some) test files?  We obviously don't want to rearrange the test
files themselves, but perhaps there's something simple we could do here.

Like...  have a SELECTOR that says "run the Xth 20% of the tests",
perhaps?  And then we can fire off 5 Emacsen to do 20% each.

The tests are supposed to be independent of each other, but are they
always in practice?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no




^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: Splitting some erc tests
  2022-05-24 19:11 Splitting some erc tests Lars Ingebrigtsen
@ 2022-05-24 19:59 ` Stefan Monnier
  2022-05-25  1:13   ` Lars Ingebrigtsen
  0 siblings, 1 reply; 5+ messages in thread
From: Stefan Monnier @ 2022-05-24 19:59 UTC (permalink / raw)
  To: Lars Ingebrigtsen; +Cc: emacs-devel@gnu.org

> The tests are supposed to be independent of each other, but are they
> always in practice?

My bet is that they aren't.  But IIRC the tests aren't run in the order
in which they appear in the file, so I suspect that in *most* cases they
are indeed sufficiently independent for your parallelism use-case.
And those that aren't should be fixed accordingly.


        Stefan




^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: Splitting some erc tests
  2022-05-24 19:59 ` Stefan Monnier
@ 2022-05-25  1:13   ` Lars Ingebrigtsen
  2022-05-25  2:17     ` Stefan Monnier
  0 siblings, 1 reply; 5+ messages in thread
From: Lars Ingebrigtsen @ 2022-05-25  1:13 UTC (permalink / raw)
  To: Stefan Monnier; +Cc: emacs-devel@gnu.org

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> My bet is that they aren't.  But IIRC the tests aren't run in the order
> in which they appear in the file, so I suspect that in *most* cases they
> are indeed sufficiently independent for your parallelism use-case.
> And those that aren't should be fixed accordingly.

I've done some slight testing, and it seems to work OK.  I've extended
ert-run-tests-batch-and-exit to take a "modulo" parameter to select the
nth test, and then you can pass that in from the Makefile.

However, I'm a bit lost as to how to express this stuff in the Makefile
itself.  If we're running without "-j<large-num>", then we don't want to
do this, but that information doesn't seem to be available without
something like this?

  https://blog.jgc.org/2015/03/gnu-make-insanity-finding-value-of-j.html

We'd also just want this for a select number of the tests, and the tests
are run as such:

%.log: %.elc
	$(AM_V_GEN)${MKDIR_P} $(dir $@)
	$(AM_V_at)HOME=$(TEST_HOME) $(emacs) \
	  -l ert ${ert_opts} -l $(testloadfile) \
	  $(TEST_RUN_ERT)

What would be the most convenient way to have a different rule for a set
of named tests?  I was first thinking of the moral equivalent of 

%.log: %.elc
       for i in `seq $jobs`; do
         $(emacs) ... (ert-run-tests-batch-and-exit ... $i $jobs) &
       done > $*.log

but that goes against the Make grain -- Make should control the
parallelism, of course.  So a number of rules like:

%.log-1: %.elc
       $(emacs) ... (ert-run-tests-batch-and-exit ... $* $jobs)

or something...  but...  this is way out of my Makefile comfort zone.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no



^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: Splitting some erc tests
  2022-05-25  1:13   ` Lars Ingebrigtsen
@ 2022-05-25  2:17     ` Stefan Monnier
  2022-05-25 12:02       ` Lars Ingebrigtsen
  0 siblings, 1 reply; 5+ messages in thread
From: Stefan Monnier @ 2022-05-25  2:17 UTC (permalink / raw)
  To: Lars Ingebrigtsen; +Cc: emacs-devel@gnu.org

> but that goes against the Make grain -- Make should control the
> parallelism, of course.  So a number of rules like:
>
> %.log-1: %.elc
>        $(emacs) ... (ert-run-tests-batch-and-exit ... $* $jobs)
>
> or something...  but...  this is way out of my Makefile comfort zone.

Something like that, yes.

My intuition tells me to first give those tests some unique name
pattern, so that we can use a different rule like

    %-par2.log: %-par2.elc %-2.1-part.log %-2.2-part.log
        cat $*-2.1-part.log $*-2.2-part.log >$@

Generalizing this to `%-parN.log` and writing the rules for
`%-N.M-part.log` is left as an exercise for the reader.

BTW I don't think we need to know the `-j` arg passed to `make` to
decide how to split the files: just split them into coarse enough grains
that splitting them doesn't slow down the -j1 case significantly.


        Stefan




^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: Splitting some erc tests
  2022-05-25  2:17     ` Stefan Monnier
@ 2022-05-25 12:02       ` Lars Ingebrigtsen
  0 siblings, 0 replies; 5+ messages in thread
From: Lars Ingebrigtsen @ 2022-05-25 12:02 UTC (permalink / raw)
  To: Stefan Monnier; +Cc: emacs-devel@gnu.org

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> My intuition tells me to first give those tests some unique name
> pattern, so that we can use a different rule like
>
>     %-par2.log: %-par2.elc %-2.1-part.log %-2.2-part.log
>         cat $*-2.1-part.log $*-2.2-part.log >$@

Hm...

> Generalizing this to `%-parN.log` and writing the rules for
> `%-N.M-part.log` is left as an exercise for the reader.

Darn!

> BTW I don't think we need to know the `-j` arg passed to `make` to
> decide how to split the files: just split them into coarse enough grains
> that splitting them doesn't slow down the -j1 case significantly.

I was hoping for a solution where -j1 didn't split anything -- it'd make
things easier to debug (and debugging ert failures is already hard
enough).

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no



^ permalink raw reply	[flat|nested] 5+ messages in thread

end of thread, other threads:[~2022-05-25 12:02 UTC | newest]

Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2022-05-24 19:11 Splitting some erc tests Lars Ingebrigtsen
2022-05-24 19:59 ` Stefan Monnier
2022-05-25  1:13   ` Lars Ingebrigtsen
2022-05-25  2:17     ` Stefan Monnier
2022-05-25 12:02       ` Lars Ingebrigtsen

Code repositories for project(s) associated with this external index

	https://git.savannah.gnu.org/cgit/emacs.git
	https://git.savannah.gnu.org/cgit/emacs/org-mode.git

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.