bug#56424: hydra.nixos coverage job times out

bug#56424: hydra.nixos coverage job times out

[Glenn Morris]

Package: emacs
Version: 29.0.50

Ref: https://hydra.nixos.org/jobset/gnu/emacs-trunk

The hydra.nixos "coverage" job, which is the one that runs "make check",
had been timing out for a month. If this has not been flagged before
now, it seems like a sign that this automated build isn't useful any
more and should be stopped.

The way to do that is probably to open an issue at
https://github.com/NixOS/hydra/issues

There is also https://lists.gnu.org/mailman/listinfo/hydra-users ,
but I'm not sure it is used/monitored any more.

The start of the timeout cannot be pin-pointed, because the build was
broken for several days.

It last worked consistently around
https://hydra.nixos.org/eval/1764830

Then the build was broken for several days from:
https://hydra.nixos.org/eval/1764836

The build started working again in:
https://hydra.nixos.org/eval/1765592
but coverage was already timing out.

It has worked once or twice since then, but mostly timed out.

See https://hydra.nixos.org/jobset/gnu/emacs-trunk/evals?page=30
(at time of writing)

No-one has login access to this system. If you want to investigate this,
your options are:

1) Install Nix locally and try the "Reproduce locally" button from the
"Actions" menu on a job web page.

2) Make the check process more verbose, so you can see where it hangs.
The enevironment variable EMACS_HYDRA_CI is set in these builds.

3) Modify the hydra build recipe. To do this, get access to
https://savannah.nongnu.org/p/hydra-recipes in the usual way, and modify
the file emacs/release.nix.

bug#56424: hydra.nixos coverage job times out

[Lars Ingebrigtsen]

Glenn Morris <rgm@gnu.org> writes:

> Ref: https://hydra.nixos.org/jobset/gnu/emacs-trunk
>
> The hydra.nixos "coverage" job, which is the one that runs "make check",
> had been timing out for a month. If this has not been flagged before
> now, it seems like a sign that this automated build isn't useful any
> more and should be stopped.

Doesn't seem to be timing out now, but it's failing.

https://hydra.nixos.org/build/182893714/nixlog/1

lisp/dnd-tests.log:
   FAILED  dnd-tests-begin-drag-files  "Method `mock' should specify both encoding and decoding command or an scp program"
   FAILED  dnd-tests-begin-file-drag  "Method `mock' should specify both encoding and decoding command or an scp program"

I've added Po to the CCs.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#56424: hydra.nixos coverage job times out

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Doesn't seem to be timing out now, but it's failing.
>
> https://hydra.nixos.org/build/182893714/nixlog/1
>
> lisp/dnd-tests.log:
>    FAILED  dnd-tests-begin-drag-files  "Method `mock' should specify both encoding and decoding command or an scp program"
>    FAILED  dnd-tests-begin-file-drag  "Method `mock' should specify both encoding and decoding command or an scp program"
>
> I've added Po to the CCs.

Does the build system have scp installed?

Also, most of the remote file test code involved here wasn't written by
me.  I think it was Michael Albinus.

bug#56424: hydra.nixos coverage job times out

[Lars Ingebrigtsen]

Po Lu <luangruo@yahoo.com> writes:

>> Doesn't seem to be timing out now, but it's failing.
>>
>> https://hydra.nixos.org/build/182893714/nixlog/1
>>
>> lisp/dnd-tests.log:
>>    FAILED dnd-tests-begin-drag-files "Method `mock' should specify
>> both encoding and decoding command or an scp program"
>>    FAILED dnd-tests-begin-file-drag "Method `mock' should specify
>> both encoding and decoding command or an scp program"
>>
>> I've added Po to the CCs.
>
> Does the build system have scp installed?
>
> Also, most of the remote file test code involved here wasn't written by
> me.  I think it was Michael Albinus.

Right; adding Michael to the CCs.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#56424: hydra.nixos coverage job times out

[Michael Albinus]

Lars Ingebrigtsen <larsi@gnus.org> writes:

Hi,

>>> Doesn't seem to be timing out now, but it's failing.
>>>
>>> https://hydra.nixos.org/build/182893714/nixlog/1
>>>
>>> lisp/dnd-tests.log:
>>>    FAILED dnd-tests-begin-drag-files "Method `mock' should specify
>>> both encoding and decoding command or an scp program"
>>>    FAILED dnd-tests-begin-file-drag "Method `mock' should specify
>>> both encoding and decoding command or an scp program"
>>>
>>> I've added Po to the CCs.
>>
>> Does the build system have scp installed?
>>
>> Also, most of the remote file test code involved here wasn't written by
>> me.  I think it was Michael Albinus.
>
> Right; adding Michael to the CCs.

Thanks for the heads-up. I've instrumented dnd-tests-begin-drag-files
for hydra; let's see what happens.

Best regards, Michael.

bug#56424: hydra.nixos coverage job times out

[Michael Albinus]

Michael Albinus <michael.albinus@gmx.de> writes:

Hi,

>>>> Doesn't seem to be timing out now, but it's failing.
>>>>
>>>> https://hydra.nixos.org/build/182893714/nixlog/1
>>>>
>>>> lisp/dnd-tests.log:
>>>>    FAILED dnd-tests-begin-drag-files "Method `mock' should specify
>>>> both encoding and decoding command or an scp program"
>>>>    FAILED dnd-tests-begin-file-drag "Method `mock' should specify
>>>> both encoding and decoding command or an scp program"
>>>>
>>>> I've added Po to the CCs.
>>>
>>> Does the build system have scp installed?
>>>
>>> Also, most of the remote file test code involved here wasn't written by
>>> me.  I think it was Michael Albinus.
>>
>> Right; adding Michael to the CCs.
>
> Thanks for the heads-up. I've instrumented dnd-tests-begin-drag-files
> for hydra; let's see what happens.

The dnd-tests on hydra shall be fixed now.

Since this bug report is about process-tests, we shall keep it open.

Best regards, Michael.

bug#56424: hydra.nixos coverage job times out

[Glenn Morris]

You happened to look at one job that for some reason did not time out.
Browsing the past month of jobs shows timeout after timeout.

With increased logging, and a serial build, we can see that it times out
in process-tests:
https://hydra.nixos.org/build/183466954

Again; if no-one is looking at this CI system any more (which is fine),
then it would be better to stop wasting the electricity.

bug#56424: hydra.nixos coverage job times out

[Michael Albinus]

Glenn Morris <rgm@gnu.org> writes:

Hi Glenn,

> With increased logging, and a serial build, we can see that it times out
> in process-tests:
> https://hydra.nixos.org/build/183466954

The incrimed test is process-tests/fd-setsize-no-crash/make-process.
From EMBA, I know that there are sometimes problems with this test
package, that's why we have suppressed
process-tests/multiple-threads-waiting on EMBA, for example. Maybe
something similar shall be done for HYDRA?

I've CC'ed Philipp Stephani, the author of this test package, for
further analysis.

Best regards, Michael.

bug#56424: hydra.nixos coverage job times out

[Philipp Stephani]

Am So., 10. Juli 2022 um 14:40 Uhr schrieb Michael Albinus
<michael.albinus@gmx.de>:
>
> Glenn Morris <rgm@gnu.org> writes:
>
> Hi Glenn,
>
> > With increased logging, and a serial build, we can see that it times out
> > in process-tests:
> > https://hydra.nixos.org/build/183466954
>
> The incrimed test is process-tests/fd-setsize-no-crash/make-process.
> From EMBA, I know that there are sometimes problems with this test
> package, that's why we have suppressed
> process-tests/multiple-threads-waiting on EMBA, for example. Maybe
> something similar shall be done for HYDRA?
>
> I've CC'ed Philipp Stephani, the author of this test package, for
> further analysis.


My 2 cents: These timeouts/failures indicate real bugs in Emacs. They
are incredibly hard to debug due to the enormous complexity in Emacs's
asynchronous process handling code, but just skipping them won't make
the bugs go away. Somebody should at least try to dig into these
issues. But probably that requires something like the CI system
creating a snapshot of the failed tests as a Docker container that one
could SSH into, because the failures tend to be impossible to
reproduce locally.

bug#56424: hydra.nixos coverage job times out

[Lars Ingebrigtsen]

Glenn Morris <rgm@gnu.org> writes:

> Again; if no-one is looking at this CI system any more (which is fine),
> then it would be better to stop wasting the electricity.

I look at both Nixos and EMBA regularly, but if I see that it's all red,
I assume that the CI system has a problem (which is correct 93% of the
time, because those systems seem both odd and catastrophically
underpowered) and wait for somebody to reboot them.

I'd love for somebody to finally set up a real CI system with reasonable
power (so that a build takes a few minutes instead of a few hours) and a
normal OS.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#56424: hydra.nixos coverage job times out

[Lars Ingebrigtsen]

Everything on Nixos seems to be failing now.  For instance:

https://hydra.nixos.org/build/193879284

I'm not sure what the actual failure there is at all, but there's a
bunch of warnings like:

Unescaped left brace in regex is deprecated here (and will be fatal in Perl 5.32), passed through in regex; marked by <-- HERE in m/^\s+@([[:alnum:]][[:alnum:]\-]*)({ <-- HERE })?\s*/ at /nix/store/xbxabhr44naykgf768533sijfcamkyr9-texinfo-6.5/share/texinfo/Texinfo/Parser.pm line 5481.

The actual log ends with:

  GEN      ../../lisp/international/cp51932.el
  GEN      ../../lisp/international/eucjp-ms.el
  GEN      ../../etc/charsets/GB180304.map
  GEN      ../../etc/charsets/JISX2131.map
  GEN      charsets.stamp
make[3]: Leaving directory '/build/bc1m3dggd3vw7606h3asba5nsbx3d3qz-source/admin/charsets'
make[2]: Leaving directory '/build/bc1m3dggd3vw7606h3asba5nsbx3d3qz-source/src'
make[1]: *** [Makefile:537: src] Error 2
make[1]: Leaving directory '/build/bc1m3dggd3vw7606h3asba5nsbx3d3qz-source'
make[1]: Entering directory '/build/bc1m3dggd3vw7606h3asba5nsbx3d3qz-source'
***
*** "make all" failed with exit status 2.

Which isn't very helpful.

Hm...  Oh, the actual error is earlier:


mv: cannot stat 'lisp.mk.tmp': No such file or directory
  GEN      globals.h
  GEN      buildobj.h
  GEN      globals.h
  GEN      buildobj.h
cmp: globals.tmp: No such file or directory
mv: cannot stat 'globals.tmp': No such file or directory
cmp: globals.tmp: No such file or directory
make[1]: *** [Makefile:671: gl-stamp] Error 2
make[1]: Leaving directory '/build/bc1m3dggd3vw7606h3asba5nsbx3d3qz-source/src'
make: *** [Makefile:537: src] Error 2
make: *** Waiting for unfinished jobs....

Either something is very wrong on that build machine, or there's
something very wrong in our build.