* bug#56424: hydra.nixos coverage job times out
@ 2022-07-06 17:47 Glenn Morris
2022-07-07 9:04 ` Lars Ingebrigtsen
2022-10-11 1:03 ` Lars Ingebrigtsen
0 siblings, 2 replies; 11+ messages in thread
From: Glenn Morris @ 2022-07-06 17:47 UTC (permalink / raw)
To: 56424
Package: emacs
Version: 29.0.50
Ref: https://hydra.nixos.org/jobset/gnu/emacs-trunk
The hydra.nixos "coverage" job, which is the one that runs "make check",
had been timing out for a month. If this has not been flagged before
now, it seems like a sign that this automated build isn't useful any
more and should be stopped.
The way to do that is probably to open an issue at
https://github.com/NixOS/hydra/issues
There is also https://lists.gnu.org/mailman/listinfo/hydra-users ,
but I'm not sure it is used/monitored any more.
The start of the timeout cannot be pin-pointed, because the build was
broken for several days.
It last worked consistently around
https://hydra.nixos.org/eval/1764830
Then the build was broken for several days from:
https://hydra.nixos.org/eval/1764836
The build started working again in:
https://hydra.nixos.org/eval/1765592
but coverage was already timing out.
It has worked once or twice since then, but mostly timed out.
See https://hydra.nixos.org/jobset/gnu/emacs-trunk/evals?page=30
(at time of writing)
No-one has login access to this system. If you want to investigate this,
your options are:
1) Install Nix locally and try the "Reproduce locally" button from the
"Actions" menu on a job web page.
2) Make the check process more verbose, so you can see where it hangs.
The enevironment variable EMACS_HYDRA_CI is set in these builds.
3) Modify the hydra build recipe. To do this, get access to
https://savannah.nongnu.org/p/hydra-recipes in the usual way, and modify
the file emacs/release.nix.
^ permalink raw reply [flat|nested] 11+ messages in thread
* bug#56424: hydra.nixos coverage job times out
2022-07-06 17:47 bug#56424: hydra.nixos coverage job times out Glenn Morris
@ 2022-07-07 9:04 ` Lars Ingebrigtsen
2022-07-07 10:07 ` Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors
2022-07-10 2:02 ` Glenn Morris
2022-10-11 1:03 ` Lars Ingebrigtsen
1 sibling, 2 replies; 11+ messages in thread
From: Lars Ingebrigtsen @ 2022-07-07 9:04 UTC (permalink / raw)
To: Glenn Morris; +Cc: Po Lu, 56424
Glenn Morris <rgm@gnu.org> writes:
> Ref: https://hydra.nixos.org/jobset/gnu/emacs-trunk
>
> The hydra.nixos "coverage" job, which is the one that runs "make check",
> had been timing out for a month. If this has not been flagged before
> now, it seems like a sign that this automated build isn't useful any
> more and should be stopped.
Doesn't seem to be timing out now, but it's failing.
https://hydra.nixos.org/build/182893714/nixlog/1
lisp/dnd-tests.log:
FAILED dnd-tests-begin-drag-files "Method `mock' should specify both encoding and decoding command or an scp program"
FAILED dnd-tests-begin-file-drag "Method `mock' should specify both encoding and decoding command or an scp program"
I've added Po to the CCs.
--
(domestic pets only, the antidote for overdose, milk.)
bloggy blog: http://lars.ingebrigtsen.no
^ permalink raw reply [flat|nested] 11+ messages in thread
* bug#56424: hydra.nixos coverage job times out
2022-07-07 9:04 ` Lars Ingebrigtsen
@ 2022-07-07 10:07 ` Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors
2022-07-07 17:34 ` Lars Ingebrigtsen
2022-07-10 2:02 ` Glenn Morris
1 sibling, 1 reply; 11+ messages in thread
From: Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors @ 2022-07-07 10:07 UTC (permalink / raw)
To: Lars Ingebrigtsen; +Cc: Glenn Morris, 56424
Lars Ingebrigtsen <larsi@gnus.org> writes:
> Doesn't seem to be timing out now, but it's failing.
>
> https://hydra.nixos.org/build/182893714/nixlog/1
>
> lisp/dnd-tests.log:
> FAILED dnd-tests-begin-drag-files "Method `mock' should specify both encoding and decoding command or an scp program"
> FAILED dnd-tests-begin-file-drag "Method `mock' should specify both encoding and decoding command or an scp program"
>
> I've added Po to the CCs.
Does the build system have scp installed?
Also, most of the remote file test code involved here wasn't written by
me. I think it was Michael Albinus.
^ permalink raw reply [flat|nested] 11+ messages in thread
* bug#56424: hydra.nixos coverage job times out
2022-07-07 10:07 ` Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors
@ 2022-07-07 17:34 ` Lars Ingebrigtsen
2022-07-10 10:44 ` Michael Albinus
0 siblings, 1 reply; 11+ messages in thread
From: Lars Ingebrigtsen @ 2022-07-07 17:34 UTC (permalink / raw)
To: Po Lu; +Cc: Glenn Morris, Michael Albinus, 56424
Po Lu <luangruo@yahoo.com> writes:
>> Doesn't seem to be timing out now, but it's failing.
>>
>> https://hydra.nixos.org/build/182893714/nixlog/1
>>
>> lisp/dnd-tests.log:
>> FAILED dnd-tests-begin-drag-files "Method `mock' should specify
>> both encoding and decoding command or an scp program"
>> FAILED dnd-tests-begin-file-drag "Method `mock' should specify
>> both encoding and decoding command or an scp program"
>>
>> I've added Po to the CCs.
>
> Does the build system have scp installed?
>
> Also, most of the remote file test code involved here wasn't written by
> me. I think it was Michael Albinus.
Right; adding Michael to the CCs.
--
(domestic pets only, the antidote for overdose, milk.)
bloggy blog: http://lars.ingebrigtsen.no
^ permalink raw reply [flat|nested] 11+ messages in thread
* bug#56424: hydra.nixos coverage job times out
2022-07-07 17:34 ` Lars Ingebrigtsen
@ 2022-07-10 10:44 ` Michael Albinus
2022-07-10 20:09 ` Michael Albinus
0 siblings, 1 reply; 11+ messages in thread
From: Michael Albinus @ 2022-07-10 10:44 UTC (permalink / raw)
To: Lars Ingebrigtsen; +Cc: Po Lu, Glenn Morris, 56424
Lars Ingebrigtsen <larsi@gnus.org> writes:
Hi,
>>> Doesn't seem to be timing out now, but it's failing.
>>>
>>> https://hydra.nixos.org/build/182893714/nixlog/1
>>>
>>> lisp/dnd-tests.log:
>>> FAILED dnd-tests-begin-drag-files "Method `mock' should specify
>>> both encoding and decoding command or an scp program"
>>> FAILED dnd-tests-begin-file-drag "Method `mock' should specify
>>> both encoding and decoding command or an scp program"
>>>
>>> I've added Po to the CCs.
>>
>> Does the build system have scp installed?
>>
>> Also, most of the remote file test code involved here wasn't written by
>> me. I think it was Michael Albinus.
>
> Right; adding Michael to the CCs.
Thanks for the heads-up. I've instrumented dnd-tests-begin-drag-files
for hydra; let's see what happens.
Best regards, Michael.
^ permalink raw reply [flat|nested] 11+ messages in thread
* bug#56424: hydra.nixos coverage job times out
2022-07-10 10:44 ` Michael Albinus
@ 2022-07-10 20:09 ` Michael Albinus
0 siblings, 0 replies; 11+ messages in thread
From: Michael Albinus @ 2022-07-10 20:09 UTC (permalink / raw)
To: Lars Ingebrigtsen; +Cc: Po Lu, Glenn Morris, 56424
Michael Albinus <michael.albinus@gmx.de> writes:
Hi,
>>>> Doesn't seem to be timing out now, but it's failing.
>>>>
>>>> https://hydra.nixos.org/build/182893714/nixlog/1
>>>>
>>>> lisp/dnd-tests.log:
>>>> FAILED dnd-tests-begin-drag-files "Method `mock' should specify
>>>> both encoding and decoding command or an scp program"
>>>> FAILED dnd-tests-begin-file-drag "Method `mock' should specify
>>>> both encoding and decoding command or an scp program"
>>>>
>>>> I've added Po to the CCs.
>>>
>>> Does the build system have scp installed?
>>>
>>> Also, most of the remote file test code involved here wasn't written by
>>> me. I think it was Michael Albinus.
>>
>> Right; adding Michael to the CCs.
>
> Thanks for the heads-up. I've instrumented dnd-tests-begin-drag-files
> for hydra; let's see what happens.
The dnd-tests on hydra shall be fixed now.
Since this bug report is about process-tests, we shall keep it open.
Best regards, Michael.
^ permalink raw reply [flat|nested] 11+ messages in thread
* bug#56424: hydra.nixos coverage job times out
2022-07-07 9:04 ` Lars Ingebrigtsen
2022-07-07 10:07 ` Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors
@ 2022-07-10 2:02 ` Glenn Morris
2022-07-10 12:39 ` Michael Albinus
2022-07-10 13:00 ` Lars Ingebrigtsen
1 sibling, 2 replies; 11+ messages in thread
From: Glenn Morris @ 2022-07-10 2:02 UTC (permalink / raw)
To: Lars Ingebrigtsen; +Cc: Po Lu, 56424
You happened to look at one job that for some reason did not time out.
Browsing the past month of jobs shows timeout after timeout.
With increased logging, and a serial build, we can see that it times out
in process-tests:
https://hydra.nixos.org/build/183466954
Again; if no-one is looking at this CI system any more (which is fine),
then it would be better to stop wasting the electricity.
^ permalink raw reply [flat|nested] 11+ messages in thread
* bug#56424: hydra.nixos coverage job times out
2022-07-10 2:02 ` Glenn Morris
@ 2022-07-10 12:39 ` Michael Albinus
2022-07-19 12:52 ` Philipp Stephani
2022-07-10 13:00 ` Lars Ingebrigtsen
1 sibling, 1 reply; 11+ messages in thread
From: Michael Albinus @ 2022-07-10 12:39 UTC (permalink / raw)
To: Glenn Morris; +Cc: Po Lu, Philipp Stephani, Lars Ingebrigtsen, 56424
Glenn Morris <rgm@gnu.org> writes:
Hi Glenn,
> With increased logging, and a serial build, we can see that it times out
> in process-tests:
> https://hydra.nixos.org/build/183466954
The incrimed test is process-tests/fd-setsize-no-crash/make-process.
From EMBA, I know that there are sometimes problems with this test
package, that's why we have suppressed
process-tests/multiple-threads-waiting on EMBA, for example. Maybe
something similar shall be done for HYDRA?
I've CC'ed Philipp Stephani, the author of this test package, for
further analysis.
Best regards, Michael.
^ permalink raw reply [flat|nested] 11+ messages in thread
* bug#56424: hydra.nixos coverage job times out
2022-07-10 12:39 ` Michael Albinus
@ 2022-07-19 12:52 ` Philipp Stephani
0 siblings, 0 replies; 11+ messages in thread
From: Philipp Stephani @ 2022-07-19 12:52 UTC (permalink / raw)
To: Michael Albinus
Cc: Po Lu, Glenn Morris, Lars Ingebrigtsen, Philipp Stephani, 56424
Am So., 10. Juli 2022 um 14:40 Uhr schrieb Michael Albinus
<michael.albinus@gmx.de>:
>
> Glenn Morris <rgm@gnu.org> writes:
>
> Hi Glenn,
>
> > With increased logging, and a serial build, we can see that it times out
> > in process-tests:
> > https://hydra.nixos.org/build/183466954
>
> The incrimed test is process-tests/fd-setsize-no-crash/make-process.
> From EMBA, I know that there are sometimes problems with this test
> package, that's why we have suppressed
> process-tests/multiple-threads-waiting on EMBA, for example. Maybe
> something similar shall be done for HYDRA?
>
> I've CC'ed Philipp Stephani, the author of this test package, for
> further analysis.
My 2 cents: These timeouts/failures indicate real bugs in Emacs. They
are incredibly hard to debug due to the enormous complexity in Emacs's
asynchronous process handling code, but just skipping them won't make
the bugs go away. Somebody should at least try to dig into these
issues. But probably that requires something like the CI system
creating a snapshot of the failed tests as a Docker container that one
could SSH into, because the failures tend to be impossible to
reproduce locally.
^ permalink raw reply [flat|nested] 11+ messages in thread
* bug#56424: hydra.nixos coverage job times out
2022-07-10 2:02 ` Glenn Morris
2022-07-10 12:39 ` Michael Albinus
@ 2022-07-10 13:00 ` Lars Ingebrigtsen
1 sibling, 0 replies; 11+ messages in thread
From: Lars Ingebrigtsen @ 2022-07-10 13:00 UTC (permalink / raw)
To: Glenn Morris; +Cc: Po Lu, 56424
Glenn Morris <rgm@gnu.org> writes:
> Again; if no-one is looking at this CI system any more (which is fine),
> then it would be better to stop wasting the electricity.
I look at both Nixos and EMBA regularly, but if I see that it's all red,
I assume that the CI system has a problem (which is correct 93% of the
time, because those systems seem both odd and catastrophically
underpowered) and wait for somebody to reboot them.
I'd love for somebody to finally set up a real CI system with reasonable
power (so that a build takes a few minutes instead of a few hours) and a
normal OS.
--
(domestic pets only, the antidote for overdose, milk.)
bloggy blog: http://lars.ingebrigtsen.no
^ permalink raw reply [flat|nested] 11+ messages in thread
* bug#56424: hydra.nixos coverage job times out
2022-07-06 17:47 bug#56424: hydra.nixos coverage job times out Glenn Morris
2022-07-07 9:04 ` Lars Ingebrigtsen
@ 2022-10-11 1:03 ` Lars Ingebrigtsen
1 sibling, 0 replies; 11+ messages in thread
From: Lars Ingebrigtsen @ 2022-10-11 1:03 UTC (permalink / raw)
To: Glenn Morris; +Cc: 56424
Everything on Nixos seems to be failing now. For instance:
https://hydra.nixos.org/build/193879284
I'm not sure what the actual failure there is at all, but there's a
bunch of warnings like:
Unescaped left brace in regex is deprecated here (and will be fatal in Perl 5.32), passed through in regex; marked by <-- HERE in m/^\s+@([[:alnum:]][[:alnum:]\-]*)({ <-- HERE })?\s*/ at /nix/store/xbxabhr44naykgf768533sijfcamkyr9-texinfo-6.5/share/texinfo/Texinfo/Parser.pm line 5481.
The actual log ends with:
GEN ../../lisp/international/cp51932.el
GEN ../../lisp/international/eucjp-ms.el
GEN ../../etc/charsets/GB180304.map
GEN ../../etc/charsets/JISX2131.map
GEN charsets.stamp
make[3]: Leaving directory '/build/bc1m3dggd3vw7606h3asba5nsbx3d3qz-source/admin/charsets'
make[2]: Leaving directory '/build/bc1m3dggd3vw7606h3asba5nsbx3d3qz-source/src'
make[1]: *** [Makefile:537: src] Error 2
make[1]: Leaving directory '/build/bc1m3dggd3vw7606h3asba5nsbx3d3qz-source'
make[1]: Entering directory '/build/bc1m3dggd3vw7606h3asba5nsbx3d3qz-source'
***
*** "make all" failed with exit status 2.
Which isn't very helpful.
Hm... Oh, the actual error is earlier:
mv: cannot stat 'lisp.mk.tmp': No such file or directory
GEN globals.h
GEN buildobj.h
GEN globals.h
GEN buildobj.h
cmp: globals.tmp: No such file or directory
mv: cannot stat 'globals.tmp': No such file or directory
cmp: globals.tmp: No such file or directory
make[1]: *** [Makefile:671: gl-stamp] Error 2
make[1]: Leaving directory '/build/bc1m3dggd3vw7606h3asba5nsbx3d3qz-source/src'
make: *** [Makefile:537: src] Error 2
make: *** Waiting for unfinished jobs....
Either something is very wrong on that build machine, or there's
something very wrong in our build.
^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2022-10-11 1:03 UTC | newest]
Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-07-06 17:47 bug#56424: hydra.nixos coverage job times out Glenn Morris
2022-07-07 9:04 ` Lars Ingebrigtsen
2022-07-07 10:07 ` Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors
2022-07-07 17:34 ` Lars Ingebrigtsen
2022-07-10 10:44 ` Michael Albinus
2022-07-10 20:09 ` Michael Albinus
2022-07-10 2:02 ` Glenn Morris
2022-07-10 12:39 ` Michael Albinus
2022-07-19 12:52 ` Philipp Stephani
2022-07-10 13:00 ` Lars Ingebrigtsen
2022-10-11 1:03 ` Lars Ingebrigtsen
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/emacs.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).