unofficial mirror of bug-gnu-emacs@gnu.org 
 help / color / mirror / code / Atom feed
* bug#56424: hydra.nixos coverage job times out
@ 2022-07-06 17:47 Glenn Morris
  2022-07-07  9:04 ` Lars Ingebrigtsen
  2022-10-11  1:03 ` Lars Ingebrigtsen
  0 siblings, 2 replies; 11+ messages in thread
From: Glenn Morris @ 2022-07-06 17:47 UTC (permalink / raw)
  To: 56424

Package: emacs
Version: 29.0.50

Ref: https://hydra.nixos.org/jobset/gnu/emacs-trunk

The hydra.nixos "coverage" job, which is the one that runs "make check",
had been timing out for a month. If this has not been flagged before
now, it seems like a sign that this automated build isn't useful any
more and should be stopped.

The way to do that is probably to open an issue at
https://github.com/NixOS/hydra/issues

There is also https://lists.gnu.org/mailman/listinfo/hydra-users ,
but I'm not sure it is used/monitored any more.

The start of the timeout cannot be pin-pointed, because the build was
broken for several days.

It last worked consistently around
https://hydra.nixos.org/eval/1764830

Then the build was broken for several days from:
https://hydra.nixos.org/eval/1764836

The build started working again in:
https://hydra.nixos.org/eval/1765592
but coverage was already timing out.

It has worked once or twice since then, but mostly timed out.

See https://hydra.nixos.org/jobset/gnu/emacs-trunk/evals?page=30
(at time of writing)

No-one has login access to this system. If you want to investigate this,
your options are:

1) Install Nix locally and try the "Reproduce locally" button from the
"Actions" menu on a job web page.

2) Make the check process more verbose, so you can see where it hangs.
The enevironment variable EMACS_HYDRA_CI is set in these builds.

3) Modify the hydra build recipe. To do this, get access to
https://savannah.nongnu.org/p/hydra-recipes in the usual way, and modify
the file emacs/release.nix.





^ permalink raw reply	[flat|nested] 11+ messages in thread

* bug#56424: hydra.nixos coverage job times out
  2022-07-06 17:47 bug#56424: hydra.nixos coverage job times out Glenn Morris
@ 2022-07-07  9:04 ` Lars Ingebrigtsen
  2022-07-07 10:07   ` Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors
  2022-07-10  2:02   ` Glenn Morris
  2022-10-11  1:03 ` Lars Ingebrigtsen
  1 sibling, 2 replies; 11+ messages in thread
From: Lars Ingebrigtsen @ 2022-07-07  9:04 UTC (permalink / raw)
  To: Glenn Morris; +Cc: Po Lu, 56424

Glenn Morris <rgm@gnu.org> writes:

> Ref: https://hydra.nixos.org/jobset/gnu/emacs-trunk
>
> The hydra.nixos "coverage" job, which is the one that runs "make check",
> had been timing out for a month. If this has not been flagged before
> now, it seems like a sign that this automated build isn't useful any
> more and should be stopped.

Doesn't seem to be timing out now, but it's failing.

https://hydra.nixos.org/build/182893714/nixlog/1

lisp/dnd-tests.log:
   FAILED  dnd-tests-begin-drag-files  "Method `mock' should specify both encoding and decoding command or an scp program"
   FAILED  dnd-tests-begin-file-drag  "Method `mock' should specify both encoding and decoding command or an scp program"

I've added Po to the CCs.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no





^ permalink raw reply	[flat|nested] 11+ messages in thread

* bug#56424: hydra.nixos coverage job times out
  2022-07-07  9:04 ` Lars Ingebrigtsen
@ 2022-07-07 10:07   ` Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors
  2022-07-07 17:34     ` Lars Ingebrigtsen
  2022-07-10  2:02   ` Glenn Morris
  1 sibling, 1 reply; 11+ messages in thread
From: Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors @ 2022-07-07 10:07 UTC (permalink / raw)
  To: Lars Ingebrigtsen; +Cc: Glenn Morris, 56424

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Doesn't seem to be timing out now, but it's failing.
>
> https://hydra.nixos.org/build/182893714/nixlog/1
>
> lisp/dnd-tests.log:
>    FAILED  dnd-tests-begin-drag-files  "Method `mock' should specify both encoding and decoding command or an scp program"
>    FAILED  dnd-tests-begin-file-drag  "Method `mock' should specify both encoding and decoding command or an scp program"
>
> I've added Po to the CCs.

Does the build system have scp installed?

Also, most of the remote file test code involved here wasn't written by
me.  I think it was Michael Albinus.





^ permalink raw reply	[flat|nested] 11+ messages in thread

* bug#56424: hydra.nixos coverage job times out
  2022-07-07 10:07   ` Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors
@ 2022-07-07 17:34     ` Lars Ingebrigtsen
  2022-07-10 10:44       ` Michael Albinus
  0 siblings, 1 reply; 11+ messages in thread
From: Lars Ingebrigtsen @ 2022-07-07 17:34 UTC (permalink / raw)
  To: Po Lu; +Cc: Glenn Morris, Michael Albinus, 56424

Po Lu <luangruo@yahoo.com> writes:

>> Doesn't seem to be timing out now, but it's failing.
>>
>> https://hydra.nixos.org/build/182893714/nixlog/1
>>
>> lisp/dnd-tests.log:
>>    FAILED dnd-tests-begin-drag-files "Method `mock' should specify
>> both encoding and decoding command or an scp program"
>>    FAILED dnd-tests-begin-file-drag "Method `mock' should specify
>> both encoding and decoding command or an scp program"
>>
>> I've added Po to the CCs.
>
> Does the build system have scp installed?
>
> Also, most of the remote file test code involved here wasn't written by
> me.  I think it was Michael Albinus.

Right; adding Michael to the CCs.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no





^ permalink raw reply	[flat|nested] 11+ messages in thread

* bug#56424: hydra.nixos coverage job times out
  2022-07-07  9:04 ` Lars Ingebrigtsen
  2022-07-07 10:07   ` Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors
@ 2022-07-10  2:02   ` Glenn Morris
  2022-07-10 12:39     ` Michael Albinus
  2022-07-10 13:00     ` Lars Ingebrigtsen
  1 sibling, 2 replies; 11+ messages in thread
From: Glenn Morris @ 2022-07-10  2:02 UTC (permalink / raw)
  To: Lars Ingebrigtsen; +Cc: Po Lu, 56424


You happened to look at one job that for some reason did not time out.
Browsing the past month of jobs shows timeout after timeout.

With increased logging, and a serial build, we can see that it times out
in process-tests:
https://hydra.nixos.org/build/183466954

Again; if no-one is looking at this CI system any more (which is fine),
then it would be better to stop wasting the electricity.





^ permalink raw reply	[flat|nested] 11+ messages in thread

* bug#56424: hydra.nixos coverage job times out
  2022-07-07 17:34     ` Lars Ingebrigtsen
@ 2022-07-10 10:44       ` Michael Albinus
  2022-07-10 20:09         ` Michael Albinus
  0 siblings, 1 reply; 11+ messages in thread
From: Michael Albinus @ 2022-07-10 10:44 UTC (permalink / raw)
  To: Lars Ingebrigtsen; +Cc: Po Lu, Glenn Morris, 56424

Lars Ingebrigtsen <larsi@gnus.org> writes:

Hi,

>>> Doesn't seem to be timing out now, but it's failing.
>>>
>>> https://hydra.nixos.org/build/182893714/nixlog/1
>>>
>>> lisp/dnd-tests.log:
>>>    FAILED dnd-tests-begin-drag-files "Method `mock' should specify
>>> both encoding and decoding command or an scp program"
>>>    FAILED dnd-tests-begin-file-drag "Method `mock' should specify
>>> both encoding and decoding command or an scp program"
>>>
>>> I've added Po to the CCs.
>>
>> Does the build system have scp installed?
>>
>> Also, most of the remote file test code involved here wasn't written by
>> me.  I think it was Michael Albinus.
>
> Right; adding Michael to the CCs.

Thanks for the heads-up. I've instrumented dnd-tests-begin-drag-files
for hydra; let's see what happens.

Best regards, Michael.





^ permalink raw reply	[flat|nested] 11+ messages in thread

* bug#56424: hydra.nixos coverage job times out
  2022-07-10  2:02   ` Glenn Morris
@ 2022-07-10 12:39     ` Michael Albinus
  2022-07-19 12:52       ` Philipp Stephani
  2022-07-10 13:00     ` Lars Ingebrigtsen
  1 sibling, 1 reply; 11+ messages in thread
From: Michael Albinus @ 2022-07-10 12:39 UTC (permalink / raw)
  To: Glenn Morris; +Cc: Po Lu, Philipp Stephani, Lars Ingebrigtsen, 56424

Glenn Morris <rgm@gnu.org> writes:

Hi Glenn,

> With increased logging, and a serial build, we can see that it times out
> in process-tests:
> https://hydra.nixos.org/build/183466954

The incrimed test is process-tests/fd-setsize-no-crash/make-process.
From EMBA, I know that there are sometimes problems with this test
package, that's why we have suppressed
process-tests/multiple-threads-waiting on EMBA, for example. Maybe
something similar shall be done for HYDRA?

I've CC'ed Philipp Stephani, the author of this test package, for
further analysis.

Best regards, Michael.





^ permalink raw reply	[flat|nested] 11+ messages in thread

* bug#56424: hydra.nixos coverage job times out
  2022-07-10  2:02   ` Glenn Morris
  2022-07-10 12:39     ` Michael Albinus
@ 2022-07-10 13:00     ` Lars Ingebrigtsen
  1 sibling, 0 replies; 11+ messages in thread
From: Lars Ingebrigtsen @ 2022-07-10 13:00 UTC (permalink / raw)
  To: Glenn Morris; +Cc: Po Lu, 56424

Glenn Morris <rgm@gnu.org> writes:

> Again; if no-one is looking at this CI system any more (which is fine),
> then it would be better to stop wasting the electricity.

I look at both Nixos and EMBA regularly, but if I see that it's all red,
I assume that the CI system has a problem (which is correct 93% of the
time, because those systems seem both odd and catastrophically
underpowered) and wait for somebody to reboot them.

I'd love for somebody to finally set up a real CI system with reasonable
power (so that a build takes a few minutes instead of a few hours) and a
normal OS.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no





^ permalink raw reply	[flat|nested] 11+ messages in thread

* bug#56424: hydra.nixos coverage job times out
  2022-07-10 10:44       ` Michael Albinus
@ 2022-07-10 20:09         ` Michael Albinus
  0 siblings, 0 replies; 11+ messages in thread
From: Michael Albinus @ 2022-07-10 20:09 UTC (permalink / raw)
  To: Lars Ingebrigtsen; +Cc: Po Lu, Glenn Morris, 56424

Michael Albinus <michael.albinus@gmx.de> writes:

Hi,

>>>> Doesn't seem to be timing out now, but it's failing.
>>>>
>>>> https://hydra.nixos.org/build/182893714/nixlog/1
>>>>
>>>> lisp/dnd-tests.log:
>>>>    FAILED dnd-tests-begin-drag-files "Method `mock' should specify
>>>> both encoding and decoding command or an scp program"
>>>>    FAILED dnd-tests-begin-file-drag "Method `mock' should specify
>>>> both encoding and decoding command or an scp program"
>>>>
>>>> I've added Po to the CCs.
>>>
>>> Does the build system have scp installed?
>>>
>>> Also, most of the remote file test code involved here wasn't written by
>>> me.  I think it was Michael Albinus.
>>
>> Right; adding Michael to the CCs.
>
> Thanks for the heads-up. I've instrumented dnd-tests-begin-drag-files
> for hydra; let's see what happens.

The dnd-tests on hydra shall be fixed now.

Since this bug report is about process-tests, we shall keep it open.

Best regards, Michael.





^ permalink raw reply	[flat|nested] 11+ messages in thread

* bug#56424: hydra.nixos coverage job times out
  2022-07-10 12:39     ` Michael Albinus
@ 2022-07-19 12:52       ` Philipp Stephani
  0 siblings, 0 replies; 11+ messages in thread
From: Philipp Stephani @ 2022-07-19 12:52 UTC (permalink / raw)
  To: Michael Albinus
  Cc: Po Lu, Glenn Morris, Lars Ingebrigtsen, Philipp Stephani, 56424

Am So., 10. Juli 2022 um 14:40 Uhr schrieb Michael Albinus
<michael.albinus@gmx.de>:
>
> Glenn Morris <rgm@gnu.org> writes:
>
> Hi Glenn,
>
> > With increased logging, and a serial build, we can see that it times out
> > in process-tests:
> > https://hydra.nixos.org/build/183466954
>
> The incrimed test is process-tests/fd-setsize-no-crash/make-process.
> From EMBA, I know that there are sometimes problems with this test
> package, that's why we have suppressed
> process-tests/multiple-threads-waiting on EMBA, for example. Maybe
> something similar shall be done for HYDRA?
>
> I've CC'ed Philipp Stephani, the author of this test package, for
> further analysis.


My 2 cents: These timeouts/failures indicate real bugs in Emacs. They
are incredibly hard to debug due to the enormous complexity in Emacs's
asynchronous process handling code, but just skipping them won't make
the bugs go away. Somebody should at least try to dig into these
issues. But probably that requires something like the CI system
creating a snapshot of the failed tests as a Docker container that one
could SSH into, because the failures tend to be impossible to
reproduce locally.





^ permalink raw reply	[flat|nested] 11+ messages in thread

* bug#56424: hydra.nixos coverage job times out
  2022-07-06 17:47 bug#56424: hydra.nixos coverage job times out Glenn Morris
  2022-07-07  9:04 ` Lars Ingebrigtsen
@ 2022-10-11  1:03 ` Lars Ingebrigtsen
  1 sibling, 0 replies; 11+ messages in thread
From: Lars Ingebrigtsen @ 2022-10-11  1:03 UTC (permalink / raw)
  To: Glenn Morris; +Cc: 56424

Everything on Nixos seems to be failing now.  For instance:

https://hydra.nixos.org/build/193879284

I'm not sure what the actual failure there is at all, but there's a
bunch of warnings like:

Unescaped left brace in regex is deprecated here (and will be fatal in Perl 5.32), passed through in regex; marked by <-- HERE in m/^\s+@([[:alnum:]][[:alnum:]\-]*)({ <-- HERE })?\s*/ at /nix/store/xbxabhr44naykgf768533sijfcamkyr9-texinfo-6.5/share/texinfo/Texinfo/Parser.pm line 5481.

The actual log ends with:

  GEN      ../../lisp/international/cp51932.el
  GEN      ../../lisp/international/eucjp-ms.el
  GEN      ../../etc/charsets/GB180304.map
  GEN      ../../etc/charsets/JISX2131.map
  GEN      charsets.stamp
make[3]: Leaving directory '/build/bc1m3dggd3vw7606h3asba5nsbx3d3qz-source/admin/charsets'
make[2]: Leaving directory '/build/bc1m3dggd3vw7606h3asba5nsbx3d3qz-source/src'
make[1]: *** [Makefile:537: src] Error 2
make[1]: Leaving directory '/build/bc1m3dggd3vw7606h3asba5nsbx3d3qz-source'
make[1]: Entering directory '/build/bc1m3dggd3vw7606h3asba5nsbx3d3qz-source'
***
*** "make all" failed with exit status 2.

Which isn't very helpful.

Hm...  Oh, the actual error is earlier:


mv: cannot stat 'lisp.mk.tmp': No such file or directory
  GEN      globals.h
  GEN      buildobj.h
  GEN      globals.h
  GEN      buildobj.h
cmp: globals.tmp: No such file or directory
mv: cannot stat 'globals.tmp': No such file or directory
cmp: globals.tmp: No such file or directory
make[1]: *** [Makefile:671: gl-stamp] Error 2
make[1]: Leaving directory '/build/bc1m3dggd3vw7606h3asba5nsbx3d3qz-source/src'
make: *** [Makefile:537: src] Error 2
make: *** Waiting for unfinished jobs....

Either something is very wrong on that build machine, or there's
something very wrong in our build.





^ permalink raw reply	[flat|nested] 11+ messages in thread

end of thread, other threads:[~2022-10-11  1:03 UTC | newest]

Thread overview: 11+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2022-07-06 17:47 bug#56424: hydra.nixos coverage job times out Glenn Morris
2022-07-07  9:04 ` Lars Ingebrigtsen
2022-07-07 10:07   ` Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors
2022-07-07 17:34     ` Lars Ingebrigtsen
2022-07-10 10:44       ` Michael Albinus
2022-07-10 20:09         ` Michael Albinus
2022-07-10  2:02   ` Glenn Morris
2022-07-10 12:39     ` Michael Albinus
2022-07-19 12:52       ` Philipp Stephani
2022-07-10 13:00     ` Lars Ingebrigtsen
2022-10-11  1:03 ` Lars Ingebrigtsen

Code repositories for project(s) associated with this public inbox

	https://git.savannah.gnu.org/cgit/emacs.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).