unofficial mirror of emacs-devel@gnu.org 
 help / color / mirror / code / Atom feed
* EMBA builds failing
@ 2019-04-03 23:57 Dmitry Gutov
  2019-04-04  6:40 ` Michael Albinus
  2019-04-11 13:20 ` Dmitry Gutov
  0 siblings, 2 replies; 19+ messages in thread
From: Dmitry Gutov @ 2019-04-03 23:57 UTC (permalink / raw)
  To: emacs-devel, Ted Zlatanov

Hi Ted,

Most of the recent builds have been failing with

   execution took longer than 1h0m0s seconds

and the log ending with

   Pulling docker image gitlab/gitlab-runner-helper:x86_64-f100a208 ...

Could you look into that?

Thanks,
Dmitry



^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: EMBA builds failing
  2019-04-03 23:57 EMBA builds failing Dmitry Gutov
@ 2019-04-04  6:40 ` Michael Albinus
  2019-04-11 13:20 ` Dmitry Gutov
  1 sibling, 0 replies; 19+ messages in thread
From: Michael Albinus @ 2019-04-04  6:40 UTC (permalink / raw)
  To: Ted Zlatanov; +Cc: Toon Claes, Dmitry Gutov, emacs-devel

Dmitry Gutov <dgutov@yandex.ru> writes:

> Most of the recent builds have been failing with
>
>   execution took longer than 1h0m0s seconds
>
> and the log ending with
>
>   Pulling docker image gitlab/gitlab-runner-helper:x86_64-f100a208 ...

And while we are at this: the emails about failed jobs do not contain
the link to the raw log, again, so I stopped to check regularly. Could
we fix this problem forever? I don't know gitlab by heart, so I don't
know where to go.

> Thanks,
> Dmitry

Best regards, Michael.



^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: EMBA builds failing
  2019-04-03 23:57 EMBA builds failing Dmitry Gutov
  2019-04-04  6:40 ` Michael Albinus
@ 2019-04-11 13:20 ` Dmitry Gutov
  2019-04-11 13:26   ` Toon Claes
  2019-04-16  7:07   ` Toon Claes
  1 sibling, 2 replies; 19+ messages in thread
From: Dmitry Gutov @ 2019-04-11 13:20 UTC (permalink / raw)
  To: emacs-devel, Ted Zlatanov; +Cc: Michael Albinus

The timeout problem seems to be gone now, and the builds are working, 
but if Ted doesn't rejoin us here (hope he's okay), who should I contact 
for administrative access to EMBA?

If we're discussing the option of migrating our workflow, there should 
at least be someone who can configure the Gitlab instance we already have.



^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: EMBA builds failing
  2019-04-11 13:20 ` Dmitry Gutov
@ 2019-04-11 13:26   ` Toon Claes
  2019-04-11 13:31     ` Michael Albinus
  2019-04-11 13:36     ` Dmitry Gutov
  2019-04-16  7:07   ` Toon Claes
  1 sibling, 2 replies; 19+ messages in thread
From: Toon Claes @ 2019-04-11 13:26 UTC (permalink / raw)
  To: Dmitry Gutov; +Cc: Ted Zlatanov, Michael Albinus, emacs-devel

[-- Attachment #1: Type: text/plain, Size: 750 bytes --]

Dmitry Gutov <dgutov@yandex.ru> writes:

> The timeout problem seems to be gone now, and the builds are working,

Thanks! Did you change anything?

> If we're discussing the option of migrating our workflow, there should
> at least be someone who can configure the Gitlab instance we already
> have.

I also have admin access to EMBA. I'm most of the time only lurking this
mailing list, and I've seen the failures, but couldn't find the time to
have a thorough look.

Next to Ted and me, also Michael Albinus and NicolasPetton have an admin
account. Although I'm not sure they also have SSH access to VM.

I'm really welcoming more people to be actively involved in EMBA, so
whenever someone want access, let us know and I'll get it done.


-- Toon

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: EMBA builds failing
  2019-04-11 13:26   ` Toon Claes
@ 2019-04-11 13:31     ` Michael Albinus
  2019-04-11 19:28       ` Toon Claes
  2019-04-11 13:36     ` Dmitry Gutov
  1 sibling, 1 reply; 19+ messages in thread
From: Michael Albinus @ 2019-04-11 13:31 UTC (permalink / raw)
  To: Toon Claes; +Cc: Ted Zlatanov, emacs-devel, Dmitry Gutov

Toon Claes <toon@iotcl.com> writes:

> Next to Ted and me, also Michael Albinus and NicolasPetton have an admin
> account. Although I'm not sure they also have SSH access to VM.

I'm not aware of having ssh access. If I had, I would have tried to fix
the problem with the mail pattern.

> -- Toon

Best regards, Michael.



^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: EMBA builds failing
  2019-04-11 13:26   ` Toon Claes
  2019-04-11 13:31     ` Michael Albinus
@ 2019-04-11 13:36     ` Dmitry Gutov
  1 sibling, 0 replies; 19+ messages in thread
From: Dmitry Gutov @ 2019-04-11 13:36 UTC (permalink / raw)
  To: Toon Claes; +Cc: Ted Zlatanov, Michael Albinus, emacs-devel

On 11.04.2019 16:26, Toon Claes wrote:

>> The timeout problem seems to be gone now, and the builds are working,
> 
> Thanks! Did you change anything?

Not at all. Just went there again to take a look, intending to write a 
more negative letter afterward. Was pleasantly surprised.

> I also have admin access to EMBA. I'm most of the time only lurking this
> mailing list, and I've seen the failures, but couldn't find the time to
> have a thorough look.

Okay, that's good to hear.

> I'm really welcoming more people to be actively involved in EMBA, so
> whenever someone want access, let us know and I'll get it done.

I don't have any immediate plans, but having access wouldn't hurt. Do I 
need to register at the web interface first? It says "Email domain is 
not authorized for sign-up".



^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: EMBA builds failing
  2019-04-11 13:31     ` Michael Albinus
@ 2019-04-11 19:28       ` Toon Claes
  0 siblings, 0 replies; 19+ messages in thread
From: Toon Claes @ 2019-04-11 19:28 UTC (permalink / raw)
  To: Michael Albinus; +Cc: Ted Zlatanov, emacs-devel, Dmitry Gutov

[-- Attachment #1: Type: text/plain, Size: 237 bytes --]

Michael Albinus <michael.albinus@gmx.de> writes:

> I'm not aware of having ssh access. If I had, I would have tried to fix
> the problem with the mail pattern.

I think you'd need to ask FSF admins to add SSH keys to that VM.


-- Toon

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: EMBA builds failing
  2019-04-11 13:20 ` Dmitry Gutov
  2019-04-11 13:26   ` Toon Claes
@ 2019-04-16  7:07   ` Toon Claes
  2019-04-16  9:24     ` Michael Albinus
  1 sibling, 1 reply; 19+ messages in thread
From: Toon Claes @ 2019-04-16  7:07 UTC (permalink / raw)
  To: Dmitry Gutov; +Cc: Ted Zlatanov, Michael Albinus, emacs-devel

[-- Attachment #1: Type: text/plain, Size: 521 bytes --]

I have again applied the patch so the email gives a direct link to the
raw log file.

I've also started working on the change in GitLab itself, so this patch
is no longer needed, see
https://gitlab.com/gitlab-org/gitlab-ce/issues/56849

Hopefully we'll get this change deployed in a month or so.

But I couldn't figure out why we always needed to reapply this patch?
Ted, have you been doing updates of GitLab? Or does it do it
automatically? Cause I see it runs 11.9.7 now, while we initially
deployed 11.1.4.


-- Toon

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: EMBA builds failing
  2019-04-16  7:07   ` Toon Claes
@ 2019-04-16  9:24     ` Michael Albinus
  2019-04-19 19:29       ` Toon Claes
  0 siblings, 1 reply; 19+ messages in thread
From: Michael Albinus @ 2019-04-16  9:24 UTC (permalink / raw)
  To: Toon Claes; +Cc: Ted Zlatanov, emacs-devel, Dmitry Gutov

Toon Claes <toon@iotcl.com> writes:

> I have again applied the patch so the email gives a direct link to the
> raw log file.

It still doesn't work. I'm getting emails with the link

Job #1378 ( https://emba.gnu.org/emacs/emacs/-/jobs/1378 )

> -- Toon

Best regards, Michael.



^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: EMBA builds failing
  2019-04-16  9:24     ` Michael Albinus
@ 2019-04-19 19:29       ` Toon Claes
  2019-04-19 19:34         ` Michael Albinus
                           ` (2 more replies)
  0 siblings, 3 replies; 19+ messages in thread
From: Toon Claes @ 2019-04-19 19:29 UTC (permalink / raw)
  To: Michael Albinus; +Cc: Ted Zlatanov, Dmitry Gutov, emacs-devel

[-- Attachment #1: Type: text/plain, Size: 1338 bytes --]

Michael Albinus <michael.albinus@gmx.de> writes:

> It still doesn't work. I'm getting emails with the link
>
> Job #1378 ( https://emba.gnu.org/emacs/emacs/-/jobs/1378 )

Ah, the Ansible playbook I ran didn't restart GitLab after patching. The
link should be good now.

I also finalized the change to be included in GitLab Core, it will be
available in GitLab 11.11:
https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/27409

I'm still seeing quite a number of timeout failures, so I upped the
limit to 1h 30min. Hopefully that gives more reliable results.

Although the latest failure on master seems to be legit:
See https://emba.gnu.org/emacs/emacs/-/jobs/1430/raw

> SUMMARY OF TEST RESULTS
> -----------------------
> Files examined: 212
> Ran 3102 tests, 2981 results as expected, 1 unexpected, 120 skipped
> 1 files contained unexpected results:
>   lisp/progmodes/flymake-tests.log
> Makefile:313: recipe for target 'check-doit' failed
> make[2]: *** [check-doit] Error 1
> make[2]: Leaving directory '/builds/emacs/emacs/test'
> Makefile:288: recipe for target 'check-expensive' failed
> make[1]: *** [check-expensive] Error 2
> make[1]: Leaving directory '/builds/emacs/emacs/test'
> Makefile:962: recipe for target 'check-expensive' failed
> make: *** [check-expensive] Error 2
> ERROR: Job failed: exit code 1


-- Toon



[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: EMBA builds failing
  2019-04-19 19:29       ` Toon Claes
@ 2019-04-19 19:34         ` Michael Albinus
  2019-04-20  9:16           ` Robert Pluim
  2019-04-19 23:10         ` Dmitry Gutov
  2019-04-22 20:36         ` Michael Albinus
  2 siblings, 1 reply; 19+ messages in thread
From: Michael Albinus @ 2019-04-19 19:34 UTC (permalink / raw)
  To: Toon Claes; +Cc: Ted Zlatanov, Dmitry Gutov, emacs-devel

Toon Claes <toon@iotcl.com> writes:

Hi Toon,

>> It still doesn't work. I'm getting emails with the link
>>
>> Job #1378 ( https://emba.gnu.org/emacs/emacs/-/jobs/1378 )
>
> Ah, the Ansible playbook I ran didn't restart GitLab after patching. The
> link should be good now.

Yes, it is. Thanks!

> Although the latest failure on master seems to be legit:
> See https://emba.gnu.org/emacs/emacs/-/jobs/1430/raw
>
>> SUMMARY OF TEST RESULTS
>> -----------------------
>> Files examined: 212
>> Ran 3102 tests, 2981 results as expected, 1 unexpected, 120 skipped
>> 1 files contained unexpected results:
>>   lisp/progmodes/flymake-tests.log

This happens since one week or two. Maybe I could find the time to
bisect this, and write a bug report about.

> -- Toon

Best regards, Michael.



^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: EMBA builds failing
  2019-04-19 19:29       ` Toon Claes
  2019-04-19 19:34         ` Michael Albinus
@ 2019-04-19 23:10         ` Dmitry Gutov
  2019-04-23 10:21           ` Toon Claes
  2019-04-22 20:36         ` Michael Albinus
  2 siblings, 1 reply; 19+ messages in thread
From: Dmitry Gutov @ 2019-04-19 23:10 UTC (permalink / raw)
  To: Toon Claes, Michael Albinus; +Cc: Ted Zlatanov, emacs-devel

On 19.04.2019 22:29, Toon Claes wrote:

> I also finalized the change to be included in GitLab Core, it will be
> available in GitLab 11.11:
> https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/27409

Terrific. That's certainly the best way to do this.

> I'm still seeing quite a number of timeout failures, so I upped the
> limit to 1h 30min. Hopefully that gives more reliable results.

Any ideas on the cause of the problem? In the jobs that I saw timeout, 
it was pulling the aforementioned docker image.

But in the pipelines that run to completion, there's usually no mention 
of gitlab-runner-helper at all in the log. Odd, isn't it?

> See https://emba.gnu.org/emacs/emacs/-/jobs/1430/raw



^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: EMBA builds failing
  2019-04-19 19:34         ` Michael Albinus
@ 2019-04-20  9:16           ` Robert Pluim
  2019-04-20 14:57             ` Paul Eggert
  2019-04-20 19:18             ` Michael Albinus
  0 siblings, 2 replies; 19+ messages in thread
From: Robert Pluim @ 2019-04-20  9:16 UTC (permalink / raw)
  To: Michael Albinus; +Cc: Toon Claes, emacs-devel, Ted Zlatanov, Dmitry Gutov

>>>>> On Fri, 19 Apr 2019 21:34:48 +0200, Michael Albinus <michael.albinus@gmx.de> said:

    >>> SUMMARY OF TEST RESULTS ----------------------- Files
    >>> examined: 212 Ran 3102 tests, 2981 results as expected, 1
    >>> unexpected, 120 skipped 1 files contained unexpected results:
    >>> lisp/progmodes/flymake-tests.log

    Michael> This happens since one week or two. Maybe I could find
    Michael> the time to bisect this, and write a bug report about.

67c3a3af1d9e2582193d5ea33b6c190e79ad56e6

Joao said he was looking at it, but that was a while ago. Maybe we
should disable that test for now?

Robert



^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: EMBA builds failing
  2019-04-20  9:16           ` Robert Pluim
@ 2019-04-20 14:57             ` Paul Eggert
  2019-04-20 19:18             ` Michael Albinus
  1 sibling, 0 replies; 19+ messages in thread
From: Paul Eggert @ 2019-04-20 14:57 UTC (permalink / raw)
  To: emacs-devel

Robert Pluim wrote:
>      >>> lisp/progmodes/flymake-tests.log
> 
>      Michael> This happens since one week or two. Maybe I could find
>      Michael> the time to bisect this, and write a bug report about.
> 
> 67c3a3af1d9e2582193d5ea33b6c190e79ad56e6
> 
> Joao said he was looking at it, but that was a while ago. Maybe we
> should disable that test for now?

+1. That test has been failing for quite some time now. I ignore the failure, 
but it's confusing to people who don't often run tests (and it sets a bad 
example to those of us who do).



^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: EMBA builds failing
  2019-04-20  9:16           ` Robert Pluim
  2019-04-20 14:57             ` Paul Eggert
@ 2019-04-20 19:18             ` Michael Albinus
  1 sibling, 0 replies; 19+ messages in thread
From: Michael Albinus @ 2019-04-20 19:18 UTC (permalink / raw)
  To: emacs-devel

Robert Pluim <rpluim@gmail.com> writes:

> Joao said he was looking at it, but that was a while ago. Maybe we
> should disable that test for now?

Glenn did this for hydra already; I've marked it as failed also for emba.

> Robert

Best regards, Michael.



^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: EMBA builds failing
  2019-04-19 19:29       ` Toon Claes
  2019-04-19 19:34         ` Michael Albinus
  2019-04-19 23:10         ` Dmitry Gutov
@ 2019-04-22 20:36         ` Michael Albinus
  2019-04-23 10:09           ` Toon Claes
  2 siblings, 1 reply; 19+ messages in thread
From: Michael Albinus @ 2019-04-22 20:36 UTC (permalink / raw)
  To: Toon Claes; +Cc: Ted Zlatanov, emacs-devel, Dmitry Gutov

Toon Claes <toon@iotcl.com> writes:

Hi Toon,

>> It still doesn't work. I'm getting emails with the link
>>
>> Job #1378 ( https://emba.gnu.org/emacs/emacs/-/jobs/1378 )
>
> Ah, the Ansible playbook I ran didn't restart GitLab after patching. The
> link should be good now.

Unfortunately, with recent messages this is broken, again.

> -- Toon

Best regards, Michael.



^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: EMBA builds failing
  2019-04-22 20:36         ` Michael Albinus
@ 2019-04-23 10:09           ` Toon Claes
  2019-04-23 12:57             ` Michael Albinus
  0 siblings, 1 reply; 19+ messages in thread
From: Toon Claes @ 2019-04-23 10:09 UTC (permalink / raw)
  To: Michael Albinus; +Cc: Ted Zlatanov, emacs-devel, Dmitry Gutov

[-- Attachment #1: Type: text/plain, Size: 196 bytes --]

Michael Albinus <michael.albinus@gmx.de> writes:

> Unfortunately, with recent messages this is broken, again.


I've noticed. emba was upgraded to GitLab 11.10. I've reapplied the fix.


-- Toon

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: EMBA builds failing
  2019-04-19 23:10         ` Dmitry Gutov
@ 2019-04-23 10:21           ` Toon Claes
  0 siblings, 0 replies; 19+ messages in thread
From: Toon Claes @ 2019-04-23 10:21 UTC (permalink / raw)
  To: Dmitry Gutov; +Cc: Ted Zlatanov, Michael Albinus, emacs-devel

[-- Attachment #1: Type: text/plain, Size: 1015 bytes --]

Dmitry Gutov <dgutov@yandex.ru> writes:

> Any ideas on the cause of the problem? In the jobs that I saw timeout,
> it was pulling the aforementioned docker image.
>
> But in the pipelines that run to completion, there's usually no
> mention of gitlab-runner-helper at all in the log. Odd, isn't it?

I'm not sure, but it seems to me this gitlab-runner-helper is used to
upload artifacts in case there's a failure. So the timeout happened
*before* that image was pulled.

There must be a reason why the builds time out, maybe because the
machine was too busy doing other things?

Unfortunately GitLab CI does not provide timestamps in their log, cause
that could've helped diagnose, see:
https://gitlab.com/gitlab-org/gitlab-ce/issues/22745

I have not other ideas to diagnose.

We were planning to move the GitLab CI runner to a separate machine, and
we've requested FSF sysadmins to set it up, but that never got
completed. Ted was following that up, unfortunately I haven't heard from
him for a while.


-- Toon

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: EMBA builds failing
  2019-04-23 10:09           ` Toon Claes
@ 2019-04-23 12:57             ` Michael Albinus
  0 siblings, 0 replies; 19+ messages in thread
From: Michael Albinus @ 2019-04-23 12:57 UTC (permalink / raw)
  To: Toon Claes; +Cc: Ted Zlatanov, emacs-devel, Dmitry Gutov

Toon Claes <toon@iotcl.com> writes:

>> Unfortunately, with recent messages this is broken, again.
>
> I've noticed. emba was upgraded to GitLab 11.10. I've reapplied the fix.

Thanks!

> -- Toon

Best regards, Michael.



^ permalink raw reply	[flat|nested] 19+ messages in thread

end of thread, other threads:[~2019-04-23 12:57 UTC | newest]

Thread overview: 19+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2019-04-03 23:57 EMBA builds failing Dmitry Gutov
2019-04-04  6:40 ` Michael Albinus
2019-04-11 13:20 ` Dmitry Gutov
2019-04-11 13:26   ` Toon Claes
2019-04-11 13:31     ` Michael Albinus
2019-04-11 19:28       ` Toon Claes
2019-04-11 13:36     ` Dmitry Gutov
2019-04-16  7:07   ` Toon Claes
2019-04-16  9:24     ` Michael Albinus
2019-04-19 19:29       ` Toon Claes
2019-04-19 19:34         ` Michael Albinus
2019-04-20  9:16           ` Robert Pluim
2019-04-20 14:57             ` Paul Eggert
2019-04-20 19:18             ` Michael Albinus
2019-04-19 23:10         ` Dmitry Gutov
2019-04-23 10:21           ` Toon Claes
2019-04-22 20:36         ` Michael Albinus
2019-04-23 10:09           ` Toon Claes
2019-04-23 12:57             ` Michael Albinus

Code repositories for project(s) associated with this public inbox

	https://git.savannah.gnu.org/cgit/emacs.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).