EMBA builds failing

EMBA builds failing

[Dmitry Gutov]

Hi Ted,

Most of the recent builds have been failing with

   execution took longer than 1h0m0s seconds

and the log ending with

   Pulling docker image gitlab/gitlab-runner-helper:x86_64-f100a208 ...

Could you look into that?

Thanks,
Dmitry

Re: EMBA builds failing

[Michael Albinus]

Dmitry Gutov <dgutov@yandex.ru> writes:

> Most of the recent builds have been failing with
>
>   execution took longer than 1h0m0s seconds
>
> and the log ending with
>
>   Pulling docker image gitlab/gitlab-runner-helper:x86_64-f100a208 ...

And while we are at this: the emails about failed jobs do not contain
the link to the raw log, again, so I stopped to check regularly. Could
we fix this problem forever? I don't know gitlab by heart, so I don't
know where to go.

> Thanks,
> Dmitry

Best regards, Michael.

Re: EMBA builds failing

[Dmitry Gutov]

The timeout problem seems to be gone now, and the builds are working, 
but if Ted doesn't rejoin us here (hope he's okay), who should I contact 
for administrative access to EMBA?

If we're discussing the option of migrating our workflow, there should 
at least be someone who can configure the Gitlab instance we already have.

Re: EMBA builds failing

[Toon Claes]

[-- Attachment #1: Type: text/plain, Size: 750 bytes --]

Dmitry Gutov <dgutov@yandex.ru> writes:

> The timeout problem seems to be gone now, and the builds are working,

Thanks! Did you change anything?

> If we're discussing the option of migrating our workflow, there should
> at least be someone who can configure the Gitlab instance we already
> have.

I also have admin access to EMBA. I'm most of the time only lurking this
mailing list, and I've seen the failures, but couldn't find the time to
have a thorough look.

Next to Ted and me, also Michael Albinus and NicolasPetton have an admin
account. Although I'm not sure they also have SSH access to VM.

I'm really welcoming more people to be actively involved in EMBA, so
whenever someone want access, let us know and I'll get it done.


-- Toon

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: EMBA builds failing

[Michael Albinus]

Toon Claes <toon@iotcl.com> writes:

> Next to Ted and me, also Michael Albinus and NicolasPetton have an admin
> account. Although I'm not sure they also have SSH access to VM.

I'm not aware of having ssh access. If I had, I would have tried to fix
the problem with the mail pattern.

> -- Toon

Best regards, Michael.

Re: EMBA builds failing

[Dmitry Gutov]

On 11.04.2019 16:26, Toon Claes wrote:

>> The timeout problem seems to be gone now, and the builds are working,
> 
> Thanks! Did you change anything?

Not at all. Just went there again to take a look, intending to write a 
more negative letter afterward. Was pleasantly surprised.

> I also have admin access to EMBA. I'm most of the time only lurking this
> mailing list, and I've seen the failures, but couldn't find the time to
> have a thorough look.

Okay, that's good to hear.

> I'm really welcoming more people to be actively involved in EMBA, so
> whenever someone want access, let us know and I'll get it done.

I don't have any immediate plans, but having access wouldn't hurt. Do I 
need to register at the web interface first? It says "Email domain is 
not authorized for sign-up".

Re: EMBA builds failing

[Toon Claes]

[-- Attachment #1: Type: text/plain, Size: 237 bytes --]

Michael Albinus <michael.albinus@gmx.de> writes:

> I'm not aware of having ssh access. If I had, I would have tried to fix
> the problem with the mail pattern.

I think you'd need to ask FSF admins to add SSH keys to that VM.


-- Toon

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: EMBA builds failing

[Toon Claes]

[-- Attachment #1: Type: text/plain, Size: 521 bytes --]

I have again applied the patch so the email gives a direct link to the
raw log file.

I've also started working on the change in GitLab itself, so this patch
is no longer needed, see
https://gitlab.com/gitlab-org/gitlab-ce/issues/56849

Hopefully we'll get this change deployed in a month or so.

But I couldn't figure out why we always needed to reapply this patch?
Ted, have you been doing updates of GitLab? Or does it do it
automatically? Cause I see it runs 11.9.7 now, while we initially
deployed 11.1.4.


-- Toon

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: EMBA builds failing

[Michael Albinus]

Toon Claes <toon@iotcl.com> writes:

> I have again applied the patch so the email gives a direct link to the
> raw log file.

It still doesn't work. I'm getting emails with the link

Job #1378 ( https://emba.gnu.org/emacs/emacs/-/jobs/1378 )

> -- Toon

Best regards, Michael.

Re: EMBA builds failing

[Toon Claes]

[-- Attachment #1: Type: text/plain, Size: 1338 bytes --]

Michael Albinus <michael.albinus@gmx.de> writes:

> It still doesn't work. I'm getting emails with the link
>
> Job #1378 ( https://emba.gnu.org/emacs/emacs/-/jobs/1378 )

Ah, the Ansible playbook I ran didn't restart GitLab after patching. The
link should be good now.

I also finalized the change to be included in GitLab Core, it will be
available in GitLab 11.11:
https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/27409

I'm still seeing quite a number of timeout failures, so I upped the
limit to 1h 30min. Hopefully that gives more reliable results.

Although the latest failure on master seems to be legit:
See https://emba.gnu.org/emacs/emacs/-/jobs/1430/raw

> SUMMARY OF TEST RESULTS
> -----------------------
> Files examined: 212
> Ran 3102 tests, 2981 results as expected, 1 unexpected, 120 skipped
> 1 files contained unexpected results:
>   lisp/progmodes/flymake-tests.log
> Makefile:313: recipe for target 'check-doit' failed
> make[2]: *** [check-doit] Error 1
> make[2]: Leaving directory '/builds/emacs/emacs/test'
> Makefile:288: recipe for target 'check-expensive' failed
> make[1]: *** [check-expensive] Error 2
> make[1]: Leaving directory '/builds/emacs/emacs/test'
> Makefile:962: recipe for target 'check-expensive' failed
> make: *** [check-expensive] Error 2
> ERROR: Job failed: exit code 1


-- Toon



[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: EMBA builds failing

[Michael Albinus]

Toon Claes <toon@iotcl.com> writes:

Hi Toon,

>> It still doesn't work. I'm getting emails with the link
>>
>> Job #1378 ( https://emba.gnu.org/emacs/emacs/-/jobs/1378 )
>
> Ah, the Ansible playbook I ran didn't restart GitLab after patching. The
> link should be good now.

Yes, it is. Thanks!

> Although the latest failure on master seems to be legit:
> See https://emba.gnu.org/emacs/emacs/-/jobs/1430/raw
>
>> SUMMARY OF TEST RESULTS
>> -----------------------
>> Files examined: 212
>> Ran 3102 tests, 2981 results as expected, 1 unexpected, 120 skipped
>> 1 files contained unexpected results:
>>   lisp/progmodes/flymake-tests.log

This happens since one week or two. Maybe I could find the time to
bisect this, and write a bug report about.

> -- Toon

Best regards, Michael.

Re: EMBA builds failing

[Dmitry Gutov]

On 19.04.2019 22:29, Toon Claes wrote:

> I also finalized the change to be included in GitLab Core, it will be
> available in GitLab 11.11:
> https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/27409

Terrific. That's certainly the best way to do this.

> I'm still seeing quite a number of timeout failures, so I upped the
> limit to 1h 30min. Hopefully that gives more reliable results.

Any ideas on the cause of the problem? In the jobs that I saw timeout, 
it was pulling the aforementioned docker image.

But in the pipelines that run to completion, there's usually no mention 
of gitlab-runner-helper at all in the log. Odd, isn't it?

> See https://emba.gnu.org/emacs/emacs/-/jobs/1430/raw

Re: EMBA builds failing

[Robert Pluim]

>>>>> On Fri, 19 Apr 2019 21:34:48 +0200, Michael Albinus <michael.albinus@gmx.de> said:

    >>> SUMMARY OF TEST RESULTS ----------------------- Files
    >>> examined: 212 Ran 3102 tests, 2981 results as expected, 1
    >>> unexpected, 120 skipped 1 files contained unexpected results:
    >>> lisp/progmodes/flymake-tests.log

    Michael> This happens since one week or two. Maybe I could find
    Michael> the time to bisect this, and write a bug report about.

67c3a3af1d9e2582193d5ea33b6c190e79ad56e6

Joao said he was looking at it, but that was a while ago. Maybe we
should disable that test for now?

Robert

Re: EMBA builds failing

[Paul Eggert]

Robert Pluim wrote:
>      >>> lisp/progmodes/flymake-tests.log
> 
>      Michael> This happens since one week or two. Maybe I could find
>      Michael> the time to bisect this, and write a bug report about.
> 
> 67c3a3af1d9e2582193d5ea33b6c190e79ad56e6
> 
> Joao said he was looking at it, but that was a while ago. Maybe we
> should disable that test for now?

+1. That test has been failing for quite some time now. I ignore the failure, 
but it's confusing to people who don't often run tests (and it sets a bad 
example to those of us who do).

Re: EMBA builds failing

[Michael Albinus]

Robert Pluim <rpluim@gmail.com> writes:

> Joao said he was looking at it, but that was a while ago. Maybe we
> should disable that test for now?

Glenn did this for hydra already; I've marked it as failed also for emba.

> Robert

Best regards, Michael.

Re: EMBA builds failing

[Michael Albinus]

Toon Claes <toon@iotcl.com> writes:

Hi Toon,

>> It still doesn't work. I'm getting emails with the link
>>
>> Job #1378 ( https://emba.gnu.org/emacs/emacs/-/jobs/1378 )
>
> Ah, the Ansible playbook I ran didn't restart GitLab after patching. The
> link should be good now.

Unfortunately, with recent messages this is broken, again.

> -- Toon

Best regards, Michael.

Re: EMBA builds failing

[Toon Claes]

[-- Attachment #1: Type: text/plain, Size: 196 bytes --]

Michael Albinus <michael.albinus@gmx.de> writes:

> Unfortunately, with recent messages this is broken, again.


I've noticed. emba was upgraded to GitLab 11.10. I've reapplied the fix.


-- Toon

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: EMBA builds failing

[Toon Claes]

[-- Attachment #1: Type: text/plain, Size: 1015 bytes --]

Dmitry Gutov <dgutov@yandex.ru> writes:

> Any ideas on the cause of the problem? In the jobs that I saw timeout,
> it was pulling the aforementioned docker image.
>
> But in the pipelines that run to completion, there's usually no
> mention of gitlab-runner-helper at all in the log. Odd, isn't it?

I'm not sure, but it seems to me this gitlab-runner-helper is used to
upload artifacts in case there's a failure. So the timeout happened
*before* that image was pulled.

There must be a reason why the builds time out, maybe because the
machine was too busy doing other things?

Unfortunately GitLab CI does not provide timestamps in their log, cause
that could've helped diagnose, see:
https://gitlab.com/gitlab-org/gitlab-ce/issues/22745

I have not other ideas to diagnose.

We were planning to move the GitLab CI runner to a separate machine, and
we've requested FSF sysadmins to set it up, but that never got
completed. Ted was following that up, unfortunately I haven't heard from
him for a while.


-- Toon

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: EMBA builds failing

[Michael Albinus]

Toon Claes <toon@iotcl.com> writes:

>> Unfortunately, with recent messages this is broken, again.
>
> I've noticed. emba was upgraded to GitLab 11.10. I've reapplied the fix.

Thanks!

> -- Toon

Best regards, Michael.