How to block tests on hydra

How to block tests on hydra

[Michael Albinus]

Hi,

recent changes of Tramp block the test on hydra in
gnu:emacs-trunk:coverage. I've instrumented test/Makefile.in in order to
see the output of tramp-tests, and it looks like the blocking test is
tramp-test36-asynchronous-requests. It works properly when running locally.

Well, I have worked on this test the last two weeks. Problems in this
test have always been bad timings and race conditions. In order to find
the cause on hydra, I would need to add traces to this test case again
and again. This would take time; I would not be surprised, if it will
take one or two weeks.

Would it be OK to block all the other Emacs tests on hydra for that
time? If not, I will skip the test if it detecs hydra environment, and
we won't know ever what's behind.

Best regards, Michael.

Re: How to block tests on hydra

[Noam Postavsky]

On Sat, Jul 1, 2017 at 7:48 AM, Michael Albinus <michael.albinus@gmx.de> wrote:

> recent changes of Tramp block the test on hydra in
> gnu:emacs-trunk:coverage. I've instrumented test/Makefile.in in order to
> see the output of tramp-tests, and it looks like the blocking test is
> tramp-test36-asynchronous-requests. It works properly when running locally.

I'm surprised it does anything at all actually, isn't the (add-to-list
'buffers ...) call a bug? Should be using `push' I think.

-              (add-to-list 'buffers (generate-new-buffer "foo")))
+              (push (generate-new-buffer "foo") buffers))

I noticed you added a with-timeout on that test, but it doesn't seem
to be working. Maybe a stronger method would be to use

    (start-process "*watchdog*" nil shell-file-name shell-command-switch
                   (format "sleep 350 ; kill -SIGUSR2 %d" (emacs-pid)))

By the way, I hit the "`tramp-test36-asynchronous-requests' timed out"
message when running locally in an -O0 build, although it succeeds
with an -O2 build. Maybe I just have a weak CPU.

Re: How to block tests on hydra

[Michael Albinus]

Noam Postavsky <npostavs@users.sourceforge.net> writes:

Hi Noam,

>> recent changes of Tramp block the test on hydra in
>> gnu:emacs-trunk:coverage. I've instrumented test/Makefile.in in order to
>> see the output of tramp-tests, and it looks like the blocking test is
>> tramp-test36-asynchronous-requests. It works properly when running locally.
>
> I'm surprised it does anything at all actually, isn't the (add-to-list
> 'buffers ...) call a bug? Should be using `push' I think.
>
> -              (add-to-list 'buffers (generate-new-buffer "foo")))
> +              (push (generate-new-buffer "foo") buffers))

Why that? `add-to-list' is as good as `push' in this case. I haven't seen a
problem with that. If you run in edebug, you'll see that `buffers' keeps
all process buffers.

> I noticed you added a with-timeout on that test, but it doesn't seem
> to be working.

The timeout is a self-defense. And it doesn't trigger at least for me,
because (I believe) the test case is working properly now, and finishes
in time.

> By the way, I hit the "`tramp-test36-asynchronous-requests' timed out"
> message when running locally in an -O0 build, although it succeeds
> with an -O2 build. Maybe I just have a weak CPU.

How does it make a difference? Is an asynchronous process to be intended
to run in another thread, on another processor?

Best regards, Michael.

Re: How to block tests on hydra

[Noam Postavsky]

On Tue, Jul 11, 2017 at 3:20 AM, Michael Albinus <michael.albinus@gmx.de> wrote:
>>
>> I'm surprised it does anything at all actually, isn't the (add-to-list
>> 'buffers ...) call a bug? Should be using `push' I think.
>>
>> -              (add-to-list 'buffers (generate-new-buffer "foo")))
>> +              (push (generate-new-buffer "foo") buffers))
>
> Why that? `add-to-list' is as good as `push' in this case. I haven't seen a
> problem with that. If you run in edebug, you'll see that `buffers' keeps
> all process buffers.

But you have lexical-binding:t set at the top of the file, and
`buffer' is a let-bound variable, whereas `add-to-list' operates on
dynamic variables. Hmm, looking at the disassembly it seems that
add-to-list's compiler macro fixed up the problem, although I thought
it should give a compile warning in this situation.

>
>> I noticed you added a with-timeout on that test, but it doesn't seem
>> to be working.
>
> The timeout is a self-defense. And it doesn't trigger at least for me,
> because (I believe) the test case is working properly now, and finishes
> in time.

What about this one:

http://hydra.nixos.org/eval/1373949
https://nix-cache.s3.amazonaws.com/log/v7ndmrhdhjw76v9mzghjyijnsmw2npl8-emacs-coverage-unknown.drv

>
>> By the way, I hit the "`tramp-test36-asynchronous-requests' timed out"
>> message when running locally in an -O0 build, although it succeeds
>> with an -O2 build. Maybe I just have a weak CPU.
>
> How does it make a difference? Is an asynchronous process to be intended
> to run in another thread, on another processor?

I don't know why it makes a difference.

Re: How to block tests on hydra

[Michael Albinus]

Noam Postavsky <npostavs@users.sourceforge.net> writes:

Hi Noam,

>>> I'm surprised it does anything at all actually, isn't the (add-to-list
>>> 'buffers ...) call a bug? Should be using `push' I think.
>>>
>>> -              (add-to-list 'buffers (generate-new-buffer "foo")))
>>> +              (push (generate-new-buffer "foo") buffers))
>>
>> Why that? `add-to-list' is as good as `push' in this case. I haven't seen a
>> problem with that. If you run in edebug, you'll see that `buffers' keeps
>> all process buffers.
>
> But you have lexical-binding:t set at the top of the file, and
> `buffer' is a let-bound variable, whereas `add-to-list' operates on
> dynamic variables. Hmm, looking at the disassembly it seems that
> add-to-list's compiler macro fixed up the problem, although I thought
> it should give a compile warning in this situation.

Well, as compromise I use now

(setq buffers (cons (generate-new-buffer "foo") buffers))

>>> I noticed you added a with-timeout on that test, but it doesn't seem
>>> to be working.
>>
>> The timeout is a self-defense. And it doesn't trigger at least for me,
>> because (I believe) the test case is working properly now, and finishes
>> in time.
>
> What about this one:
>
> http://hydra.nixos.org/eval/1373949
> https://nix-cache.s3.amazonaws.com/log/v7ndmrhdhjw76v9mzghjyijnsmw2npl8-emacs-coverage-unknown.drv

Hmm, it happens sporadically. I have enabled traces for tramp-tests,
hopefully we'll get more information when it happens next time.

>>> By the way, I hit the "`tramp-test36-asynchronous-requests' timed out"
>>> message when running locally in an -O0 build, although it succeeds
>>> with an -O2 build. Maybe I just have a weak CPU.

Could you try to reproduce it? tramp-tests.log might help then.

Best regards, Michael.

Re: How to block tests on hydra

[Noam Postavsky]

[-- Attachment #1: Type: text/plain, Size: 447 bytes --]

On Wed, Jul 12, 2017 at 8:54 AM, Michael Albinus <michael.albinus@gmx.de> wrote:
>
>>>> By the way, I hit the "`tramp-test36-asynchronous-requests' timed out"
>>>> message when running locally in an -O0 build, although it succeeds
>>>> with an -O2 build. Maybe I just have a weak CPU.
>
> Could you try to reproduce it? tramp-tests.log might help then.

After running the test, I see the attached in *Messages*, not sure
what we're looking for...

[-- Attachment #2: tramp-test36.log.gz --]
[-- Type: application/x-gzip, Size: 1441 bytes --]

Re: How to block tests on hydra

[Michael Albinus]

Noam Postavsky <npostavs@users.sourceforge.net> writes:

Hi Noam,

> After running the test, I see the attached in *Messages*, not sure
> what we're looking for...

This means that the timer eats all resources. I've added an adaptive
adjustment of the timer for this case, could you, pls, retest?

Best regards, Michael.

Re: How to block tests on hydra

[Noam Postavsky]

[-- Attachment #1: Type: text/plain, Size: 260 bytes --]

On Fri, Jul 14, 2017 at 4:58 AM, Michael Albinus <michael.albinus@gmx.de> wrote:
> This means that the timer eats all resources. I've added an adaptive
> adjustment of the timer for this case, could you, pls, retest?

After this change the test passes for me.

[-- Attachment #2: tramp-test36-2.log.gz --]
[-- Type: application/x-gzip, Size: 760 bytes --]

Re: How to block tests on hydra

[Michael Albinus]

Noam Postavsky <npostavs@users.sourceforge.net> writes:

>> This means that the timer eats all resources. I've added an adaptive
>> adjustment of the timer for this case, could you, pls, retest?
>
> After this change the test passes for me.

Thanks. What's left to do is observing hydra. I'll do.

Best regards, Michael.