icalendar-real-world test sometimes fails

icalendar-real-world test sometimes fails

[Glenn Morris]

Hi,

On hydra (which runs automated builds), the icalendar-real-world
sometimes fails. See eg:

http://hydra.nixos.org/build/12699658/log/raw

At first glance, it seems like the problem is that the output sometimes
has a "# coding: utf-8" header, and sometimes it does not.

Any ideas on how to make this test more robust?

Re: icalendar-real-world test sometimes fails

[Ulf Jasper]

Glenn Morris <rgm@gnu.org> writes:

> On hydra (which runs automated builds), the icalendar-real-world
> sometimes fails. See eg:
>
> http://hydra.nixos.org/build/12699658/log/raw
>
> At first glance, it seems like the problem is that the output sometimes
> has a "# coding: utf-8" header, and sometimes it does not.

Unfortunately I cannot reproduce the problem.  But I agree that the "#
coding..." thing appears to be causing the failure.

> Any ideas on how to make this test more robust?

I'm wondering why the test fails "sometimes".  Does that really mean
that the test finishes successfully one day but fails the next day?

In any case I would prefer to fix the root cause instead of modifying
the test case.  The "# coding..."  should not get inserted into the
diary file, should it?  Where does it come from?

Re: icalendar-real-world test sometimes fails

[Glenn Morris]

Ulf Jasper wrote:

> I'm wondering why the test fails "sometimes".  Does that really mean
> that the test finishes successfully one day but fails the next day?

Yes! :)
It (randomly?) pings back and forth between success and failure for no
reason I can fathom.

http://hydra.nixos.org/build/12698548    # works, trunk r117565
http://hydra.nixos.org/build/12699658    # fails, trunk r117566
http://hydra.nixos.org/build/12711693    # works, trunk r117567

It has been doing this for a while, in emacs-24 branch too.
It usually works, but once in a while it fails.

> In any case I would prefer to fix the root cause instead of modifying
> the test case.  The "# coding..."  should not get inserted into the
> diary file, should it?  Where does it come from?

That's the question...

Re: icalendar-real-world test sometimes fails

[Ulf Jasper]

Glenn Morris <rgm@gnu.org> writes:

> Ulf Jasper wrote:
>
>> In any case I would prefer to fix the root cause instead of modifying
>> the test case.  The "# coding..."  should not get inserted into the
>> diary file, should it?  Where does it come from?
>
> That's the question...

The order in which the tests are called?  No.  The machine on which the
tests are running?  No.  I give up.

I added a workaround and some debugging output in
test/automated/icalendar-tests.el.  Now the tests should succeed and we
may get a clue as to why the mysterious header gets inserted.

It's in the trunk (rev. 117578).

Re: icalendar-real-world test sometimes fails

[Glenn Morris]

Ulf Jasper wrote:

> I added a workaround and some debugging output in
> test/automated/icalendar-tests.el.  Now the tests should succeed and we
> may get a clue as to why the mysterious header gets inserted.

I don't think so, because if there is no test failure, we don't get to
see the output from hydra.

Re: icalendar-real-world test sometimes fails

[Ulf Jasper]

Glenn Morris <rgm@gnu.org> writes:

> Ulf Jasper wrote:
>
>> I added a workaround and some debugging output in
>> test/automated/icalendar-tests.el.  Now the tests should succeed and we
>> may get a clue as to why the mysterious header gets inserted.
>
> I don't think so, because if there is no test failure, we don't get to
> see the output from hydra.

I modified the tests so that when the header is present it is removed
and then some "debugging output" is printed.  Currently it contains just the
active test and the list of active modes.  Of course one has to check
the log files manually.

Re: icalendar-real-world test sometimes fails

[Glenn Morris]

Ulf Jasper wrote:

> Of course one has to check the log files manually.

We can't do that on hydra, AFAIK.

Re: icalendar-real-world test sometimes fails

[Ulf Jasper]

Glenn Morris <rgm@gnu.org> writes:

> Ulf Jasper wrote:
>
>> Of course one has to check the log files manually.
>
> We can't do that on hydra, AFAIK.

I wrote a small program which downloads the latest raw log files and
looks for the debugging output I added.  That should be enough for the
time being.

Re: icalendar-real-world test sometimes fails

[Glenn Morris]

Ulf Jasper wrote:

> I wrote a small program which downloads the latest raw log files and
> looks for the debugging output I added.  That should be enough for the
> time being.

I think we are miscommunicating, because I'm saying that you won't see
that output. Do you see messages from any other successful tests?

Re: icalendar-real-world test sometimes fails

[Ulf Jasper]

Glenn Morris <rgm@gnu.org> writes:

> Ulf Jasper wrote:
>
>> I wrote a small program which downloads the latest raw log files and
>> looks for the debugging output I added.  That should be enough for the
>> time being.
>
> I think we are miscommunicating, because I'm saying that you won't see
> that output. Do you see messages from any other successful tests?

Stupid me!  No, I don't see any messages.

If I understand test/automated/Makefile correctly then log files are
written for each test.  Are they kept when the hydra job is finished?

Re: icalendar-real-world test sometimes fails

[Glenn Morris]

Ulf Jasper wrote:

> If I understand test/automated/Makefile correctly then log files are
> written for each test.  Are they kept when the hydra job is finished?

I don't think so, though it may be possible for me to make it so they are.

Anyway, we just had another failure, this time in
icalendar--decode-isodatetime:

http://hydra.nixos.org/build/12974200/log/raw

It's probably some quirk of the system, but it would be good to suppress
such failures if possible.

Re: icalendar-real-world test sometimes fails

[Ulf Jasper]

Glenn Morris <rgm@gnu.org> writes:

> Ulf Jasper wrote:
>
>> If I understand test/automated/Makefile correctly then log files are
>> written for each test.  Are they kept when the hydra job is finished?
>
> I don't think so, though it may be possible for me to make it so they
> are.

That would be very helpful. 

> Anyway, we just had another failure, this time in
> icalendar--decode-isodatetime:
>
> http://hydra.nixos.org/build/12974200/log/raw
>
> It's probably some quirk of the system, but it would be good to suppress
> such failures if possible.

That is the new test `icalendar--decode-isodatetime' which I added just
yesterday.  Before committing I have checked it on two different
systems.

Luckily I forgot to remove the lines

  (message "%s" (current-time-zone (encode-time 0 0 10 1 1 2013 0)))
  (message "%s" (current-time-zone (encode-time 0 0 10 1 8 2013 0)))

from the test.  They should result in the ouput

  (3600 EET)
  (7200 EET)

but the log file shows
  
  (0 EET)
  (0 EET)

It looks as if hydra does not know the time zone "EET".  That is strange.

Re: icalendar-real-world test sometimes fails

[Glenn Morris]

Ulf Jasper wrote:

> It looks as if hydra does not know the time zone "EET".  That is strange.

You can use a full TZ specification when testing to avoid relying on the
system TZ database - see my recent and similar earlier changes.

Re: icalendar-real-world test sometimes fails

[Ulf Jasper]

Glenn Morris <rgm@gnu.org> writes:

> Ulf Jasper wrote:
>
>> It looks as if hydra does not know the time zone "EET".  That is strange.
>
> You can use a full TZ specification when testing to avoid relying on the
> system TZ database - see my recent and similar earlier changes.

I see that you have fixed it already.  Thank you.

Re: icalendar-real-world test sometimes fails

[Glenn Morris]

I finally was able to make the test logs available for download, see eg
http://hydra.nixos.org/build/13071520 .
(This is only for successful builds, for failing tests the log will still
be in the standard logfile.)

Re: icalendar-real-world test sometimes fails

[Ulf Jasper]

Glenn Morris <rgm@gnu.org> writes:

> I finally was able to make the test logs available for download, see eg
> http://hydra.nixos.org/build/13071520 .
> (This is only for successful builds, for failing tests the log will still
> be in the standard logfile.)

Very good!  I've updated my script.

Thanks.