bug#32764: Somne flymake tests seem unstable

bug#32764: Somne flymake tests seem unstable

[Glenn Morris]

Package: emacs
Version: 27.0.50
Severity: minor

Ref: https://hydra.nixos.org/build/81615914

I've seen some flymake test failures like the following a few times.
Perhaps some timeout needs to be increased.

    Test included-c-header-files condition:
        (ert-test-failed "Some backends not reporting yet (flymake-cc)")
       FAILED  4/9  included-c-header-files (10.545006 sec)

    Test perl-backend condition:
        (error "Selecting deleted buffer")
       FAILED  5/9  perl-backend (0.289559 sec)

    Test recurrent-backend condition:
        (error "Selecting deleted buffer")
       FAILED  6/9  recurrent-backend (0.005780 sec)

    Test warning-predicate-function-gcc condition:
        (ert-test-failed "Some backends not reporting yet (flymake-cc)")
       FAILED  8/9  warning-predicate-function-gcc (5.444343 sec)

bug#32764: Somne flymake tests seem unstable

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 1556 bytes --]

Hi Glenn,

Thanks.

Increasing timeouts, in my experience, rarely solves these problems (still,
if you suspect one in particular, feel free to adjust it).  It's usually a
case of unusual process scheduling in the ci machine that my code isn't
accounting for.  It'd be better to analyze the code to figure out what
might be happening (Something that I haven't done yet...) Also IIRC there's
a reference  in the test code to a mysterious existing bug somewhere...

Maybe someone with deeper knowledge of Emacs's processes can take a look
too.

Thanks again,
João

On Tue, Sep 18, 2018, 19:19 Glenn Morris <rgm@gnu.org> wrote:

> Package: emacs
> Version: 27.0.50
> Severity: minor
>
> Ref: https://hydra.nixos.org/build/81615914
>
> I've seen some flymake test failures like the following a few times.
> Perhaps some timeout needs to be increased.
>
>     Test included-c-header-files condition:
>         (ert-test-failed "Some backends not reporting yet (flymake-cc)")
>        FAILED  4/9  included-c-header-files (10.545006 sec)
>
>     Test perl-backend condition:
>         (error "Selecting deleted buffer")
>        FAILED  5/9  perl-backend (0.289559 sec)
>
>     Test recurrent-backend condition:
>         (error "Selecting deleted buffer")
>        FAILED  6/9  recurrent-backend (0.005780 sec)
>
>     Test warning-predicate-function-gcc condition:
>         (ert-test-failed "Some backends not reporting yet (flymake-cc)")
>        FAILED  8/9  warning-predicate-function-gcc (5.444343 sec)
>
>
>

[-- Attachment #2: Type: text/html, Size: 2298 bytes --]

bug#32764: Some flymake tests seem unstable

[Paul Eggert]

[-- Attachment #1: Type: text/plain, Size: 168 bytes --]

For what it's worth, the flymake tests typically fail for me on master. I 
normally test on Fedora 28 x86-64 on a circa-2010 AMD Phenom II X4 910e. Log 
file attached.

[-- Attachment #2: flymake-tests.log --]
[-- Type: text/x-log, Size: 1947 bytes --]

Running 9 tests (2018-09-19 21:06:16-0700, selector `(not (or (tag :expensive-test) (tag :unstable)))')
Test different-diagnostic-types backtrace:
  signal(ert-test-failed (((should (eq 'flymake-error (face-at-point))
  ert-fail(((should (eq 'flymake-error (face-at-point))) :form (eq fly
  #f(compiled-function () #<bytecode 0x513fe9>)()
  flymake-tests--call-with-fixture(#f(compiled-function () #<bytecode 
  #f(compiled-function () #<bytecode 0x5143d1>)()
  ert--run-test-internal(#s(ert--test-execution-info :test #s(ert-test
  ert-run-test(#s(ert-test :name different-diagnostic-types :documenta
  ert-run-or-rerun-test(#s(ert--stats :selector (not (or ... ...)) :te
  ert-run-tests((not (or (tag :expensive-test) (tag :unstable))) #f(co
  ert-run-tests-batch((not (or (tag :expensive-test) (tag :unstable)))
  ert-run-tests-batch-and-exit((not (or (tag :expensive-test) (tag :un
  eval((ert-run-tests-batch-and-exit '(not (or (tag :expensive-test) (
  command-line-1(("-L" ":." "-l" "ert" "-l" "lisp/progmodes/flymake-te
  command-line()
  normal-top-level()
Test different-diagnostic-types condition:
    (ert-test-failed
     ((should
       (eq 'flymake-error
	    (face-at-point)))
      :form
      (eq flymake-error flymake-note)
      :value nil))
   FAILED  1/9  different-diagnostic-types (1.170822 sec)
   passed  2/9  dummy-backends (1.003490 sec)
   passed  3/9  eob-region-and-trailing-newline (0.000554 sec)
   passed  4/9  included-c-header-files (2.058290 sec)
   passed  5/9  perl-backend (1.037708 sec)
   passed  6/9  recurrent-backend (1.002835 sec)
  skipped  7/9  ruby-backend (0.000520 sec)
   passed  8/9  warning-predicate-function-gcc (1.027205 sec)
   passed  9/9  warning-predicate-rx-gcc (1.044844 sec)

Ran 9 tests, 7 results as expected, 1 unexpected, 1 skipped (2018-09-19 21:06:25-0700, 8.515835 sec)

1 unexpected results:
   FAILED  different-diagnostic-types

1 skipped results:
  SKIPPED  ruby-backend

bug#32764: Some flymake tests seem unstable

[João Távora]

Paul Eggert <eggert@cs.ucla.edu> writes:

> For what it's worth, the flymake tests typically fail for me on master. I 
> normally test on Fedora 28 x86-64 on a circa-2010 AMD Phenom II X4 910e. Log 
> file attached.
>
>       :form
>       (eq flymake-error flymake-note)
>       :value nil))


Hi Paul, this test failure is compatible with a problem with GCC 8.2
that I recently fixed in 04d3315271cd1357fbbc192505d00c760be02952.  Can
you check if your compiler's version?  If it doesn't report the column
for "In file included from" errors, it would have triggered the failure
before my fix.

João

bug#32764: Some flymake tests seem unstable

[Paul Eggert]

João Távora wrote:

> Hi Paul, this test failure is compatible with a problem with GCC 8.2
> that I recently fixed in 04d3315271cd1357fbbc192505d00c760be02952.  Can
> you check if your compiler's version?  If it doesn't report the column
> for "In file included from" errors, it would have triggered the failure
> before my fix.

It's gcc (GCC) 8.2.1 20181215 (Red Hat 8.2.1-6).

I'm not observing the error now; thanks.

bug#32764: Some flymake tests seem unstable

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 576 bytes --]

I guess we can close this bug, then.

João


On Sun, Jan 6, 2019, 01:20 Paul Eggert <eggert@cs.ucla.edu wrote:

> João Távora wrote:
>
> > Hi Paul, this test failure is compatible with a problem with GCC 8.2
> > that I recently fixed in 04d3315271cd1357fbbc192505d00c760be02952.  Can
> > you check if your compiler's version?  If it doesn't report the column
> > for "In file included from" errors, it would have triggered the failure
> > before my fix.
>
> It's gcc (GCC) 8.2.1 20181215 (Red Hat 8.2.1-6).
>
> I'm not observing the error now; thanks.
>

[-- Attachment #2: Type: text/html, Size: 932 bytes --]

bug#32764: Some flymake tests seem unstable

[Glenn Morris]

João Távora wrote:

> I guess we can close this bug, then.

I don't object, but for the record Paul's issue has nothing to do with
the original report.

bug#32764: Some flymake tests seem unstable

[João Távora]

You're right. I read the Paul's "typically" as the "unstable" bit.

You can reopen the bug.

How often would you say this happens on hydra, Glenn?

João


On Sun, Jan 6, 2019 at 7:20 PM Glenn Morris <rgm@gnu.org> wrote:
>
> João Távora wrote:
>
> > I guess we can close this bug, then.
>
> I don't object, but for the record Paul's issue has nothing to do with
> the original report.



-- 
João Távora

bug#32764: Some flymake tests seem unstable

[Glenn Morris]

João Távora wrote:

> How often would you say this happens on hydra, Glenn?

From paging through

https://hydra.nixos.org/jobset/gnu/emacs-trunk/evals

the last time was Dec 26

https://hydra.nixos.org/build/86281444

I had hoped 7ca9bb7 would avoid this, but apparently not.
If it means anything, all the errors were "Selecting deleted buffer"
that last time.