bug#29187: flymake ruby and perl tests fail on rhel7.4

bug#29187: flymake ruby and perl tests fail on rhel7.4

[Glenn Morris]

Package: emacs
Version: 26.0.90

Hi,

The flymake tests added in 58bb346 for Ruby and Perl fail on RHEL7.4.
I guess it's again due to different versions of an external tool
reporting errors/warnings differently.

perl version: v5.16.3
ruby version: 2.0.0p648

Test perl-backend condition:
    (ert-test-failed
     ((should
       (eq 'flymake-error
           (face-at-point)))
      :form
      (eq flymake-error flymake-warning)
      :value nil))
   FAILED  4/8  perl-backend

Test ruby-backend condition:
    (ert-test-failed
     ((should
       (eq 'flymake-warning
           (face-at-point)))
      :form
      (eq flymake-warning nil)
      :value nil))
   FAILED  6/8  ruby-backend

bug#29187: flymake ruby and perl tests fail on rhel7.4

[Dmitry Gutov]

On 11/7/17 1:44 AM, Glenn Morris wrote:

> Test ruby-backend condition:
>      (ert-test-failed
>       ((should
>         (eq 'flymake-warning
>             (face-at-point)))
>        :form
>        (eq flymake-warning nil)
>        :value nil))
>     FAILED  6/8  ruby-backend

It also fails here via 'make check' (with Ruby 1.9.3) but succeeds 
interactively. So maybe the cause is something different.

bug#29187: flymake ruby and perl tests fail on rhel7.4

[João Távora]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 11/7/17 1:44 AM, Glenn Morris wrote:
>
>> Test ruby-backend condition:
>>      (ert-test-failed
>>       ((should
>>         (eq 'flymake-warning
>>             (face-at-point)))
>>        :form
>>        (eq flymake-warning nil)
>>        :value nil))
>>     FAILED  6/8  ruby-backend
>
> It also fails here via 'make check' (with Ruby 1.9.3) but succeeds
> interactively. So maybe the cause is something different.

I have ruby 2.3 and debian buster/sid (testing). Tests work both
interactively and non-interactively.

Dmitry does that happen with all the tests?  The difference between
interactive and non-interactive shows that you may be experiencing some
variation on the strange behaviour of https://debbugs.gnu.org/17647,
which I try to work around in flymake tests.

Anyway, Glenn and Dmitry can you please show the output of

  ruby -wc test/lisp/progmodes/flymake-resources/test.rb

and

  perl -wc test/lisp/progmodes/flymake-resources/test.pl

on RHEL7.4? I get, respectively

  test/lisp/progmodes/flymake-resources/test.rb:3: warning: statement not reached
  test/lisp/progmodes/flymake-resources/test.rb:5: syntax error, unexpected end-of-input, expecting keyword_end

and

  Unquoted string "unknown" may clash with future reserved word at test/lisp/progmodes/flymake-resources/test.pl line 2.
  syntax error at test/lisp/progmodes/flymake-resources/test.pl line 4, at EOF
  Missing right curly or square bracket at test/lisp/progmodes/flymake-resources/test.pl line 4, at end of line
  test/lisp/progmodes/flymake-resources/test.pl had compilation errors.

bug#29187: flymake ruby and perl tests fail on rhel7.4

[Glenn Morris]

João Távora wrote:

> Dmitry Gutov <dgutov@yandex.ru> writes:
[...]
>> It also fails here via 'make check' (with Ruby 1.9.3) but succeeds
>> interactively.

Same here.

> Anyway, Glenn and Dmitry can you please show the output of
>
>   ruby -wc test/lisp/progmodes/flymake-resources/test.rb
>
> and
>
>   perl -wc test/lisp/progmodes/flymake-resources/test.pl
>
> on RHEL7.4? I get, respectively
>
>   test/lisp/progmodes/flymake-resources/test.rb:3: warning: statement
> not reached
>   test/lisp/progmodes/flymake-resources/test.rb:5: syntax error,
> unexpected end-of-input, expecting keyword_end

I get the same.

>   Unquoted string "unknown" may clash with future reserved word at
> test/lisp/progmodes/flymake-resources/test.pl line 2.
>   syntax error at test/lisp/progmodes/flymake-resources/test.pl line 4, at EOF
>   Missing right curly or square bracket at
> test/lisp/progmodes/flymake-resources/test.pl line 4, at end of line
>   test/lisp/progmodes/flymake-resources/test.pl had compilation errors.

I get an extra line:

Unquoted string "unknown" may clash with future reserved word at test/lisp/progmodes/flymake-resources/test.pl line 2.
Scalar value @arr[1] better written as $arr[1] at test/lisp/progmodes/flymake-resources/test.pl line 3.
syntax error at test/lisp/progmodes/flymake-resources/test.pl line 4, at EOF
Missing right curly or square bracket at test/lisp/progmodes/flymake-resources/test.pl line 4, at end of line
test/lisp/progmodes/flymake-resources/test.pl had compilation errors.

bug#29187: flymake ruby and perl tests fail on rhel7.4

[João Távora]

Glenn Morris <rgm@gnu.org> writes:

> João Távora wrote:
>
>> Dmitry Gutov <dgutov@yandex.ru> writes:
> [...]
>>> It also fails here via 'make check' (with Ruby 1.9.3) but succeeds
>>> interactively.
>
> Same here.

This is bizarre, because I did manage to reproduce and fix, on a
separate Ubuntu machine I have access to, one of the errors that you
had, Glenn, and that was indeed due to the extra diagnostic line
produced by your perl.

But I don't get any differences between interactive and non-interactive
cases and I can't understand how your test run could possibly have
succeeded interactively since it produces that same extra line.

Dmitry, your case baffles me even more. Are you sure you get the exact
same output as Glenn?

Perhaps there's something strange and we might be dealing with different
problems. Let's perhaps first agree to run tests in a common
manner. Interactively:

   $ src/emacs -Q --eval "(setq load-prefer-newer t)" -l test/lisp/progmodes/flymake-tests.el --eval "(ert \"\")"

and non-interactively (make check also works, obviuosly)

   $ src/emacs -Q --eval "(setq load-prefer-newer t)" -l test/lisp/progmodes/flymake-tests.el --batch --eval "(ert-run-tests-batch-and-exit \"\")"

I get the same results in either case, no failures.

Now Glenn, can you try those 2 tests again after applying this patch to
flymake-tests.el? The first part should theoretically fix your perl
failure and the second part should help me understand if launching ruby
failed.

diff --git a/test/lisp/progmodes/flymake-tests.el b/test/lisp/progmodes/flymake-tests.el
index c60f910034..98c74624b5 100644
--- a/test/lisp/progmodes/flymake-tests.el
+++ b/test/lisp/progmodes/flymake-tests.el
@@ -114,13 +114,16 @@ flymake-tests--wait-for-backends
   (flymake-tests--with-flymake ("test.pl")
     (flymake-goto-next-error)
     (should (eq 'flymake-warning (face-at-point)))
-    (flymake-goto-next-error)
+    (goto-char (point-max))
+    (flymake-goto-prev-error)
     (should (eq 'flymake-error (face-at-point)))))
 
 (ert-deftest ruby-backend ()
   "Test the ruby backend"
   (skip-unless (executable-find "ruby"))
   (flymake-tests--with-flymake ("test.rb")
+    (if (memq 'ruby-flymake (flymake-disabled-backends))
+        (ert-fail "Something went wrong setting up ruby flymake"))
     (flymake-goto-next-error)
     (should (eq 'flymake-warning (face-at-point)))
     (flymake-goto-next-error)

bug#29187: flymake ruby and perl tests fail on rhel7.4

[Dmitry Gutov]

On 11/7/17 5:57 PM, João Távora wrote:

> Dmitry, your case baffles me even more. Are you sure you get the exact
> same output as Glenn?

No, my Perl output is the same as yours. I quoted your message in my 
reply. And my Ruby output differs from you both.

Anyway, never mind, please. My problem with the Ruby backend test was 
due to local modification (using the Rubocop backend unconditionally, 
which I'm testing out).

I'm not exactly sure about how it worked "fine interactively" anymore; 
probably in an pre-existing Emacs session. So I'm bowing out, sorry.

bug#29187: flymake ruby and perl tests fail on rhel7.4

[Glenn Morris]

João Távora wrote:

> This is bizarre, because I did manage to reproduce and fix, on a
> separate Ubuntu machine I have access to, one of the errors that you
> had, Glenn, and that was indeed due to the extra diagnostic line
> produced by your perl.
>
> But I don't get any differences between interactive and non-interactive
> cases and I can't understand how your test run could possibly have
> succeeded interactively since it produces that same extra line.

There's no difference between the interactive and non-interactive case
for me with the Perl test (I don't think I said there was?).

> and non-interactively (make check also works, obviuosly)
>
>    $ src/emacs -Q --eval "(setq load-prefer-newer t)" -l test/lisp/progmodes/flymake-tests.el --batch --eval "(ert-run-tests-batch-and-exit \"\")"

This is not equivalent to make check, which sets HOME=/nonexistent.
Indeed that is the cause of the issue for me:

HOME=/nonexistent ruby -wc test/lisp/progmodes/flymake-resources/test.rb 

fails with a ruby error:

/usr/share/rubygems/rubygems/path_support.rb:68:in `path=': undefined
method `+' for nil:NilClass (NoMethodError)
       from /usr/share/rubygems/rubygems/path_support.rb:30:in `initialize'

I don't use ruby and have no idea if this is normal.

> Now Glenn, can you try those 2 tests again after applying this patch to
> flymake-tests.el? The first part should theoretically fix your perl
> failure 

It does, thanks.

bug#29187: flymake ruby and perl tests fail on rhel7.4

[Glenn Morris]

Glenn Morris wrote:

> HOME=/nonexistent ruby -wc test/lisp/progmodes/flymake-resources/test.rb 
>
> fails with a ruby error:
>
> /usr/share/rubygems/rubygems/path_support.rb:68:in `path=': undefined
> method `+' for nil:NilClass (NoMethodError)
>        from /usr/share/rubygems/rubygems/path_support.rb:30:in `initialize'
>
> I don't use ruby and have no idea if this is normal.

Perhaps this is relevant: https://bugs.ruby-lang.org/issues/12695

bug#29187: flymake ruby and perl tests fail on rhel7.4

[João Távora]

Glenn Morris <rgm@gnu.org> writes:

> João Távora wrote:
>
>> This is bizarre, because I did manage to reproduce and fix, on a
>> separate Ubuntu machine I have access to, one of the errors that you
>> had, Glenn, and that was indeed due to the extra diagnostic line
>> produced by your perl.
>>
>> But I don't get any differences between interactive and non-interactive
>> cases and I can't understand how your test run could possibly have
>> succeeded interactively since it produces that same extra line.
>
> There's no difference between the interactive and non-interactive case
> for me with the Perl test (I don't think I said there was?).

Well, you did write "same here" after Dmitry said there was a
difference.

>> Dmitry Gutov <dgutov@yandex.ru> writes:
>[...]
>>> It also fails here via 'make check' (with Ruby 1.9.3) but succeeds
>>> interactively.
>Same here.
 ^^^^^^^^^

>> and non-interactively (make check also works, obviuosly)
>>
>>    $ src/emacs -Q --eval "(setq load-prefer-newer t)" -l test/lisp/progmodes/flymake-tests.el --batch --eval "(ert-run-tests-batch-and-exit \"\")"
>
> This is not equivalent to make check, which sets HOME=/nonexistent.
> Indeed that is the cause of the issue for me:
>
> HOME=/nonexistent ruby -wc test/lisp/progmodes/flymake-resources/test.rb 
>
> fails with a ruby error:
>
> /usr/share/rubygems/rubygems/path_support.rb:68:in `path=': undefined
> method `+' for nil:NilClass (NoMethodError)
>        from /usr/share/rubygems/rubygems/path_support.rb:30:in `initialize'
>
> I don't use ruby and have no idea if this is normal.

Wow, subtle stuff. My ruby (2.3) doesn't crash. Does yours also fail if
you use a absolute path after -wc instead of a relative one?
>
> It does, thanks.

So I guess all that's missing is a fix to the ruby case. If the absolute
path doesn't fix it, I think we can fake a dummy HOME

bug#29187: flymake ruby and perl tests fail on rhel7.4

[Glenn Morris]

João Távora wrote:

> Wow, subtle stuff. My ruby (2.3) doesn't crash. Does yours also fail if
> you use a absolute path after -wc instead of a relative one?

Still fails.

> So I guess all that's missing is a fix to the ruby case. If the absolute
> path doesn't fix it, I think we can fake a dummy HOME

This works for me:

--- i/test/lisp/progmodes/flymake-tests.el
+++ w/test/lisp/progmodes/flymake-tests.el
@@ -120,11 +120,17 @@ flymake-tests--wait-for-backends
 (ert-deftest ruby-backend ()
   "Test the ruby backend"
   (skip-unless (executable-find "ruby"))
-  (flymake-tests--with-flymake ("test.rb")
-    (flymake-goto-next-error)
-    (should (eq 'flymake-warning (face-at-point)))
-    (flymake-goto-next-error)
-    (should (eq 'flymake-error (face-at-point)))))
+  ;; Some versions of ruby fail if HOME doesn't exist (bug#29187).
+  (let* ((tempdir (make-temp-file "flymake-tests-ruby" t))
+         (process-environment (cons (format "HOME=%s" tempdir)
+                                    process-environment)))
+    (unwind-protect
+        (flymake-tests--with-flymake ("test.rb")
+          (flymake-goto-next-error)
+          (should (eq 'flymake-warning (face-at-point)))
+          (flymake-goto-next-error)
+          (should (eq 'flymake-error (face-at-point))))
+      (delete-directory tempdir t))))
 
 (ert-deftest different-diagnostic-types ()
   "Test GCC warning via function predicate."

bug#29187: flymake ruby and perl tests fail on rhel7.4

[João Távora]

Glenn Morris <rgm@gnu.org> writes:

> João Távora wrote:
>
>> Wow, subtle stuff. My ruby (2.3) doesn't crash. Does yours also fail if
>> you use a absolute path after -wc instead of a relative one?
>
> Still fails.
>
>> So I guess all that's missing is a fix to the ruby case. If the absolute
>> path doesn't fix it, I think we can fake a dummy HOME
>
> This works for me:
>

Thanks. Hmmm, trying to decide if this should be done just for the tests
or in the ruby backend in general. I lean towards your solution, since
this is appars to be a ruby bug anyway. WDYT?

>
> --- i/test/lisp/progmodes/flymake-tests.el
> +++ w/test/lisp/progmodes/flymake-tests.el
> @@ -120,11 +120,17 @@ flymake-tests--wait-for-backends
>  (ert-deftest ruby-backend ()
>    "Test the ruby backend"
>    (skip-unless (executable-find "ruby"))
> -  (flymake-tests--with-flymake ("test.rb")
> -    (flymake-goto-next-error)
> -    (should (eq 'flymake-warning (face-at-point)))
> -    (flymake-goto-next-error)
> -    (should (eq 'flymake-error (face-at-point)))))
> +  ;; Some versions of ruby fail if HOME doesn't exist (bug#29187).
> +  (let* ((tempdir (make-temp-file "flymake-tests-ruby" t))
> +         (process-environment (cons (format "HOME=%s" tempdir)
> +                                    process-environment)))
> +    (unwind-protect
> +        (flymake-tests--with-flymake ("test.rb")
> +          (flymake-goto-next-error)
> +          (should (eq 'flymake-warning (face-at-point)))
> +          (flymake-goto-next-error)
> +          (should (eq 'flymake-error (face-at-point))))
> +      (delete-directory tempdir t))))
>  
>  (ert-deftest different-diagnostic-types ()
>    "Test GCC warning via function predicate."

bug#29187: flymake ruby and perl tests fail on rhel7.4

[Glenn Morris]

João Távora wrote:

> Thanks. Hmmm, trying to decide if this should be done just for the tests
> or in the ruby backend in general. I lean towards your solution, since
> this is appars to be a ruby bug anyway. WDYT?

I'd say just for ruby.

bug#29187: flymake ruby and perl tests fail on rhel7.4

[João Távora]

Glenn Morris <rgm@gnu.org> writes:

> João Távora wrote:
>
>> Thanks. Hmmm, trying to decide if this should be done just for the tests
>> or in the ruby backend in general. I lean towards your solution, since
>> this is appars to be a ruby bug anyway. WDYT?
>
> I'd say just for ruby.

I think you meant "just for the ruby tests", since that's what your
patch contained, and that's what I just commited in
0da08f2f8ef1946978f0974e9e8cdb87fece018c, which also contains the perl
fix.

João

bug#29187: flymake ruby and perl tests fail on rhel7.4

[Dmitry Gutov]

On 11/7/17 2:45 AM, João Távora wrote:

> Dmitry does that happen with all the tests?

Only the "ruby backend" test. And the rest of 'make check' looks like this:

Ran 2576 tests, 2552 results as expected, 1 unexpected, 23 skipped

> The difference between
> interactive and non-interactive shows that you may be experiencing some
> variation on the strange behaviour of https://debbugs.gnu.org/17647,
> which I try to work around in flymake tests.

Is there any way I can check?

> Anyway, Glenn and Dmitry can you please show the output of
> 
>    ruby -wc test/lisp/progmodes/flymake-resources/test.rb
> 
> and
> 
>    perl -wc test/lisp/progmodes/flymake-resources/test.pl
> 
> on RHEL7.4? I get, respectively
> 
>    test/lisp/progmodes/flymake-resources/test.rb:3: warning: statement not reached
>    test/lisp/progmodes/flymake-resources/test.rb:5: syntax error, unexpected end-of-input, expecting keyword_end

Almost the same:

test/lisp/progmodes/flymake-resources/test.rb:2: warning: statement not 
reached
test/lisp/progmodes/flymake-resources/test.rb:5: syntax error, 
unexpected $end, expecting keyword_end

> and
> 
>    Unquoted string "unknown" may clash with future reserved word at test/lisp/progmodes/flymake-resources/test.pl line 2.
>    syntax error at test/lisp/progmodes/flymake-resources/test.pl line 4, at EOF
>    Missing right curly or square bracket at test/lisp/progmodes/flymake-resources/test.pl line 4, at end of line
>    test/lisp/progmodes/flymake-resources/test.pl had compilation errors.

Exactly the same here.