filenotify-tests.el failing inside Docker debian:stretch

filenotify-tests.el failing inside Docker debian:stretch

[Ted Zlatanov]

The only Emacs tests that currently fail inside Docker (using
debian:stretch image) are as follows. Is that expected because of Docker
specifics or is it something else?

Thanks
Ted

Running 8 tests (2018-12-31 15:28:15+0000, selector `(not (or (tag :expensive-test) (tag :unstable)))')
Library: `inotify'
   passed  1/8  file-notify-test00-availability (0.001403 sec)
   passed  2/8  file-notify-test01-add-watch (0.001401 sec)
   passed  3/8  file-notify-test02-rm-watch (0.024561 sec)
Test file-notify-test03-events backtrace:
  signal(ert-test-failed (((should (file-notify--test-with-events-chec
  ert-fail(((should (file-notify--test-with-events-check events)) :for
  #f(compiled-function () #<bytecode 0xbae5b9>)()
  ert--run-test-internal(#s(ert--test-execution-info :test #s(ert-test
  ert-run-test(#s(ert-test :name file-notify-test03-events :documentat
  ert-run-or-rerun-test(#s(ert--stats :selector (not (or ... ...)) :te
  ert-run-tests((not (or (tag :expensive-test) (tag :unstable))) #f(co
  ert-run-tests-batch((not (or (tag :expensive-test) (tag :unstable)))
  ert-run-tests-batch-and-exit((not (or (tag :expensive-test) (tag :un
  eval((ert-run-tests-batch-and-exit '(not (or (tag :expensive-test) (
  command-line-1(("-L" ":." "-l" "ert" "-l" "lisp/filenotify-tests" "-
  command-line()
  normal-top-level()
Test file-notify-test03-events condition:
    (ert-test-failed
     ((should
       (file-notify--test-with-events-check events))
      :form
      (file-notify--test-with-events-check
       ((created changed deleted deleted stopped)))
      :value nil :explanation "Received events do not match expected events
(created changed deleted)
(created changed deleted deleted stopped)"))
   FAILED  4/8  file-notify-test03-events (6.078224 sec)
Reverting buffer `file-notify-testac92zf'.
Reverting buffer `file-notify-testac92zf'.
   passed  5/8  file-notify-test04-autorevert (6.008533 sec)
Test file-notify-test05-file-validity backtrace:
  signal(ert-test-failed (((should (file-notify--test-with-events-chec
  ert-fail(((should (file-notify--test-with-events-check events)) :for
  #f(compiled-function () #<bytecode 0xbb16fd>)()
  ert--run-test-internal(#s(ert--test-execution-info :test #s(ert-test
  ert-run-test(#s(ert-test :name file-notify-test05-file-validity :doc
  ert-run-or-rerun-test(#s(ert--stats :selector (not (or ... ...)) :te
  ert-run-tests((not (or (tag :expensive-test) (tag :unstable))) #f(co
  ert-run-tests-batch((not (or (tag :expensive-test) (tag :unstable)))
  ert-run-tests-batch-and-exit((not (or (tag :expensive-test) (tag :un
  eval((ert-run-tests-batch-and-exit '(not (or (tag :expensive-test) (
  command-line-1(("-L" ":." "-l" "ert" "-l" "lisp/filenotify-tests" "-
  command-line()
  normal-top-level()
Test file-notify-test05-file-validity condition:
    (ert-test-failed
     ((should
       (file-notify--test-with-events-check events))
      :form
      (file-notify--test-with-events-check
       ((created changed deleted deleted stopped)))
      :value nil :explanation "Received events do not match expected events
(created changed deleted)
(created changed deleted deleted stopped)"))
   FAILED  6/8  file-notify-test05-file-validity (6.045527 sec)
Test file-notify-test06-dir-validity backtrace:
  signal(ert-test-failed (((should-not (file-notify-valid-p file-notif
  ert-fail(((should-not (file-notify-valid-p file-notify--test-desc)) 
  #f(compiled-function () #<bytecode 0xbb1af5>)()
  ert--run-test-internal(#s(ert--test-execution-info :test #s(ert-test
  ert-run-test(#s(ert-test :name file-notify-test06-dir-validity :docu
  ert-run-or-rerun-test(#s(ert--stats :selector (not (or ... ...)) :te
  ert-run-tests((not (or (tag :expensive-test) (tag :unstable))) #f(co
  ert-run-tests-batch((not (or (tag :expensive-test) (tag :unstable)))
  ert-run-tests-batch-and-exit((not (or (tag :expensive-test) (tag :un
  eval((ert-run-tests-batch-and-exit '(not (or (tag :expensive-test) (
  command-line-1(("-L" ":." "-l" "ert" "-l" "lisp/filenotify-tests" "-
  command-line()
  normal-top-level()
Test file-notify-test06-dir-validity condition:
    (ert-test-failed
     ((should-not
       (file-notify-valid-p file-notify--test-desc))
      :form
      (file-notify-valid-p
       (1 . 0))
      :value t))
   FAILED  7/8  file-notify-test06-dir-validity (3.001878 sec)
   passed  8/8  file-notify-test08-backup (3.058565 sec)

Ran 8 tests, 5 results as expected, 3 unexpected (2018-12-31 15:28:41+0000, 25.120320 sec)

3 unexpected results:
   FAILED  file-notify-test03-events
   FAILED  file-notify-test05-file-validity
   FAILED  file-notify-test06-dir-validity

Re: filenotify-tests.el failing inside Docker debian:stretch

[Michael Albinus]

Ted Zlatanov <tzz@lifelogs.com> writes:

Hi Ted,

> The only Emacs tests that currently fail inside Docker (using
> debian:stretch image) are as follows. Is that expected because of Docker
> specifics or is it something else?

It is not expected. Perhaps, you shall write a bug report. It would be
great, if you describe also how it could be reproduced.

> Thanks
> Ted

Best regards, Michael.

Re: filenotify-tests.el failing inside Docker debian:stretch

[Ted Zlatanov]

On Mon, 31 Dec 2018 18:27:23 +0100 Michael Albinus <michael.albinus@gmx.de> wrote: 

> Ted wrote:
>> The only Emacs tests that currently fail inside Docker (using
>> debian:stretch image) are as follows. Is that expected because of Docker
>> specifics or is it something else?

MA> It is not expected. Perhaps, you shall write a bug report. It would be
MA> great, if you describe also how it could be reproduced.

I simply ran `make check' inside the Docker image. The only
customization was to allow `make bootstrap' to work on the host machine:

echo 0 > /proc/sys/kernel/randomize_va_space
docker run -it --rm -v /my/emacs/checkout:/src debian:stretch bash

...and inside the container...

apt update -qq
DEBIAN_FRONTEND=noninteractive apt install --no-install-recommends -y -qq -o=Dpkg::Use-Pty=0 libc-dev gcc make autoconf automake libncurses-dev gnutls-dev git
cd /src
make check

This is just what EMBA does. But it seems like a Docker issue rather
than something about the specific EMBA setup. And since Docker is not a
primary virtualization platform supported by Emacs, maybe it's better to
just skip these tests? Your call... I can create the bug report if you
prefer.

Thanks
Ted

Re: filenotify-tests.el failing inside Docker debian:stretch

[Ted Zlatanov]

On Mon, 31 Dec 2018 18:48:28 +0000 Ted Zlatanov <tzz@lifelogs.com> wrote: 

TZ> echo 0 > /proc/sys/kernel/randomize_va_space
...

One-liner version with configure, to be run after the kernel setting above:

docker run -it --rm -v /root/emacs-clone-tmp:/src debian:stretch bash -c "apt update -qq && DEBIAN_FRONTEND=noninteractive apt install --no-install-recommends -y -qq -o=Dpkg::Use-Pty=0 libc-dev gcc make autoconf automake libncurses-dev gnutls-dev git && cd /src && ./autogen.sh autoconf && ./configure --without-makeinfo && make check"

It's inefficient but if you just run bash you should be able to run all
of the commands one by one.

Ted

Re: filenotify-tests.el failing inside Docker debian:stretch

[Michael Albinus]

Ted Zlatanov <tzz@lifelogs.com> writes:

> I simply ran `make check' inside the Docker image.

Since these are automated tests, they could be more expensive. On hydra,
it is

make check-expensive EMACS_HYDRA_CI=1 TEST_BACKTRACE_LINE_LENGTH=150 TEST_LOAD_EL=no SUMMARIZE_TESTS=25

> Thanks
> Ted

Best regards, Michael.

Re: filenotify-tests.el failing inside Docker debian:stretch

[Ted Zlatanov]

On Wed, 02 Jan 2019 09:36:53 +0100 Michael Albinus <michael.albinus@gmx.de> wrote: 

MA> Since these are automated tests, they could be more expensive. On hydra,
MA> it is

MA> make check-expensive EMACS_HYDRA_CI=1 TEST_BACKTRACE_LINE_LENGTH=150 TEST_LOAD_EL=no SUMMARIZE_TESTS=25

I'd start with full testing, this is a dedicated machine for running
tests and emacs.git doesn't have too many commits per day. We can
improve later as follows (all controlled via .gitlab-ci.yml):

* don't test scratch branches
* test master and release branches fully (expensive checks)
* test other branches quickly

Ted

Re: filenotify-tests.el failing inside Docker debian:stretch

[Michael Albinus]

Ted Zlatanov <tzz@lifelogs.com> writes:

Hi Ted,

> This is just what EMBA does. But it seems like a Docker issue rather
> than something about the specific EMBA setup. And since Docker is not a
> primary virtualization platform supported by Emacs, maybe it's better to
> just skip these tests? Your call... I can create the bug report if you
> prefer.

Looks like I fixed this. On emba (docker?), inotify does not generate a
delete event, when a file is watched, and the upper directory is
deleted. No idea why.

It is not the most prominent feature of file notifications, and we know
it from other monitor libraries as well (w32notify). So I've adapted the
tests, pushed to master.

I will continue to play with the scratch/albinus branch on emba, in
order to check other monitor libraries (gfilenotify, inotifywatch, gio).

> Thanks
> Ted

Best regards, Michael.

Re: filenotify-tests.el failing inside Docker debian:stretch

[Michael Albinus]

[-- Attachment #1: Type: text/plain, Size: 367 bytes --]

Michael Albinus <michael.albinus@gmx.de> writes:

Hi Ted,

> I will continue to play with the scratch/albinus branch on emba, in
> order to check other monitor libraries (gfilenotify, inotifywatch, gio).

Done. Needed changes are pushed to master.

Finally, I've ended up with the appended .gitlab-ci.yml file. Is it OK
to push it to master?

Best regards, Michael.


[-- Attachment #2: .gitlab-ci.yml --]
[-- Type: application/octet-stream, Size: 2238 bytes --]

# Copyright (C) 2017-2018 Free Software Foundation, Inc.
#
#  This file is part of GNU Emacs.
#
#  GNU Emacs is free software: you can redistribute it and/or modify
#  it under the terms of the GNU General Public License as published by
#  the Free Software Foundation, either version 3 of the License, or
#  (at your option) any later version.
#
#  GNU Emacs is distributed in the hope that it will be useful,
#  but WITHOUT ANY WARRANTY; without even the implied warranty of
#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#  GNU General Public License for more details.
#
#  You should have received a copy of the GNU General Public License
#  along with GNU Emacs.  If not, see <https://www.gnu.org/licenses/>.

# GNU Emacs support for the GitLab protocol for CI

# The presence of this file does not imply any FSF/GNU endorsement of
# any particular service that uses that protocol.  Also, it is intended for
# evaluation purposes, thus possibly temporary.

# Maintainer: tzz@lifelogs.com
# URL: https://emba.gnu.org/emacs/emacs

image: debian:stretch

variables:
  GIT_STRATEGY: fetch
  EMACS_EMBA_CI: 1

before_script:
  - apt update -qq
  - DEBIAN_FRONTEND=noninteractive apt install --no-install-recommends -y -qq -o=Dpkg::Use-Pty=0 libc-dev gcc make autoconf automake libncurses-dev gnutls-dev git

stages:
  - test

test-all:
  # This tests also file monitor libraries inotify and inotifywatch.
  stage: test
  script:
    - DEBIAN_FRONTEND=noninteractive apt install --no-install-recommends -y -qq -o=Dpkg::Use-Pty=0 inotify-tools
    - ./autogen.sh autoconf
    - ./configure --without-makeinfo
    - make bootstrap
    - make check-expensive

test-filenotify-gio:
  stage: test
  # This tests file monitor libraries gfilemonitor and gio.
  only:
    changes:
      - .gitlab-ci.yml
      - lisp/filenotify.el
      - lisp/net/tramp-sh.el
      - src/gfilenotify.c
      - test/lisp/filenotify-tests.el
  script:
    - DEBIAN_FRONTEND=noninteractive apt install --no-install-recommends -y -qq -o=Dpkg::Use-Pty=0 libglib2.0-dev libglib2.0-bin libglib2.0-0
    - ./autogen.sh autoconf
    - ./configure --without-makeinfo --with-file-notification=gfile
    - make bootstrap
    - make -C test filenotify-tests

Re: filenotify-tests.el failing inside Docker debian:stretch

[Ted Zlatanov]

On Mon, 07 Jan 2019 14:17:43 +0100 Michael Albinus <michael.albinus@gmx.de> wrote: 

MA> Done. Needed changes are pushed to master.

MA> Finally, I've ended up with the appended .gitlab-ci.yml file. Is it OK
MA> to push it to master?

Yes, absolutely. The change looks good.

After that merge, I will work on speeding up the build but I think once
we have passing builds we can open EMBA to public read-only access.

Thank you for working on this!
Ted

Re: filenotify-tests.el failing inside Docker debian:stretch

[Michael Albinus]

Ted Zlatanov <tzz@lifelogs.com> writes:

Hi Ted,

> MA> Finally, I've ended up with the appended .gitlab-ci.yml file. Is it OK
> MA> to push it to master?
>
> Yes, absolutely. The change looks good.

Done.

> After that merge, I will work on speeding up the build but I think once
> we have passing builds we can open EMBA to public read-only access.

One idea is to exclude *.texi changes from test-all runs. Something like
(untested)

except:
  changes:
    - doc/**/*

I have no idea what happens, if a lisp file and a texi file are touched
in the same commit.

> Thank you for working on this!
> Ted

Best regards, Michael.

Re: filenotify-tests.el failing inside Docker debian:stretch

[Ted Zlatanov]

On Tue, 08 Jan 2019 09:35:33 +0100 Michael Albinus <michael.albinus@gmx.de> wrote: 

MA> Ted Zlatanov <tzz@lifelogs.com> writes:
>> After that merge, I will work on speeding up the build but I think once
>> we have passing builds we can open EMBA to public read-only access.

MA> One idea is to exclude *.texi changes from test-all runs. Something like
MA> (untested)

MA> except:
MA>   changes:
MA>     - doc/**/*

MA> I have no idea what happens, if a lisp file and a texi file are touched
MA> in the same commit.

It would break :) The `only` logic may be better here. But the
optimization won't buy us too much, because there aren't enough commits
per hour in the Emacs repo to stress the build runners, and builds are
easy to run in parallel with the Docker runner.

I think the important optimization is to shrink throughput time (the
time to do a build from trigger to finish) by pre-building the Docker
image with the packages instead of installing them every time. Also
running make with higher parallelism might help, I don't know for sure.

Ted

Re: filenotify-tests.el failing inside Docker debian:stretch

[Michael Albinus]

Ted Zlatanov <tzz@lifelogs.com> writes:

Hi Ted,

> The only Emacs tests that currently fail inside Docker (using
> debian:stretch image) are as follows. Is that expected because of Docker
> specifics or is it something else?

I would like to instrument the tests. In order to not annoy everybody,
it would be great if there would be an indication, that a test runs on
emba.gnu.org. Would it be possible to set an environment variable there,
say $EMACS_EMBA_CI? Then I could wrap my instrumentation code inside

(when (getenv "EMACS_EMABA_CI")
  ...)

This is similar to the environment of hydra.nixos.org, where I could
test for $EMACS_HYDRA_CI.

> Thanks
> Ted

Best regards, Michael.

Re: filenotify-tests.el failing inside Docker debian:stretch

[Ted Zlatanov]

On Wed, 02 Jan 2019 09:31:37 +0100 Michael Albinus <michael.albinus@gmx.de> wrote: 

MA> I would like to instrument the tests. In order to not annoy everybody,
MA> it would be great if there would be an indication, that a test runs on
MA> emba.gnu.org. Would it be possible to set an environment variable there,
MA> say $EMACS_EMBA_CI? Then I could wrap my instrumentation code inside

MA> (when (getenv "EMACS_EMABA_CI")
MA>   ...)

MA> This is similar to the environment of hydra.nixos.org, where I could
MA> test for $EMACS_HYDRA_CI.

Of course, yes. This is simply added under `stage: test` in .gitlab-ci.yml:

  variables:
    IN_CI: 1

I would just make a branch with the test fix that also changes
.gitlab-ci.yml, and that should pass.

But if this is a Docker issue, should we disable these tests in Docker
for everyone?

Ted

Re: filenotify-tests.el failing inside Docker debian:stretch

[Michael Albinus]

Ted Zlatanov <tzz@lifelogs.com> writes:

Hi Ted,

> MA> This is similar to the environment of hydra.nixos.org, where I could
> MA> test for $EMACS_HYDRA_CI.
>
> Of course, yes. This is simply added under `stage: test` in .gitlab-ci.yml:
>
>   variables:
>     IN_CI: 1
>
> I would just make a branch with the test fix that also changes
> .gitlab-ci.yml, and that should pass.

Yes, but pls use the variable name EMACS_EMBA_CI

> But if this is a Docker issue, should we disable these tests in Docker
> for everyone?

I don't know whether this is a docker issue. First, I need some tests.

> Ted

Best regards, Michael.

Re: filenotify-tests.el failing inside Docker debian:stretch

[Ted Zlatanov]

On Wed, 02 Jan 2019 15:31:18 +0100 Michael Albinus <michael.albinus@gmx.de> wrote: 

>> Of course, yes. This is simply added under `stage: test` in .gitlab-ci.yml:
>> 
>> variables:
>> IN_CI: 1
>> 
>> I would just make a branch with the test fix that also changes
>> .gitlab-ci.yml, and that should pass.

MA> Yes, but pls use the variable name EMACS_EMBA_CI

OK. In your branch change

test:
  stage: test

to

test:
  stage: test
  variables:
    EMACS_EMBA_CI: 1

and then add the code which knows about EMACS_EMBA_CI. If you want me to
do it, tell me what tests to condition on that environment var.

Thanks
Ted

Re: filenotify-tests.el failing inside Docker debian:stretch

[Michael Albinus]

Ted Zlatanov <tzz@lifelogs.com> writes:

> OK. In your branch change
>
> test:
>   stage: test
>
> to
>
> test:
>   stage: test
>   variables:
>     EMACS_EMBA_CI: 1
>
> and then add the code which knows about EMACS_EMBA_CI. If you want me to
> do it, tell me what tests to condition on that environment var.

Pls apply the changes in .gitlab-ci.yml. I'll continue then to adapt 
test/lisp/filenotify-tests.el. Likely, this needs several rounds.

> Thanks
> Ted

Best regards, Michael.

Re: filenotify-tests.el failing inside Docker debian:stretch

[Ted Zlatanov]

On Wed, 02 Jan 2019 15:48:57 +0100 Michael Albinus <michael.albinus@gmx.de> wrote: 

MA> Pls apply the changes in .gitlab-ci.yml. I'll continue then to adapt 
MA> test/lisp/filenotify-tests.el. Likely, this needs several rounds.

OK; I've put EMACS_EMBA_CI=1 in the branch scratch/tzz/cicd

Ted

Re: filenotify-tests.el failing inside Docker debian:stretch

[Michael Albinus]

Ted Zlatanov <tzz@lifelogs.com> writes:

Hi Ted,

> MA> Pls apply the changes in .gitlab-ci.yml. I'll continue then to adapt 
> MA> test/lisp/filenotify-tests.el. Likely, this needs several rounds.
>
> OK; I've put EMACS_EMBA_CI=1 in the branch scratch/tzz/cicd

I've created branch scratch/albinus on top of your scratch/tzz/cicd, and
did enable prints in filenotify-tests.el. However, the Job on EMBA
failed already during compilation, see
https://emba.gnu.org/emacs/emacs/-/jobs/443/raw

Strange, since the compilation in your branch succeeds. Is there
something I should know?

> Ted

Best regards, Michael.

Re: filenotify-tests.el failing inside Docker debian:stretch

[Ted Zlatanov]

On Thu, 03 Jan 2019 15:32:52 +0100 Michael Albinus <michael.albinus@gmx.de> wrote: 

MA> I've created branch scratch/albinus on top of your scratch/tzz/cicd, and
MA> did enable prints in filenotify-tests.el. However, the Job on EMBA
MA> failed already during compilation, see
MA> https://emba.gnu.org/emacs/emacs/-/jobs/443/raw

MA> Strange, since the compilation in your branch succeeds. Is there
MA> something I should know?

The machine was rebooted and the system change to enable Docker builds
didn't persist. I made the change manually and will make it persistent
shortly. I'll kick off your job again.

Ted