Re: Bug#683505: notmuch: FTBFS if built twice in a row: unrepresentable changes to source

Re: Bug#683505: notmuch: FTBFS if built twice in a row: unrepresentable changes to source

[David Bremner]

Jakub Wilk <jwilk@debian.org> writes:

> | dpkg-source: error: cannot represent change to test/tmp.json/mail/.notmuch/xapian/record.baseB: binary file contents changed
> | dpkg-source: error: add test/tmp.json/mail/.notmuch/xapian/record.baseB in debian/source/include-binaries if you want to store the modified binary in the debian tarball
> -- 
> Jakub Wilk

I guess the problem here is that the notmuch build process does not
clean up the test directories in case of failing tests. I guess it
probably should if the user explicitly runs "make clean".

Any thoughts on that notmuch people? Should we add a list of test/tmp.*
to be cleaned?

d

Re: Bug#683505: notmuch: FTBFS if built twice in a row: unrepresentable changes to source

[Jameson Graef Rollins]

[-- Attachment #1: Type: text/plain, Size: 575 bytes --]

On Wed, Aug 01 2012, David Bremner <david@tethera.net> wrote:
> I guess the problem here is that the notmuch build process does not
> clean up the test directories in case of failing tests. I guess it
> probably should if the user explicitly runs "make clean".

If a test is failing shouldn't the package build fail as well?  That
sounds like the real issue to me.

Jakub, can you provide more context of this failure?  It would be
helpful to see the test failure itself.

But yes, I agree that make clean should purge everything, including
remnants of failed tests.

jamie.

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: Bug#683505: notmuch: FTBFS if built twice in a row: unrepresentable changes to source

[Tomi Ollila]

On Wed, Aug 01 2012, David Bremner <david@tethera.net> wrote:

> Jakub Wilk <jwilk@debian.org> writes:
>
>> | dpkg-source: error: cannot represent change to test/tmp.json/mail/.notmuch/xapian/record.baseB: binary file contents changed
>> | dpkg-source: error: add test/tmp.json/mail/.notmuch/xapian/record.baseB in debian/source/include-binaries if you want to store the modified binary in the debian tarball
>> -- 
>> Jakub Wilk
>
> I guess the problem here is that the notmuch build process does not
> clean up the test directories in case of failing tests. I guess it
> probably should if the user explicitly runs "make clean".
>
> Any thoughts on that notmuch people? Should we add a list of test/tmp.*
> to be cleaned?

yes.

>
> d

Tomi

Re: Bug#683505: notmuch: FTBFS if built twice in a row: unrepresentable changes to source

[David Bremner]

Jameson Graef Rollins <jrollins@finestructure.net> writes:

> If a test is failing shouldn't the package build fail as well?  That
> sounds like the real issue to me.

As I mentioned on IRC, the test only fails on the Debian build machines
(building in a clean chroot using sbuild is not enough) so it isn't
really clear how to duplicate the it. Perhaps building in a clean
virtual machine without networking would do it.  For which tests fail,
see

https://buildd.debian.org/status/fetch.php?pkg=notmuch&arch=i386&ver=0.13.2-1&stamp=1338740444

I think the first things to fail are emacs tests. At a wild guess, it
looks like all of the failing tests are related to emacs.

I don't think failing the build is the right thing to do, since there
does not actually seem to be anything wrong with the resulting
packages. So removing them from Debian because of some problems with
running the test suite seems a bit extreme.  People not using
notmuch-emacs would probably be especially annoyed ;).

d

Re: Bug#683505: notmuch: FTBFS if built twice in a row: unrepresentable changes to source

[Jameson Graef Rollins]

[-- Attachment #1: Type: text/plain, Size: 1491 bytes --]

On Wed, Aug 01 2012, David Bremner <david@tethera.net> wrote:
> As I mentioned on IRC, the test only fails on the Debian build machines
> (building in a clean chroot using sbuild is not enough) so it isn't
> really clear how to duplicate the it. Perhaps building in a clean
> virtual machine without networking would do it.  For which tests fail,
> see
>
> https://buildd.debian.org/status/fetch.php?pkg=notmuch&arch=i386&ver=0.13.2-1&stamp=1338740444
>
> I think the first things to fail are emacs tests. At a wild guess, it
> looks like all of the failing tests are related to emacs.

From a cursory look that does appear to be the case.  The non-emacs
tests that are also failing (json and crypto) are using
emacs_deliver_message.  Do we have any idea what's going on here?

> I don't think failing the build is the right thing to do, since there
> does not actually seem to be anything wrong with the resulting
> packages. So removing them from Debian because of some problems with
> running the test suite seems a bit extreme.  People not using
> notmuch-emacs would probably be especially annoyed ;).

But it also doesn't make much sense to me to run the tests as part of
the build only to ignore the result.  How do we know that the failing
tests aren't actually affecting the distributed packages?  The emacs
failures could be masking real failures of the emacs interface (which
users of notmuch-emacs would probably also find really annoying!).

jamie.

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: Bug#683505: notmuch: FTBFS if built twice in a row: unrepresentable changes to source

[Tomi Ollila]

On Thu, Aug 02 2012, Jameson Graef Rollins <jrollins@finestructure.net> wrote:

> On Wed, Aug 01 2012, David Bremner <david@tethera.net> wrote:
>> As I mentioned on IRC, the test only fails on the Debian build machines
>> (building in a clean chroot using sbuild is not enough) so it isn't
>> really clear how to duplicate the it. Perhaps building in a clean
>> virtual machine without networking would do it.  For which tests fail,
>> see
>>
>> https://buildd.debian.org/status/fetch.php?pkg=notmuch&arch=i386&ver=0.13.2-1&stamp=1338740444
>>
>> I think the first things to fail are emacs tests. At a wild guess, it
>> looks like all of the failing tests are related to emacs.
>
> From a cursory look that does appear to be the case.  The non-emacs
> tests that are also failing (json and crypto) are using
> emacs_deliver_message.  Do we have any idea what's going on here?

is this related (still in my personal patches pool):

diff --git a/test/test-lib.el b/test/test-lib.el
index 6271da2..503a130 100644
--- a/test/test-lib.el
+++ b/test/test-lib.el
@@ -35,6 +35,16 @@
     "Disable yes-or-no-p before executing kill-emacs"
     (defun yes-or-no-p (prompt) t)))
 
+;; Work around a problem with
+;; GNU Emacs 23.1.1 (x86_64-redhat-linux-gnu) of 2011-03-26 on sl6.fnal.gov
+;; where get-buffer-process doesn't return nil without (list-processes)
+;; (value of delete-exited-process is t)
+
+(if (string= emacs-version "23.1.1")
+    (defadvice get-buffer-process (before run-list-processes activate)
+      "run (list-processes) before executing get-buffer-process"
+      (list-processes)))
+
 (defun notmuch-test-wait ()
   "Wait for process completion."
   (while (get-buffer-process (current-buffer))


I also have a commit message:

Subject: [PATCH] test: emacs: run list-processes before get-buffer-process in emacs 23.1.1

The function get-buffer-process does not return nil when process has
exited in some (if not every) emacs 23.1.1 versions, even the value
of delete-exited-process is t. Particularly, GNU Emacs 23.1.1
(x86_64-redhat-linux-gnu) of 2011-03-26 on sl6.fnal.gov has this
problem -- the execution halts when running emacs tests.

In this emacs version, when defadvice is used to run list-processes
before get-buffer-process the problem seems to be fixed -- emacs
tests run fine.

To fix this permanently, this defadvice is added to the starting
emacs when emacs-version equals "23.1.1".

---

It took me a while to get tests running on Scientific Linux 6 which
has this particular emacs version. I tried all kinds of hacks to get
it working -- this works best (or, actually, is the only one that works).
I just don't understand why... Maybe the emacs-server -interaction brings
some hard-to-spot interferences into equation...

In the above 'buildd' output emacs 23.4 is used and therefore this patch
in the current format would be no-op there.

In the buildd environment, just adding the follwing lines to test/test-lib.el

(defadvice get-buffer-process (before run-list-processes activate)
  "run (list-processes) before executing get-buffer-process"
  (list-processes))

could be tried.

Tomi

>
>> I don't think failing the build is the right thing to do, since there
>> does not actually seem to be anything wrong with the resulting
>> packages. So removing them from Debian because of some problems with
>> running the test suite seems a bit extreme.  People not using
>> notmuch-emacs would probably be especially annoyed ;).
>
> But it also doesn't make much sense to me to run the tests as part of
> the build only to ignore the result.  How do we know that the failing
> tests aren't actually affecting the distributed packages?  The emacs
> failures could be masking real failures of the emacs interface (which
> users of notmuch-emacs would probably also find really annoying!).
>
> jamie.
> _______________________________________________
> notmuch mailing list
> notmuch@notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch

Cleaning up cleaning

[david]

I ran debuild twice in a row with these patches and it seems OK.

I did notice there are still some leftover byte-compiled python (.pyc)
files left over after "make distclean"

[PATCH 1/3] build system: remove directories created by tests in "make clean"

[david]

From: David Bremner <bremner@debian.org>

These extra directories cause problems for building on Debian twice in
a row.

In order to remove directories, we need to us "rm -rf" instead of "rm
-r". So now we should be extra careful what we add to the variable CLEAN.
---
 Makefile.local      |    2 +-
 test/Makefile.local |    3 ++-
 2 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/Makefile.local b/Makefile.local
index 296995d..6d6896a 100644
--- a/Makefile.local
+++ b/Makefile.local
@@ -265,7 +265,7 @@ quiet ?= $($(shell echo $1 | sed -e s'/ .*//'))
 
 .PHONY : clean
 clean:
-	rm -f $(CLEAN); rm -rf .deps
+	rm -rf $(CLEAN); rm -rf .deps
 
 # We don't (yet) have any distributed files not in the upstream repository.
 # So distclean is currently identical to clean.
diff --git a/test/Makefile.local b/test/Makefile.local
index 4a6a4b1..c7f1435 100644
--- a/test/Makefile.local
+++ b/test/Makefile.local
@@ -31,4 +31,5 @@ check: test
 SRCS := $(SRCS) $(smtp_dummy_srcs)
 CLEAN := $(CLEAN) $(dir)/smtp-dummy $(dir)/smtp-dummy.o \
 	 $(dir)/symbol-test $(dir)/symbol-test.o \
-	 $(dir)/arg-test $(dir)/arg-test.o
+	 $(dir)/arg-test $(dir)/arg-test.o \
+	 $(dir)/corpus.mail $(dir)/test-results $(dir)/tmp.*
-- 
1.7.10.4

[PATCH 2/3] build system: remove configure output in Make distclean.

[david]

From: David Bremner <bremner@debian.org>

Create a variable DISTCLEAN which contains a list of things to clean
in the distclean target (in addition to running the clean target).

The deleted comment seems to be false these days, since we do create
files during configuration.

Use "rm -rf" here as well in case we want to add directories to DISTCLEAN.
---
 Makefile.local |    5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/Makefile.local b/Makefile.local
index 6d6896a..b3b960c 100644
--- a/Makefile.local
+++ b/Makefile.local
@@ -267,10 +267,9 @@ quiet ?= $($(shell echo $1 | sed -e s'/ .*//'))
 clean:
 	rm -rf $(CLEAN); rm -rf .deps
 
-# We don't (yet) have any distributed files not in the upstream repository.
-# So distclean is currently identical to clean.
 .PHONY: distclean
 distclean: clean
+	rm -rf $(DISTCLEAN)
 
 notmuch_client_srcs =		\
 	command-line-arguments.c\
@@ -339,6 +338,8 @@ install-desktop:
 SRCS  := $(SRCS) $(notmuch_client_srcs)
 CLEAN := $(CLEAN) notmuch notmuch-shared $(notmuch_client_modules) notmuch.elc
 
+DISTCLEAN := $(DISTCLEAN) .first-build-message Makefile.config
+
 DEPS := $(SRCS:%.c=.deps/%.d)
 DEPS := $(DEPS:%.cc=.deps/%.d)
 -include $(DEPS)
-- 
1.7.10.4

[PATCH 3/3] debian: update changelog for cleaning changes

[david]

From: David Bremner <bremner@debian.org>

---
 debian/changelog |    6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/debian/changelog b/debian/changelog
index 2518988..dfdcf35 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -3,7 +3,11 @@ notmuch (0.14-1) unstable; urgency=low
   [ Stefano Zacchiroli ]
   * notmuch-mutt: fix tag action invocation (Closes: #678012)
 
- -- David Bremner <bremner@debian.org>  Thu, 12 Jul 2012 17:47:21 -0600
+  [ David Bremner ]
+  * Do a better job of cleaning up after configuration and testing
+    (Closes: #683505)
+
+ -- David Bremner <bremner@debian.org>  Thu, 02 Aug 2012 09:36:05 -0300
 
 notmuch (0.13.2-1) unstable; urgency=low
 
-- 
1.7.10.4

Re: Cleaning up cleaning

[Tomi Ollila]

On Thu, Aug 02 2012, david@tethera.net wrote:

> I ran debuild twice in a row with these patches and it seems OK.
>
> I did notice there are still some leftover byte-compiled python (.pyc)
> files left over after "make distclean"

Looks Good To Me -- I've thought doing the same for clean but that hasn't
annoyed me enough to take an action.


Tomi

Re: Bug#683505: notmuch: FTBFS if built twice in a row: unrepresentable changes to source

[Austin Clements]

Quoth Jameson Graef Rollins on Aug 01 at  8:10 pm:
> On Wed, Aug 01 2012, David Bremner <david@tethera.net> wrote:
> > As I mentioned on IRC, the test only fails on the Debian build machines
> > (building in a clean chroot using sbuild is not enough) so it isn't
> > really clear how to duplicate the it. Perhaps building in a clean
> > virtual machine without networking would do it.  For which tests fail,
> > see
> >
> > https://buildd.debian.org/status/fetch.php?pkg=notmuch&arch=i386&ver=0.13.2-1&stamp=1338740444
> >
> > I think the first things to fail are emacs tests. At a wild guess, it
> > looks like all of the failing tests are related to emacs.
> 
> From a cursory look that does appear to be the case.  The non-emacs
> tests that are also failing (json and crypto) are using
> emacs_deliver_message.  Do we have any idea what's going on here?

There is one other illuminating tidbit in the buildd log:

emacs-subject-to-filename: Testing emacs: mail subject to filename
test-lib.sh: line 187: 30606 Terminated              sleep 1
FATAL: Unexpected exit with code 1

From a cursory glance, emacs-subject-to-filename appears to be the
only test that calls test_emacs outside of a subtest and hence without
stdout/stderr redirection.

The line number is useless, but, assuming valgrind isn't enabled,
there's only one place we sleep 1 in test-lib.sh: in the loop in
test_emacs that waits for the Emacs server to start up.  Furthermore,
timeout sends SIGTERM by default, suggesting that we're timing out
while we're spinning in that loop.

Re: Bug#683505: notmuch: FTBFS if built twice in a row: unrepresentable changes to source

[Tomi Ollila]

On Thu, Aug 02 2012, Austin Clements <amdragon@MIT.EDU> wrote:

> Quoth Jameson Graef Rollins on Aug 01 at  8:10 pm:
>> On Wed, Aug 01 2012, David Bremner <david@tethera.net> wrote:
>> > As I mentioned on IRC, the test only fails on the Debian build machines
>> > (building in a clean chroot using sbuild is not enough) so it isn't
>> > really clear how to duplicate the it. Perhaps building in a clean
>> > virtual machine without networking would do it.  For which tests fail,
>> > see
>> >
>> > https://buildd.debian.org/status/fetch.php?pkg=notmuch&arch=i386&ver=0.13.2-1&stamp=1338740444
>> >
>> > I think the first things to fail are emacs tests. At a wild guess, it
>> > looks like all of the failing tests are related to emacs.
>> 
>> From a cursory look that does appear to be the case.  The non-emacs
>> tests that are also failing (json and crypto) are using
>> emacs_deliver_message.  Do we have any idea what's going on here?
>
> There is one other illuminating tidbit in the buildd log:
>
> emacs-subject-to-filename: Testing emacs: mail subject to filename
> test-lib.sh: line 187: 30606 Terminated              sleep 1
> FATAL: Unexpected exit with code 1
>
>From a cursory glance, emacs-subject-to-filename appears to be the
> only test that calls test_emacs outside of a subtest and hence without
> stdout/stderr redirection.
>
> The line number is useless, but, assuming valgrind isn't enabled,
> there's only one place we sleep 1 in test-lib.sh: in the loop in
> test_emacs that waits for the Emacs server to start up.  Furthermore,
> timeout sends SIGTERM by default, suggesting that we're timing out
> while we're spinning in that loop.

The situation sounds strangely familiar... I remember seeing 'sleep 1'
with ascending pid in process list around the same time I had this
(notmuch-test-wait) problem... I think the system was lacking the
server socket in /tmp/emacs-<pid>/ directory...

Hmm, now I remember something -- there was some error happening
in emacs startup and therefore the (server-start) was never executed
-- the test_emacs '()' in loop can never connect the socket.

In the above case it seems like the first test
test_emacs '(notmuch-hello) (test-output)' couldn't be executed.
and as there is no test/emacs.el file "$load_emacs_tests" is empty
(instead of --eval '(load "$TEST_DIRECTORY/emacs.el") -- so that
cannot break it.

Unfortunately I did not investigate that further (or it was my own
mistake that made emacs fail) -- but if that happens again and
one is monitoring the progress (maybe using larger value than '2m' for
timeout) the failing emacs can be entered by 'dtach -a $socket'.
The $socket can be found by executing 'ps aww | grep dtach'.

Tomi