build failures on mipsel

build failures on mipsel

[Daniel Kahn Gillmor]

[-- Attachment #1: Type: text/plain, Size: 2721 bytes --]

hey folks--

the notmuch 0.27 release candidates are failing to build on the debian
mipsel build daemons:

https://buildd.debian.org/status/fetch.php?pkg=notmuch&arch=mipsel&ver=0.27%7Erc0-1&stamp=1527466396&raw=0
https://buildd.debian.org/status/fetch.php?pkg=notmuch&arch=mipsel&ver=0.27%7Erc1-1&stamp=1528017305&raw=0

They both fail the test suite here:


T357-index-decryption: Testing indexing decrypted mail
 FAIL   stash decryption during show
	--- T357-index-decryption.9.expected	2018-06-03 09:13:02.472009180 +0000
	+++ T357-index-decryption.9.output	2018-06-03 09:13:02.472009180 +0000
	@@ -1 +1 @@
	-This is a test encrypted message with a wumpus.
	+
 FAIL   search should now find the contents
	--- T357-index-decryption.10.expected	2018-06-03 09:13:02.528010019 +0000
	+++ T357-index-decryption.10.output	2018-06-03 09:13:02.528010019 +0000
	@@ -1 +1 @@
	-thread:0000000000000003   2000-01-01 [1/1] Notmuch Test Suite; test encrypted message for cleartext index 002 (encrypted inbox unread)
	+


I've tried to replicate the problem over on eller.debian.org (the mipsel
porterbox) but when i build 0.27~rc1 over there in an schroot, i don't
even get to T357 because the emacs_fcc_message() function hangs in this
loop (is this known to not work in an schroot or under similar
situations?):

+(test/test-lib.sh:988): test_emacs(): sleep 1
+(test/test-lib.sh:987): test_emacs(): test_emacs '()'
+(test/test-lib.sh:960): test_emacs(): missing_dependencies=
+(test/test-lib.sh:961): test_emacs(): test_require_external_prereq dtach
+(test/test-lib.sh:674): test_require_external_prereq(): binary=dtach
+(test/test-lib.sh:675): test_require_external_prereq(): [[ '' == t ]]
+(test/test-lib.sh:679): test_require_external_prereq(): true
+(test/test-lib.sh:962): test_emacs(): test_require_external_prereq emacs
+(test/test-lib.sh:674): test_require_external_prereq(): binary=emacs
+(test/test-lib.sh:675): test_require_external_prereq(): [[ '' == t ]]
+(test/test-lib.sh:679): test_require_external_prereq(): true
+(test/test-lib.sh:963): test_emacs(): test_require_external_prereq emacsclient
+(test/test-lib.sh:674): test_require_external_prereq(): binary=emacsclient
+(test/test-lib.sh:675): test_require_external_prereq(): [[ '' == t ]]
+(test/test-lib.sh:679): test_require_external_prereq(): true
+(test/test-lib.sh:964): test_emacs(): test -z ''
+(test/test-lib.sh:966): test_emacs(): '[' -z notmuch-test-suite-1593 ']'
+(test/test-lib.sh:997): test_emacs(): rm -f OUTPUT
+(test/test-lib.sh:998): test_emacs(): touch OUTPUT
+(test/test-lib.sh:1000): test_emacs(): emacsclient --socket-name=notmuch-test-suite-1593 --eval '(notmuch-test-progn ())'

can anyone suggest a good next debugging step?

    --dkg

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: build failures on mipsel

[David Bremner]

Daniel Kahn Gillmor <dkg@fifthhorseman.net> writes:

>
>
> I've tried to replicate the problem over on eller.debian.org (the mipsel
> porterbox) but when i build 0.27~rc1 over there in an schroot, i don't
> even get to T357 because the emacs_fcc_message() function hangs in this
> loop (is this known to not work in an schroot or under similar
> situations?):

Try:

$ make test-binaries && ./T357-index-decryption.sh

That works for me on eller, so there must be some difference between the
two environments (buildd versus porterbox)

d

Re: build failures on mipsel

[Daniel Kahn Gillmor]

On Mon 2018-06-04 23:44:25 -0300, David Bremner wrote:
> $ make test-binaries && ./T357-index-decryption.sh
>
> That works for me on eller, so there must be some difference between the
> two environments (buildd versus porterbox)

what schroot are you working from on eller?  i wonder what the
difference is between your schroot and mine, because mine fails with the
same emacs problem that tomi identified.

     --dkg

Re: build failures on mipsel

[David Bremner]

Daniel Kahn Gillmor <dkg@fifthhorseman.net> writes:

> On Mon 2018-06-04 23:44:25 -0300, David Bremner wrote:
>> $ make test-binaries && ./T357-index-decryption.sh
>>
>> That works for me on eller, so there must be some difference between the
>> two environments (buildd versus porterbox)
>
> what schroot are you working from on eller?  i wonder what the
> difference is between your schroot and mine, because mine fails with the
> same emacs problem that tomi identified.
>
>      --dkg

I'm using schroot session bremner-notmuch, which iirc I based on sid and
running ./T357-index-decryption.sh directly (in tmux or out doesn't seem
to matter).

d

Re: build failures on mipsel

[Tomi Ollila]

On Mon, Jun 04 2018, Daniel Kahn Gillmor wrote:

> hey folks--
>
> the notmuch 0.27 release candidates are failing to build on the debian
> mipsel build daemons:
>
> https://buildd.debian.org/status/fetch.php?pkg=notmuch&arch=mipsel&ver=0.27%7Erc0-1&stamp=1527466396&raw=0
> https://buildd.debian.org/status/fetch.php?pkg=notmuch&arch=mipsel&ver=0.27%7Erc1-1&stamp=1528017305&raw=0
>
> They both fail the test suite here:
>
>
> T357-index-decryption: Testing indexing decrypted mail
>  FAIL   stash decryption during show
> 	--- T357-index-decryption.9.expected	2018-06-03 09:13:02.472009180 +0000
> 	+++ T357-index-decryption.9.output	2018-06-03 09:13:02.472009180 +0000
> 	@@ -1 +1 @@
> 	-This is a test encrypted message with a wumpus.
> 	+
>  FAIL   search should now find the contents
> 	--- T357-index-decryption.10.expected	2018-06-03 09:13:02.528010019 +0000
> 	+++ T357-index-decryption.10.output	2018-06-03 09:13:02.528010019 +0000
> 	@@ -1 +1 @@
> 	-thread:0000000000000003   2000-01-01 [1/1] Notmuch Test Suite; test encrypted message for cleartext index 002 (encrypted inbox unread)
> 	+
>
>
> I've tried to replicate the problem over on eller.debian.org (the mipsel
> porterbox) but when i build 0.27~rc1 over there in an schroot, i don't
> even get to T357 because the emacs_fcc_message() function hangs in this
> loop (is this known to not work in an schroot or under similar
> situations?):
>
> +(test/test-lib.sh:988): test_emacs(): sleep 1
> +(test/test-lib.sh:987): test_emacs(): test_emacs '()'
> +(test/test-lib.sh:960): test_emacs(): missing_dependencies=
> +(test/test-lib.sh:961): test_emacs(): test_require_external_prereq dtach
> +(test/test-lib.sh:674): test_require_external_prereq(): binary=dtach
> +(test/test-lib.sh:675): test_require_external_prereq(): [[ '' == t ]]
> +(test/test-lib.sh:679): test_require_external_prereq(): true
> +(test/test-lib.sh:962): test_emacs(): test_require_external_prereq emacs
> +(test/test-lib.sh:674): test_require_external_prereq(): binary=emacs
> +(test/test-lib.sh:675): test_require_external_prereq(): [[ '' == t ]]
> +(test/test-lib.sh:679): test_require_external_prereq(): true
> +(test/test-lib.sh:963): test_emacs(): test_require_external_prereq emacsclient
> +(test/test-lib.sh:674): test_require_external_prereq(): binary=emacsclient
> +(test/test-lib.sh:675): test_require_external_prereq(): [[ '' == t ]]
> +(test/test-lib.sh:679): test_require_external_prereq(): true
> +(test/test-lib.sh:964): test_emacs(): test -z ''
> +(test/test-lib.sh:966): test_emacs(): '[' -z notmuch-test-suite-1593 ']'
> +(test/test-lib.sh:997): test_emacs(): rm -f OUTPUT
> +(test/test-lib.sh:998): test_emacs(): touch OUTPUT
> +(test/test-lib.sh:1000): test_emacs(): emacsclient --socket-name=notmuch-test-suite-1593 --eval '(notmuch-test-progn ())'
>
> can anyone suggest a good next debugging step?

emacs did not start for some reason.

You could try changing the line

sh -c "stty rows 24 cols 80; exec '$TMP_DIRECTORY/run_emacs' \

to

script -c "stty rows 24 cols 80; exec '$TMP_DIRECTORY/run_emacs' \

in test-lib.sh and then see what you get in 'typescriptä file

>
>     --dkg


Tomi

Re: build failures on mipsel

[Daniel Kahn Gillmor]

[-- Attachment #1: Type: text/plain, Size: 880 bytes --]

On Mon 2018-06-04 17:52:46 -0400, Daniel Kahn Gillmor wrote:
> I've tried to replicate the problem over on eller.debian.org (the mipsel
> porterbox) but when i build 0.27~rc1 over there in an schroot, i don't
> even get to T357 because the emacs_fcc_message() function hangs in this
> loop (is this known to not work in an schroot or under similar
> situations?):

This problem appears to be related to some bad interaction with
screen(1) inside a sid chroot on eller.  i note that bash also has
problems with command line editing in that environment as well.

I'm now running inside of tmux instead, and that problem doesn't happen
any more.

Furthermore, i can't replicate the failure from the buildd when building
on eller either.  So neither bremner nor i were able to replicate the
problem.

mipsel porters, do you have any suggestions for next-step debugging?

       --dkg

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: build failures on mipsel

[Tomi Ollila]

On Wed, Jun 06 2018, Daniel Kahn Gillmor wrote:

> On Mon 2018-06-04 17:52:46 -0400, Daniel Kahn Gillmor wrote:
>> I've tried to replicate the problem over on eller.debian.org (the mipsel
>> porterbox) but when i build 0.27~rc1 over there in an schroot, i don't
>> even get to T357 because the emacs_fcc_message() function hangs in this
>> loop (is this known to not work in an schroot or under similar
>> situations?):
>
> This problem appears to be related to some bad interaction with
> screen(1) inside a sid chroot on eller.  i note that bash also has
> problems with command line editing in that environment as well.
>
> I'm now running inside of tmux instead, and that problem doesn't happen
> any more.
>
> Furthermore, i can't replicate the failure from the buildd when building
> on eller either.  So neither bremner nor i were able to replicate the
> problem.
>
> mipsel porters, do you have any suggestions for next-step debugging?


This is the line to start emacs

  # start a detached session with an emacs server
  # user's TERM (or 'vt100' in case user's TERM is known dumb
  # or unknown) is given to dtach which assumes a minimally
  # VT100-compatible terminal -- and emacs inherits that

  TERM=$SMART_TERM dtach -n "$TEST_TMPDIR/emacs-dtach-socket.$$" \
          sh -c "stty rows 24 cols 80; exec '$TMP_DIRECTORY/run_emacs' \
                  --no-window-system \
                  $load_emacs_tests \
                  --eval '(setq server-name \"$server_name\")' \
                  --eval '(server-start)' \
                  --eval '(orphan-watchdog $$)'" || return

my first guess: SMART_TERM is 'screen', and terminfo for that is missing

if not, then some TTY problem, which may break dtach, and if not dtach,
then emacs

my usual try is strace(1): 

  TERM=$SMART_TERM strace -f dtach -n "$TEST_TMPDIR/emacs-dtach-socket.$$" \
          sh -c "stty rows 24 cols 80; exec '$TMP_DIRECTORY/run_emacs' \
                  --no-window-system \
                  $load_emacs_tests \
                  --eval '(setq server-name \"$server_name\")' \
                  --eval '(server-start)' \
                  --eval '(orphan-watchdog $$)'" &

(have to be run in background as strace -f will not exit until every child
program have exited...)

and then comb through strace output (-o option to strace may be useful...).

>
>        --dkg

Re: build failures on mipsel

[Daniel Kahn Gillmor]

On Wed 2018-06-06 10:30:15 -0400, Daniel Kahn Gillmor wrote:
> Furthermore, i can't replicate the failure from the buildd when building
> on eller either.  So neither bremner nor i were able to replicate the
> problem.
>
> mipsel porters, do you have any suggestions for next-step debugging?

I notice that the two failures for notmuch

  https://buildd.debian.org/status/fetch.php?pkg=notmuch&arch=mipsel&ver=0.27%7Erc0-1&stamp=1527466396&raw=0
  https://buildd.debian.org/status/fetch.php?pkg=notmuch&arch=mipsel&ver=0.27%7Erc1-1&stamp=1528017305&raw=0

are both built on the buildd mipsel-manda-03.debian.org.

perhaps there's something wrong with that buildd?  maybe a giveback
would let another build daemon take a crack at the package.

mipsel folks, could you give back the experimental version of notmuch to
the mipsel buildd network and try to avoid having it land on
mipsel-manda-03?

thanks,

      --dkg

Re: build failures on mipsel

[Daniel Kahn Gillmor]

On Wed 2018-06-06 13:05:15 -0400, Daniel Kahn Gillmor wrote:
> mipsel folks, could you give back the experimental version of notmuch to
> the mipsel buildd network and try to avoid having it land on
> mipsel-manda-03?

Thank you for the giveback!  However, it looks like it landed on
mipsel-manda-03 again (and had the same error).  could you try giving it
back one more time?  is there a way to force it to another host?

or do you have another recommendation for debugging?

      --dkg