bug#26253: Coreutils 8.26 test failure on arm-linux-gnueabihf

bug#26253: Coreutils 8.26 test failure on arm-linux-gnueabihf

[Ludovic Courtès]

In current ‘core-updates’ (7fdca77e007f0838ecd32b044827678f2b345ff0),
tests/misc/cut-huge-range.sh in Coreutils 8.26 fails reproducibly in the
chroot build environment; it succeeds outside of it.

When it fails, cut-huge-range.log shows this:

--8<---------------cut here---------------start------------->8---
+ CUT_MAX=4294967294
+ ulimit -v 3004
+ cut -b4294967294- /dev/null
+ ulimit -v 3004
+ cut -b1-4294967294 /dev/null
+ ulimit -v 3004
+ returns_ 1 cut -b4294967295 /dev/null
+ fail=1
+ ulimit -v 3004
+ returns_ 1 cut -b4294967296 /dev/null
+ fail=1
+ compare /dev/null err
+ compare_dev_null_ /dev/null err
+ test 2 = 2
+ test x/dev/null = x/dev/null
+ test -s err
+ return 0
+ return 0
+ Exit 1
--8<---------------cut here---------------end--------------->8---

So the two lines that fail are these:

--8<---------------cut here---------------start------------->8---
# Explicitly disallow values above CUT_MAX
(ulimit -v $vm && returns_ 1 cut -b$SIZE_MAX /dev/null 2>/dev/null) || fail=1
(ulimit -v $vm && returns_ 1 cut -b$SIZE_OFLOW /dev/null 2>/dev/null) || fail=1
--8<---------------cut here---------------end--------------->8---

I don’t understand why at this point, but I found that removing
“2>/dev/null” from these two lines lets the test pass (and AIUI, this
redirection is not necessary for the test; it just hides the

  cut: byte/character offset '4294967296' is too large
  
error message.)

I’ve tried to reduce the test case to facilitate debugging, to no
avail.

To be continued!

Ludo’.

bug#26253: Coreutils 8.26 test failure on arm-linux-gnueabihf

[Leo Famulari]

On Sat, Mar 25, 2017 at 04:45:33PM +0100, Ludovic Courtès wrote:
> In current ‘core-updates’ (7fdca77e007f0838ecd32b044827678f2b345ff0),
> tests/misc/cut-huge-range.sh in Coreutils 8.26 fails reproducibly in the
> chroot build environment; it succeeds outside of it.

[...]

> So the two lines that fail are these:
> 
> --8<---------------cut here---------------start------------->8---
> # Explicitly disallow values above CUT_MAX
> (ulimit -v $vm && returns_ 1 cut -b$SIZE_MAX /dev/null 2>/dev/null) || fail=1
> (ulimit -v $vm && returns_ 1 cut -b$SIZE_OFLOW /dev/null 2>/dev/null) || fail=1
> --8<---------------cut here---------------end--------------->8---
> 
> I don’t understand why at this point, but I found that removing
> “2>/dev/null” from these two lines lets the test pass (and AIUI, this
> redirection is not necessary for the test; it just hides the

Is the same Bash shared between these two test environments (inside and
outside of the chroot)?

bug#26253: Coreutils 8.26 test failure on arm-linux-gnueabihf

[Ludovic Courtès]

Leo Famulari <leo@famulari.name> skribis:

> On Sat, Mar 25, 2017 at 04:45:33PM +0100, Ludovic Courtès wrote:
>> In current ‘core-updates’ (7fdca77e007f0838ecd32b044827678f2b345ff0),
>> tests/misc/cut-huge-range.sh in Coreutils 8.26 fails reproducibly in the
>> chroot build environment; it succeeds outside of it.
>
> [...]
>
>> So the two lines that fail are these:
>> 
>> --8<---------------cut here---------------start------------->8---
>> # Explicitly disallow values above CUT_MAX
>> (ulimit -v $vm && returns_ 1 cut -b$SIZE_MAX /dev/null 2>/dev/null) || fail=1
>> (ulimit -v $vm && returns_ 1 cut -b$SIZE_OFLOW /dev/null 2>/dev/null) || fail=1
>> --8<---------------cut here---------------end--------------->8---
>> 
>> I don’t understand why at this point, but I found that removing
>> “2>/dev/null” from these two lines lets the test pass (and AIUI, this
>> redirection is not necessary for the test; it just hides the
>
> Is the same Bash shared between these two test environments (inside and
> outside of the chroot)?

Yes.  For the outside test, I just did “guix build -K” and then “source
environment-variables”, which should give me the right $SHELL and so on.

Ludo’.

bug#26253: Coreutils 8.26 test failure on arm-linux-gnueabihf

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 2434 bytes --]

ludo@gnu.org (Ludovic Courtès) skribis:

> So the two lines that fail are these:
>
> # Explicitly disallow values above CUT_MAX
> (ulimit -v $vm && returns_ 1 cut -b$SIZE_MAX /dev/null 2>/dev/null) || fail=1
> (ulimit -v $vm && returns_ 1 cut -b$SIZE_OFLOW /dev/null 2>/dev/null) || fail=1
>
> I don’t understand why at this point, but I found that removing
> “2>/dev/null” from these two lines lets the test pass (and AIUI, this
> redirection is not necessary for the test; it just hides the
>
>   cut: byte/character offset '4294967296' is too large
>   
> error message.)

With the attached file I (and a patch coreutils such that it would
build; #:tests? #f is enough), I was able to reproduce the problem.  The
build log I get is this:

--8<---------------cut here---------------start------------->8---
+ ulimit -v 2004
+ echo before
before
+ /gnu/store/iglljv347sj1mjx8r6li7plb4h7cv8h9-coreutils-8.26/bin/cut -b4294967296 /dev/null
/gnu/store/9bkl74hd3f1gysdkq22hp8w4z9w8022l-bash-4.4.12/bin/sh: xmalloc: variables.c:4539: cannot allocate 69 bytes (57344 bytes allocated)
+ echo after 2
after 2
/gnu/store/iglljv347sj1mjx8r6li7plb4h7cv8h9-coreutils-8.26/bin/cut: byte/character offset '4294967296' is too large
Try '/gnu/store/iglljv347sj1mjx8r6li7plb4h7cv8h9-coreutils-8.26/bin/cut --help' for more information.
/gnu/store/iglljv347sj1mjx8r6li7plb4h7cv8h9-coreutils-8.26/bin/cut: byte/character offset '4294967296' is too large
Try '/gnu/store/iglljv347sj1mjx8r6li7plb4h7cv8h9-coreutils-8.26/bin/cut --help' for more information.

;;; (bash "/gnu/store/9bkl74hd3f1gysdkq22hp8w4z9w8022l-bash-4.4.12/bin/sh")

;;; (status 0 0)

;;; (status2 256 1)

;;; (status3 256 1)
--8<---------------cut here---------------end--------------->8---

The thing that’s failing here is the shell that invokes ‘cut’, and thus
we get an exit value of 2 whereas the test explicitly checks for 1.

This is very sensitive: depending on the command sequence we pass to “sh
-c”, we hit that problem or we don’t.  The fact that removing
“2>/dev/null”  solved the problem was probably just chance.

So I think Coreutils commit 28803c8a3144d5d4363cdbd148bbe067af1a67c2,
which arbitrarily increased the value passed to “ulimit -v”, goes in the
right direction after all but is insufficient.

I’m trying out a new patch based on that and will push if it works.

Ludo’.


[-- Attachment #2: the test --]
[-- Type: text/plain, Size: 950 bytes --]

(use-modules (guix))

(define cu (@@ (gnu packages commencement) coreutils-final))
(define bash (@@ (gnu packages commencement) bash-final))

(define build
  #~(let* ((_ #$output)
           (cut (string-append #$cu "/bin/cut -b4294967296 /dev/null "))
           (status (system* (pk 'bash (string-append #$bash "/bin/sh")) "-c"
                            (string-append "set -x; (ulimit -v 2004; echo before; "
                                           cut " 2>/dev/null)"
                                           "; echo after $?")))
           (status2 (system cut))
           (status3 (apply system* (string-tokenize cut))))
      (pk 'status status (status:exit-val status))
      (pk 'status2 status2 (status:exit-val status2))
      (pk 'status3 status3 (status:exit-val status3))
      #f))

(with-store store
  (run-with-store store
    (mlet %store-monad ((drv (gexp->derivation "foo" build)))
      (built-derivations (list drv)))))

bug#26253: Coreutils 8.26 test failure on arm-linux-gnueabihf

[Ludovic Courtès]

I just pushed a fix:

  https://git.savannah.gnu.org/cgit/guix.git/commit/?h=core-updates&id=d1e3f59c9b8d1be79eded04fe23cceb56b4372f0

Ludo’.