bug#62954: Valgrind blocks R on powerpc64le

bug#62954: Valgrind blocks R on powerpc64le

[Andreas Enge]

Currently r-minimal depends on texlive-latex-xkeyval, which depends on
texlive-ms, which for a reason I do not see pulls in the valgrind variable.

This is at version 3.17, which fails on powerpc64le. The user facing
variable valgrind/interactive, however, is at 3.20, and it builds.

After the impending core-updates merge, we should update valgrind to
3.20.

I am opening a bug so that this is not forgotten.

Andreas

bug#62954: Valgrind blocks R on powerpc64le

[Simon Tournier]

Hi,

On mer., 19 avril 2023 at 22:08, Andreas Enge <andreas@enge.fr> wrote:

> Currently r-minimal depends on texlive-latex-xkeyval, which depends on
> texlive-ms, which for a reason I do not see pulls in the valgrind variable.

--8<---------------cut here---------------start------------->8---
$ ./pre-inst-env guix graph --path r-minimal texlive-ms
r-minimal@4.2.3
texlive-updmap.cfg@59745
texlive-latex-xkeyval@59745
texlive-pgf@59745
texlive-ms@59745
--8<---------------cut here---------------end--------------->8---

But I do not see either how valgrind enters in the picture as a
dependency for r-minimal,

--8<---------------cut here---------------start------------->8---
$ ./pre-inst-env guix graph --path r-minimal -e '(@@ (gnu packages valgrind) valgrind)' -t bag -s powerpc64le-linux
guix graph: error: no path from 'r-minimal@4.2.3' to 'valgrind@3.17.0'
--8<---------------cut here---------------end--------------->8---

However, please note:

--8<---------------cut here---------------start------------->8---
$ for p in $(./pre-inst-env guix refresh -l -e '(@@ (gnu packages valgrind) valgrind)' | cut -f2 -d':'); do echo $p ;done | grep '^r\-'
r-bigmelon@1.24.0
r-multidplyr@0.1.3
r-cistopic-next@0.3.0-1.04cecbb
r-cistopic@2.1.0
r-chromunity@0.0.1-1.09fce8b
r-cicero-monocle3@1.3.2-1.fa2fb65
r-tmaptools@3.1-1
r-zonebuilder@0.0.2
r-chipseeker@1.34.1
r-snapatac@2.0
r-rastervis@0.51.5
r-insol@1.2.2
r-bien@1.2.6
r-sungeo@0.2.292
r-leaflet@2.1.2
r-spectre@0.5.5-1.f6648ab
r-zonator@0.6.0
r-zoon@0.6.5
r-biocdockermanager@1.10.0
r-irkernel@1.3.2
r-prereg@0.6.0
r-doubletcollection@1.1.0-1.c0d62f1
r-rmpi@0.7-1
r-pbdmpi@0.4-6
r-torch@0.10.0
r-ctrdata@1.12.1
--8<---------------cut here---------------end--------------->8---

and I guess that’s what you are observing.  Somehow, something like,

--8<---------------cut here---------------start------------->8---
for q in $(for p in $(./pre-inst-env guix refresh -l -e '(@@ (gnu packages valgrind) valgrind)' | cut -f2 -d':' | sort); do echo $p ;done | grep '^r\-'); do  echo "# $q";./pre-inst-env guix graph --path $q -e '(@@ (gnu packages valgrind) valgrind)' ;done | grep -B 2 valgrind
--8<---------------cut here---------------end--------------->8---

shows that most of the paths do not involve texlive-ms.  Instead, it
seems related to lz4 or openmpi or jq.

> This is at version 3.17, which fails on powerpc64le. The user facing
> variable valgrind/interactive, however, is at 3.20, and it builds.

As far I can see, it is hard to cross-compile since substitutes are
missing.  Well, maybe the CI is not building them.  I do not know.


> After the impending core-updates merge, we should update valgrind to
> 3.20.

Note the update of valgrind is not “so much“. :-)

--8<---------------cut here---------------start------------->8---
$ ./pre-inst-env guix refresh -l -e '(@@ (gnu packages valgrind) valgrind)' | cut -f1 -d':'
Building the following 569 packages would ensure 1169 dependent packages are rebuilt
--8<---------------cut here---------------end--------------->8---

Well, some packages are intensive to rebuild as ungoogled-chromium but I
guess that if these:

     23 jq@1.6
     37 qtwebengine@5.15.8
     39 openmpi@4.1.4
     45 dtc@1.6.1
    405 lz4@1.9.3

passes with valgrind at 3.20, we should almost be good, IMHO. :-)

Most of the 569 packages are rebuilt because lz4 is rebuild.  Well, I am
giving a try… If it is not part of the next core-updates merge, then
using a feature branch building the substitutes, the update of valgrind
could go to master. ;-)


Cheers,
simon

bug#62954: Valgrind blocks R on powerpc64le

[Andreas Enge]

Am Thu, Apr 20, 2023 at 01:26:37PM +0200 schrieb Simon Tournier:
> shows that most of the paths do not involve texlive-ms.  Instead, it
> seems related to lz4 or openmpi or jq.

So I start to understand. lz4 depends on valgrind. I do not know why.
It is given as a native input, so probably the tests require valgrind?
If yes, that looks a bit excessive to me - the developers of lz4 are very
welcome to check for memory leaks from time to time, but doing this at
every compilation is excessive. What do you think of dropping the valgrind
input (at the same time as updating valgrind, say)? It does not seem to be
necessary, as it is already dropped on architectures that do not provide it,
without further ado. Maybe this is what hides it from "guix refresh" and
"guix graph"?

Then texlive-ms depends on lz4 and indirectly on valgrind:
$ ./pre-inst-env guix build --system=powerpc64le-linux texlive-ms -n
  /gnu/store/j0wzdc36vvgvj4zh49a71sc115m6m76b-texlive-ms-59745.drv
  /gnu/store/jl6p6m1zngi1rjl2808zvnb9wpiphhjf-texlive-ms-59745-checkout.drv
  /gnu/store/gfkdb5pys2b8dr2mqgs5gbwfflwlc4kh-subversion-1.14.2.drv
  /gnu/store/xw5kdli3i92iwd7wpplcb0p89g1p3a29-lz4-1.9.3.drv
  /gnu/store/z6kf2pg48b9a87angabkfyfv4knhhwjy-valgrind-3.17.0.drv

More precisely: texlive-ms is downloaded via subversion via
simple-texlive-package in (gnu packages texlive), which relies on
texlive-origin from (guix build-system texlive).

$ ./pre-inst-env guix build --system=powerpc64le-linux subversion -n
The following derivations would be built:
  /gnu/store/gfkdb5pys2b8dr2mqgs5gbwfflwlc4kh-subversion-1.14.2.drv
  /gnu/store/xw5kdli3i92iwd7wpplcb0p89g1p3a29-lz4-1.9.3.drv
  /gnu/store/z6kf2pg48b9a87angabkfyfv4knhhwjy-valgrind-3.17.0.drv

So subversion depends on valgrind! And all new simplex-texlive-packages
are concerned.

I think the solution is to indeed remove valgrind from the native inputs
of lz4.

And I think there is a shortcoming in "guix refresh" that it does not
take the source code of packages into account.

What do you think?

Andreas

bug#62954: Valgrind blocks R on powerpc64le

[Simon Tournier]

Hi,

On jeu., 20 avril 2023 at 13:50, Andreas Enge <andreas@enge.fr> wrote:

> So subversion depends on valgrind! And all new simplex-texlive-packages
> are concerned.

Good catch! :-)

> I think the solution is to indeed remove valgrind from the native inputs
> of lz4.

Yeah, and I think it’s safe. :-)  Considering,

--8<---------------cut here---------------start------------->8---
starting phase `check'
CFLAGS="" LDFLAGS="" make -C tests test
--8<---------------cut here---------------end--------------->8---

and the ’test’ suite reads,

--8<---------------cut here---------------start------------->8---
$ grep '^test:' $(./pre-inst-env guix build lz4 -S)/tests/Makefile
test: test-lz4 test-lz4c test-frametest test-fullbench test-fuzzer test-install test-amalgamation listTest test-decompress-partial
--8<---------------cut here---------------end--------------->8---

Last, ’valgrind’ appears only in this rule:

--8<---------------cut here---------------start------------->8---
test-mem: lz4 datagen fuzzer frametest fullbench
	@echo "\n ---- valgrind tests : memory analyzer ----"
	valgrind --leak-check=yes --error-exitcode=1 $(DATAGEN) -g50M > $(VOID)
	$(DATAGEN) -g16KB > ftmdg16K
	valgrind --leak-check=yes --error-exitcode=1 $(LZ4) -9 -BD -f ftmdg16K $(VOID)
	$(DATAGEN) -g16KB -s2 > ftmdg16K2
	$(DATAGEN) -g16KB -s3 > ftmdg16K3
	valgrind --leak-check=yes --error-exitcode=1 $(LZ4) --force --multiple ftmdg16K ftmdg16K2 ftmdg16K3
	$(DATAGEN) -g7MB > ftmdg7M
	valgrind --leak-check=yes --error-exitcode=1 $(LZ4) -9 -B5D -f ftmdg7M ftmdg16K2
	valgrind --leak-check=yes --error-exitcode=1 $(LZ4) -t ftmdg16K2
	valgrind --leak-check=yes --error-exitcode=1 $(LZ4) -bi1 ftmdg7M
	valgrind --leak-check=yes --error-exitcode=1 ./fullbench -i1 ftmdg7M ftmdg16K2
	valgrind --leak-check=yes --error-exitcode=1 $(LZ4) -B4D -f -vq ftmdg7M $(VOID)
	valgrind --leak-check=yes --error-exitcode=1 $(LZ4) --list -m ftm*.lz4
	valgrind --leak-check=yes --error-exitcode=1 $(LZ4) --list -m -v ftm*.lz4
	$(RM) ftm*
	valgrind --leak-check=yes --error-exitcode=1 ./fuzzer -i64 -t1
	valgrind --leak-check=yes --error-exitcode=1 ./frametest -i256
--8<---------------cut here---------------end--------------->8---

so ’valgrind’ is never used by the current recipe.  Ahah! :-D

And it’s confirmed: lz4 builds fine on core-updates for x68_64.

Therefore, I think this change before the merge is worth:

--8<---------------cut here---------------start------------->8---
1 file changed, 1 insertion(+), 5 deletions(-)
gnu/packages/compression.scm | 6 +-----

modified   gnu/packages/compression.scm
@@ -840,11 +840,7 @@ (define-public lz4
     (build-system gnu-build-system)
     (outputs (list "out" "static"))
     (native-inputs
-     (append
-       (list python)    ;; For tests.
-       (if (member (%current-system) (package-supported-systems valgrind))
-         (list valgrind)
-         '())))
+     (list python))
     (arguments
      `(;; Not designed for parallel testing.
        ;; See https://github.com/lz4/lz4/issues/957#issuecomment-737419821
--8<---------------cut here---------------end--------------->8---


Cheers,
simon

bug#62954: Valgrind blocks R on powerpc64le

[Simon Tournier]

Hi Andreas,

On Thu, 20 Apr 2023 at 13:50, Andreas Enge <andreas@enge.fr> wrote:

> I think the solution is to indeed remove valgrind from the native inputs
> of lz4.

See #62967 [1] for an attempt.  I am rebuilding to detect the potential problem.

1: https://issues.guix.gnu.org/62967

Cheers,
simon

bug#62954: Valgrind blocks R on powerpc64le

[Andreas Enge]

Am Thu, Apr 20, 2023 at 03:25:17PM +0200 schrieb Simon Tournier:
> See #62967 [1] for an attempt.  I am rebuilding to detect the potential problem.

Not sure why we need a second issue... All this should work, let us wait
till after the core-updates merge.

Andreas

bug#62954: Valgrind blocks R on powerpc64le

[Simon Tournier]

Hi,

On Thu, 20 Apr 2023 at 18:55, Andreas Enge <andreas@enge.fr> wrote:
> Am Thu, Apr 20, 2023 at 03:25:17PM +0200 schrieb Simon Tournier:
>> See #62967 [1] for an attempt.  I am rebuilding to detect the potential problem.
>
> Not sure why we need a second issue... All this should work, let us wait
> till after the core-updates merge.

Because it is two “mailing lists“ under the hood: guix-patches and bug-guix.

As someone who digs (time to time, and not enough these days, oups!) to
“forgotten” issues, I find easier to follow the discussion about one
specific bug and the discussion about one specific implementation
when they are separated – somehow it reduces the “noise”.

Well, for example consider #58650 [1].  It is time-consuming to know if
the PATCH is “forgotten“ or if it does not fix the issue and can be
dropped.

Moreover, the two patches of #58650 [1] does not appear if you follow
only guix-patches; because the issue had not been marked +patch.

Speaking about “forgotten” issues, #62967 (patch) does not mention
#62954 (bug) and that is a kind of “mistake”, I agree. :-)

Last, on the pragmatic side, I do not know if the CI is following
bug-guix and if it tries to extract patches from it.  (Yes, for this
case it is not relevant; the patch is dropped by CI because it implies
more rebuilds than the threshold.)

Well, it’s a matter of taste. :-)

1: https://issues.guix.gnu.org/issue/58650


Cheers,
simon

bug#62954: Valgrind blocks R on powerpc64le

[Andreas Enge]

Am Fri, Apr 21, 2023 at 09:59:37AM +0200 schrieb Simon Tournier:
> Because it is two “mailing lists“ under the hood: guix-patches and bug-guix.
> Last, on the pragmatic side, I do not know if the CI is following
> bug-guix and if it tries to extract patches from it.

Ah indeed, I suppose it does not, thanks for the explanation.

Andreas

bug#62954: Valgrind blocks R on powerpc64le

[Andreas Enge]

Closing this bug, the topic is treated through the patches in
   https://issues.guix.gnu.org/62967

Andreas