bug#53406: union-build incorrectly handles grafts

[John Kehayias via Bug reports for GNU Guix <bug-guix@gnu.org>]

[-- Attachment #1: Type: text/plain, Size: 6585 bytes --]

Hi Ludo’,

Thanks for explaining!

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

On Monday, January 24th, 2022 at 9:18 AM, Ludovic Courtès wrote:

>
> Grafting is a pretty basic process: in this case it replaces occurrences
> of /gnu/store/…-expat-2.4.1 with /gnu/store/…-expat-2.4.3, nothing more.
> It cannot guess that libexpat.so.1.8.1 was renamed to libexpat.so.1.8.3
> or anything like that.
>
> Is it a problem? Normally no, because users of shared libraries don’t
> refer to libraries by their fully-qualified name:
>
> --8<---------------cut here---------------start------------->8---
> $ objdump -x $(guix build dbus-glib)/bin/dbus-binding-tool|grep NEED.*expat
> NEEDED libexpat.so.1
> $ objdump -x $(guix build dbus-glib)/bin/dbus-binding-tool|grep RUNPATH
> RUNPATH /gnu/store/wwmxxlmlhwljn39z0gsj6iai3zk67a2g-dbus-glib-0.110/lib:/gnu/store/5s6iz5f777rh23q4kv8gvqrsyy61cbjh-dbus-1.12.20/lib:/gnu/store/s0w7szfsajdy6cnrz2w7z4h5spyl4aaj-expat-2.4.1/lib:/gnu/store/2fk1gz2s7ppdicynscra9b19byrrr866-glibc-2.33/lib:/gnu/store/90lbavffg0csrf208nw0ayj1bz5knl47-gcc-10.3.0-lib/lib:/gnu/store/qqs98rxwjrji6aaf6dqwp7q4m545g2sn-glib-2.70.0/lib:/gnu/store/90lbavffg0csrf208nw0ayj1bz5knl47-gcc-10.3.0-lib/lib/gcc/x86_64-unknown-linux-gnu/10.3.0/../../..
> --8<---------------cut here---------------end--------------->8---
>
> Likewise, ‘etc/ld.so.cache’ contains a reference to ‘libexpat.so.1’, not
> to ‘libexpat.so.1.8.1’.
>

(Your example output was not of a grafted libexpat in the RUNPATH, but point taken; I see the newer expat version on my (grafted) dbus-glib).

> Does that make sense? Or am I overlooking something?
>

Yes, that makes sense, thank you for clarifying. So this is the currently expected behavior. Ideally grafting would be smarter to maybe avoid this (missing changes in e.g. version number)? But I would guess this is not something that would be expected to cause a problem for the vast majority of cases, as you explain, and adds complexity to the process.

I'm glad to hear it all works! But...

Perhaps I was too hasty in noting this "problem" which like I said was not the error I originally encountered. I was using a package that constructs both the 64- and 32-bit libraries to put in a container (say, a /lib32 and /lib64 or something similar to an FHS environment). A collision was happening between a file and directory, one being a good symlink and the other broken, rather than a "real" mismatch in file vs directory. Anyway, going back to that what I see is that one link is broken for the above reasons, but the good one is good because it is to the *ungrafted* library store path. I don't know now if these 2 things are connected other than one led me to the other, but I turn now to what demonstrates my original problem.

I don't know why this happens or if it is something in this building process that is not correct, but I did come up with a minimal example (attached). The code is a bit odd in its stripped down form, though hopefully is clear in what way this would be used to do something useful (again, like an FHS environment or other container). Apologies for the old style and lack of gexps which I'm finally getting used to. The example package just tries to make a dummy package that has, for illustration, a "/lib64" and "/lib32" which link to the respective union-build inputs (of a single library for simplicity). I don't think the actual package being made matters so much, or how it is constructed, but that two inputs are union-builds of the same library (x86_64 and i686) which should have a graft of expat. Just my guess though.

Doing:

ls -la $(guix build -f graft-test.scm)/lib64/lib/libexpat*
lrwxrwxrwx 1 root root 71 Dec 31  1969 /gnu/store/qbh16hfdv8pnfw01k6izbs3jkji6i978-test-pkg-0.0/lib64/lib/libexpat.la -> /gnu/store/2q8wwhd3prib0swky68rbx9hl0xxs6hf-expat-2.4.3/lib/libexpat.la*
lrwxrwxrwx 1 root root 71 Dec 31  1969 /gnu/store/qbh16hfdv8pnfw01k6izbs3jkji6i978-test-pkg-0.0/lib64/lib/libexpat.so -> /gnu/store/2q8wwhd3prib0swky68rbx9hl0xxs6hf-expat-2.4.3/lib/libexpat.so*
lrwxrwxrwx 1 root root 73 Dec 31  1969 /gnu/store/qbh16hfdv8pnfw01k6izbs3jkji6i978-test-pkg-0.0/lib64/lib/libexpat.so.1 -> /gnu/store/2q8wwhd3prib0swky68rbx9hl0xxs6hf-expat-2.4.3/lib/libexpat.so.1*
lrwxrwxrwx 1 root root 77 Dec 31  1969 /gnu/store/qbh16hfdv8pnfw01k6izbs3jkji6i978-test-pkg-0.0/lib64/lib/libexpat.so.1.8.1 -> /gnu/store/2q8wwhd3prib0swky68rbx9hl0xxs6hf-expat-2.4.3/lib/libexpat.so.1.8.1

is what we saw already: libexpat is the newer (replacement, 2.4.3) version, with the full version symlink broken since the version number is wrong. Likewise in other pieces that have the version number, like share/doc. Okay, that's expected. But now, in the i686-linux union-build input:

ls -la $(guix build -f graft-test.scm)/lib32/lib/libexpat*
lrwxrwxrwx 1 root root 71 Dec 31  1969 /gnu/store/qbh16hfdv8pnfw01k6izbs3jkji6i978-test-pkg-0.0/lib32/lib/libexpat.la -> /gnu/store/b0jns3vzhhpna7lim8bc3dr0payzx5yy-expat-2.4.1/lib/libexpat.la*
lrwxrwxrwx 1 root root 71 Dec 31  1969 /gnu/store/qbh16hfdv8pnfw01k6izbs3jkji6i978-test-pkg-0.0/lib32/lib/libexpat.so -> /gnu/store/b0jns3vzhhpna7lim8bc3dr0payzx5yy-expat-2.4.1/lib/libexpat.so*
lrwxrwxrwx 1 root root 73 Dec 31  1969 /gnu/store/qbh16hfdv8pnfw01k6izbs3jkji6i978-test-pkg-0.0/lib32/lib/libexpat.so.1 -> /gnu/store/b0jns3vzhhpna7lim8bc3dr0payzx5yy-expat-2.4.1/lib/libexpat.so.1*
lrwxrwxrwx 1 root root 77 Dec 31  1969 /gnu/store/qbh16hfdv8pnfw01k6izbs3jkji6i978-test-pkg-0.0/lib32/lib/libexpat.so.1.8.1 -> /gnu/store/b0jns3vzhhpna7lim8bc3dr0payzx5yy-expat-2.4.1/lib/libexpat.so.1.8.1*

all the links are good and to the original (version 2.4.1) expat. In other words, the constructed union64 and union32 inputs (in the sample code) do not both get grafted, even though doing just the fhs-union command on it's own (not building both for another package) does graft for either architecture. At least that seems like the most obvious difference between the earlier example and this new one.

Why? Does the grafting just happen "once" somehow and misses the "same" input again (but built for different system)? Is this expected or just a weird/wrong way to do this kind of build which is causing this? I'm not sure if this is just with union-build or if it would happen just with inputs of the same library but different architectures. I didn't know how to do that quickly off hand, so I haven't tried it yet.

Thanks for taking the time to look and explain, much appreciated!

John

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: graft-test.scm --]
[-- Type: text/x-scheme; name=graft-test.scm, Size: 3559 bytes --]

;;; GNU Guix --- Functional package management for GNU
;;; Copyright © 2020 pkill-9
;;; Copyright © 2020, 2021 ison <ison@airmail.cc>
;;; Copyright © 2021 pineapples
;;; Copyright © 2021 Jean-Baptiste Volatier <jbv@pm.me>
;;; Copyright © 2021 Kozo <kozodev@runbox.com>
;;; Copyright © 2021 John Kehayias <john.kehayias@protonmail.com>
;;;
;;; This file is not part of GNU Guix.
;;;
;;; GNU Guix is free software; you can redistribute it and/or modify it
;;; under the terms of the GNU General Public License as published by
;;; the Free Software Foundation; either version 3 of the License, or (at
;;; your option) any later version.
;;;
;;; GNU Guix is distributed in the hope that it will be useful, but
;;; WITHOUT ANY WARRANTY; without even the implied warranty of
;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
;;; GNU General Public License for more details.
;;;
;;; You should have received a copy of the GNU General Public License
;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.

(use-modules
 (guix build-system trivial)
 (guix packages)
 (gnu packages fontutils)
 (gnu packages apr))

(define* (fhs-union inputs #:key (name "fhs-union") (version "0.0") (system "x86_64-linux"))
  "Create a package housing the union of inputs."
  (package
    (name name)
    (version version)
    (source #f)
    (inputs inputs)
    (build-system trivial-build-system)
    (arguments
     `(#:system ,system
       #:modules ((guix build union))
       #:builder
       (begin
         (use-modules (ice-9 match)
                      (guix build union))
         (match %build-inputs
           (((_ . directories) ...)
            (union-build (assoc-ref %outputs "out")
                         directories)
            (format #t "\n directories: ~a\n" directories))))))
    (home-page #f)
    (synopsis "Libraries used for FHS")
    (description "Libraries needed to build a guix container FHS.")
    (license #f)))

(define fontconfig-fixed
  (package
    (inherit fontconfig)
    (propagated-inputs
     (modify-inputs (package-propagated-inputs fontconfig)
       (replace "expat" (@@ (gnu packages xml) expat/fixed))))))

(define (container-package pkg-list)
  (let* ((union64 (fhs-union pkg-list #:name "fhs-union-64"))
         (union32 (fhs-union pkg-list #:name "fhs-union-32" #:system "i686-linux")))
    (package
      (name "test-pkg")
      (version "0.0")
      (source #f)
      (inputs `(("fhs-union-64" ,union64)
                ("fhs-union-32" ,union32)))
      (build-system trivial-build-system)
      (arguments
       `(#:modules ((guix build utils))
         #:builder
         (begin
           (use-modules (guix build utils))
           (let* ((out (assoc-ref %outputs "out"))
                  (lib64-target (assoc-ref %build-inputs "fhs-union-64"))
                  (lib64-dest (string-append out "/lib64"))
                  (lib32-target (assoc-ref %build-inputs "fhs-union-32"))
                  (lib32-dest (string-append out "/lib32")))
             (format #t "\nunion64: ~a\nunion32: ~a\n" lib64-target lib32-target)
             (mkdir-p (string-append out "/bin")) ; dummy dir needed to build
             (symlink lib64-target lib64-dest)
             (symlink lib32-target lib32-dest)))))
      (home-page #f)
      (synopsis #f)
      (description #f)
      (license #f))))

;(fhs-union (list fontconfig-fixed))
;(fhs-union (list fontconfig) #:system "i686-linux")
;(fhs-union (list fontconfig))
;(fhs-union (list apr-util))

(container-package `(("apr-util" ,apr-util)))