Two copies of same library in store after cleaning generations and gc

Two copies of same library in store after cleaning generations and gc

[Mike Jones]

Hi Guix!

I've been running Guix System for a few weeks now, and I've noticed that
I have two copies of some libraries in my store (e.g. mesa, gtk) even
after updating all packages, deleting old profile generations, and
running gc.

This isn't a problem, as I have plenty of disk space, but I'm just
curious to understand why this happens.

For example, I have both

  /gnu/store/lcqz4q3834bjd3dlc8zsr95mvzz9n006-mesa-21.3.8
  /gnu/store/wj1nipayc74yalfmdg7wwimkfc1macbl-mesa-21.3.8

Using 'guix gc --referrers', I can see that:

- The first copy is referred to by all applications in my profile that
  link against mesa, e.g. icecat.

- The second copy is referred to directly by my profile. Indeed, I can
  see that, for example, ~/.guix-profile/lib/libGL.so is a symlink to
  /gnu/store/wj1nipayc74yalfmdg7wwimkfc1macbl-mesa-21.3.8/lib/libGL.so.

So my question boils down to: why is the copy of mesa in
~/.guix-profile/lib different to the one to which programs in the same
profile are linked? And is there any guix command I can use to inspect
exactly how/why these copies differ?

Thanks!
- Mike

Re: Two copies of same library in store after cleaning generations and gc

[Hartmut Goebel]

Am 14.08.22 um 13:31 schrieb Mike Jones:
> So my question boils down to: why is the copy of mesa in
> ~/.guix-profile/lib different to the one to which programs in the same
> profile are linked? And is there any guix command I can use to inspect
> exactly how/why these copies differ?

I assume you installed mesa into your profile — or some other package 
installed in your profile has mesa as „propagated input“.

Try „guix package -l“ to see whether it is installed in your profile.

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          |h.goebel@crazy-compilers.com                |
|www.crazy-compilers.com  | compilers which you thought are impossible |

Re: Two copies of same library in store after cleaning generations and gc

[Mike Jones]

Hello,

Thanks for the hints.

> I assume you installed mesa into your profile — or some other package
> installed in your profile has mesa as „propagated input“.
>
> Try „guix package -l“ to see whether it is installed in your profile.

I haven't installed mesa explicitly, and it doesn't show up in `guix
package -l`, so I think you must be right that it's coming from a
propagated input.

I still wonder, though, why there are two different builds of mesa
appearing. I suspected it might be related to grafting, so I poked
around in /gnu/store and found the .drv files that produced these two
different copies. They both seem to be the result of a grafting process.

If I rebuild my profile with --no-grafts, the mesa used by programs and
the one in the profile are the same, so I'll just put this down to some
weirdness with grafting.

- Mike

Re: Two copies of same library in store after cleaning generations and gc

[Ricardo Wurmus]

"Mike Jones" <mike@mjones.io> writes:

> If I rebuild my profile with --no-grafts, the mesa used by programs and
> the one in the profile are the same, so I'll just put this down to some
> weirdness with grafting.

This sounds like grafts working as intended.  Do you also have more than
one instance of icecat – one using the graft and one without?

(Due to deduplication in /gnu/store this is not as wasteful as it may
seem.)

-- 
Ricardo

Re: Two copies of same library in store after cleaning generations and gc

[zimoun]

Hi,

On ven., 19 août 2022 at 08:13, Ricardo Wurmus <rekado@elephly.net> wrote:

> This sounds like grafts working as intended.  Do you also have more than
> one instance of icecat – one using the graft and one without?
>
> (Due to deduplication in /gnu/store this is not as wasteful as it may
> seem.)

Using Guix 8f0d45c,

--8<---------------cut here---------------start------------->8---
$ guix build mesa
/gnu/store/5y054ijr0dqpd6v4wlby9alhbzmha033-mesa-21.3.8-bin
/gnu/store/74ags388lb0xasf7v1h8j9ywj8zrsgv9-mesa-21.3.8

$ guix build mesa --no-grafts
/gnu/store/znvm7i9rgaviwi2841j8gimanif99wl1-mesa-21.3.8-bin
/gnu/store/wdm2s2si8fqsrcd5xpc29ivmpkf20s8d-mesa-21.3.8

$ guix gc --references $(guix build icecat) | grep mesa
/gnu/store/lcqz4q3834bjd3dlc8zsr95mvzz9n006-mesa-21.3.8
--8<---------------cut here---------------end--------------->8---

it can be confusing…  Let examine. :-)



First, let consider mesa referred by icecat.

--8<---------------cut here---------------start------------->8---
$ guix gc --derivers /gnu/store/lcqz4q3834bjd3dlc8zsr95mvzz9n006-mesa-21.3.8
/gnu/store/qqa192m2cx4qrlvhr1dg7vrc49jqa84h-mesa-21.3.8.drv
/gnu/store/y2yxw4zd4p916lby3sibmskdv860zvpa-mesa-21.3.8.drv
--8<---------------cut here---------------end--------------->8---

It means that the both derivations,

        /gnu/store/qqa192m2cx4qrlvhr1dg7vrc49jqa84h-mesa-21.3.8.drv
        /gnu/store/y2yxw4zd4p916lby3sibmskdv860zvpa-mesa-21.3.8.drv

have the same output:

        /gnu/store/lcqz4q3834bjd3dlc8zsr95mvzz9n006-mesa-21.3.8

Yes, that’s surprising but possible.


From /gnu/store/qqa192m2cx4qrlvhr1dg7vrc49jqa84h-mesa-21.3.8.drv, the
guile-builder reads:

--8<---------------cut here---------------start------------->8---
  (define %build-inputs
    (quote
     (("x" . "/gnu/store/wdm2s2si8fqsrcd5xpc29ivmpkf20s8d-mesa-21.3.8")
[...]
       ((old-outputs
         (quote
          (("out" . "/gnu/store/wdm2s2si8fqsrcd5xpc29ivmpkf20s8d-mesa-21.3.8"))))
--8<---------------cut here---------------end--------------->8---

which grafts, I guess.  Note that this derivation lists the other
derivation /gnu/store/0yd0gxbcrkh32zy4q8dkpyyv2xpvm1mh-mesa-21.3.8.drv.

From /gnu/store/y2yxw4zd4p916lby3sibmskdv860zvpa-mesa-21.3.8.drv, the
guile-builder is exactly the same (/gnu/store/a0h343…-mesa-21.3.8-guile-builder).
However, the derivation lists
/gnu/store/ylx0p9sl6qla4qsmp6dsvaqgd7g3m1rw-mesa-21.3.8.drv.

These two derivations,

        /gnu/store/0yd0gxbcrkh32zy4q8dkpyyv2xpvm1mh-mesa-21.3.8.drv
        /gnu/store/ylx0p9sl6qla4qsmp6dsvaqgd7g3m1rw-mesa-21.3.8.drv.

have the same outputs; which reads:

--8<---------------cut here---------------start------------->8---
Derive
([("bin","/gnu/store/znvm7i9rgaviwi2841j8gimanif99wl1-mesa-21.3.8-bin","","")
  ,("out","/gnu/store/wdm2s2si8fqsrcd5xpc29ivmpkf20s8d-mesa-21.3.8","","")]
--8<---------------cut here---------------end--------------->8---

other said, the ungrafted version.  So far, so good!


Second, let consider the grafted build,

--8<---------------cut here---------------start------------->8---
$ guix gc --derivers /gnu/store/74ags388lb0xasf7v1h8j9ywj8zrsgv9-mesa-21.3.8
/gnu/store/ji6h6y5jizwhsx2jzw67myk5fhiv87va-mesa-21.3.8.drv
--8<---------------cut here---------------end--------------->8---

and the guile-builder (/gnu/store/j4j58b…-mesa-21.3.8-guile-builder)
contains,

--8<---------------cut here---------------start------------->8---
  (define %build-inputs
    (quote
     (("x" . "/gnu/store/znvm7i9rgaviwi2841j8gimanif99wl1-mesa-21.3.8-bin")
      ("x" . "/gnu/store/wdm2s2si8fqsrcd5xpc29ivmpkf20s8d-mesa-21.3.8")
[...]
       ((old-outputs
         (quote
          (("bin" . "/gnu/store/znvm7i9rgaviwi2841j8gimanif99wl1-mesa-21.3.8-bin")
           ("out" . "/gnu/store/wdm2s2si8fqsrcd5xpc29ivmpkf20s8d-mesa-21.3.8"))))
--8<---------------cut here---------------end--------------->8---

and that’s why the hash 74ags388lb… (from “guix build mesa”) is
different from lcqz4q383… (from icecat).

This derivation /gnu/store/ji6h6y5jizwhsx2jzw67myk5fhiv87va-mesa-21.3.8.drv
lists the other derivation,
/gnu/store/0yd0gxbcrkh32zy4q8dkpyyv2xpvm1mh-mesa-21.3.8.drv.  Wait, it
has already appeared below. :-)


Well, all seems consistent, isn’t it?  I hope now it is less confusing.


However, something still appears to me confusing:

--8<---------------cut here---------------start------------->8---
$ diff -r --no-dereference /gnu/store/74ags388lb0xasf7v1h8j9ywj8zrsgv9-mesa-21.3.8 /gnu/store/lcqz4q3834bjd3dlc8zsr95mvzz9n006-mesa-21.3.8 
Binary files /gnu/store/74ags388lb0xasf7v1h8j9ywj8zrsgv9-mesa-21.3.8/lib/dri/i830_dri.so and /gnu/store/lcqz4q3834bjd3dlc8zsr95mvzz9n006-mesa-21.3.8/lib/dri/i830_dri.so differ

[...]

diff -r --no-dereference /gnu/store/74ags388lb0xasf7v1h8j9ywj8zrsgv9-mesa-21.3.8/lib/pkgconfig/dri.pc /gnu/store/lcqz4q3834bjd3dlc8zsr95mvzz9n006-mesa-21.3.8/lib/pkgconfig/dri.pc
1c1
< prefix=/gnu/store/74ags388lb0xasf7v1h8j9ywj8zrsgv9-mesa-21.3.8
---
> prefix=/gnu/store/lcqz4q3834bjd3dlc8zsr95mvzz9n006-mesa-21.3.8

[...]
--8<---------------cut here---------------end--------------->8---

Other said, the grafted item is duplicated.  Right?


Therefore, my question is: is deduplication optimal?  Or better worded:
what is the waste by the duplicated grafted items?


Cheers,
simon

Re: Two copies of same library in store after cleaning generations and gc

[Mike Jones]

Hi all!

On Fri, 19 Aug 2022, at 7:13 AM, Ricardo Wurmus wrote:
> This sounds like grafts working as intended.  Do you also have more
> than one instance of icecat – one using the graft and one without?

After building the profile do I indeed have two instances of icecat in
the store. But when I run `guix gc`, the ungrafted one is of course
deleted, and I'm left with a single instance.

With mesa, however, there are always two active instances of mesa (also
some other libraries, e.g. gtk) referenced by the profile. I expected
that there would only be one (grafted) instance.

On Fri, 19 Aug 2022, at 10:06 AM, zimoun wrote:
> Well, all seems consistent, isn’t it?  I hope now it is less
> confusing.

Yeah, thanks for working through this - I understand better now :) So we
have one original ungrafted output, which is:

  /gnu/store/wdm2s2si8fqsrcd5xpc29ivmpkf20s8d-mesa-21.3.8

This then gets separately grafted twice to produce two new outputs:

  /gnu/store/74ags388lb0xasf7v1h8j9ywj8zrsgv9-mesa-21.3.8 (result of `guix build mesa`)
  /gnu/store/lcqz4q3834bjd3dlc8zsr95mvzz9n006-mesa-21.3.8 (referred to by icecat)

But I wonder why icecat refers to a different mesa to `guix build
mesa`.

Running

  $ diffoscope /gnu/store/lcqz4q3834bjd3dlc8zsr95mvzz9n006-mesa-21.3.8 \
  /gnu/store/74ags388lb0xasf7v1h8j9ywj8zrsgv9-mesa-21.3.8`

(takes quite a long time to run!) shows me that many of the files are
different, but only in self-references.

I also diffed the guile-builder scripts for the two grafted mesa
instances. Here's the first difference:

  -    (map (lambda (o) (cons o (getenv o)))
  -         '("bin" "out")))
  +    (map (lambda (o) (cons o (getenv o))) '("out")))

It seems that one is a graft of just the default output, while the other
also includes the "bin" output! I think this explains why there are two
separate grafts, that produce the same (minus self-references) output.

Presumably these aren't deduplicated due to the differing
self-references?

- Mike

Re: Two copies of same library in store after cleaning generations and gc

[zimoun]

Hi,

On ven., 19 août 2022 at 12:37, "Mike Jones" <mike@mjones.io> wrote:

> Presumably these aren't deduplicated due to the differing
> self-references?

Yes.

Note that this happens for only the packages with multi-outputs impacted
by grafted ones.  If I am correct, using Guix 65cabb0, the number of
packages with multi-outputs is 681 (see below).

On this number, a subset will be impacted by the ~11 grafted packages.


Cheers,
simon

--8<---------------cut here---------------start------------->8---
scheme@(guix-user)> ,use(ice-9 match)
scheme@(guix-user)> ,use(gnu)
scheme@(guix-user)> ,use(guix)
scheme@(guix-user)> (length (fold-packages
  (lambda (package result)
    (match (package-outputs package)
      ((x y z ...) (cons package result))
      (_ result)))
  '()))

$1 = 681

scheme@(guix-user)> (length (fold-packages
  (lambda (package result)
    (if (package-replacement package)
      (cons package result)
      result))
  '()))

$2 = 11
--8<---------------cut here---------------end--------------->8---