bug#54350: Profile collisions are ignored, installing multiple versions of the same package is silently broken

bug#54350: Profile collisions are ignored, installing multiple versions of the same package is silently broken

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 671 bytes --]

Hi guix,

Try

$ guix shell --pure bash pango@1.42 pango@1.48
$ ~/.guix-profile/bin/ls -l $GUIX_ENVIRONMENT/lib/libpango*.so

Result:
[...]/lib/libpango-1.0.so -> [...]-pango-1.42.4/lib/libpango-1.0.so
[...]/lib/libpangocairo-1.0.so -> [...]-pango-1.42.4/lib/libpangocairo-1.0.so
[...]/lib/libpangoft2-1.0.so -> [...]-1.42.4/lib/libpangoft2-1.0.so
[...]/lib/libpangoxft-1.0.so -> [...]-pango-1.42.4/lib/libpangoxft-1.0.so

Question: how does this even work?  Aren't error messages about profile
collisions a thing?  Another question: if this use case was supported,
shouldn't it point to the newer pango instead of the older pango?

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

bug#54350: Profile collisions are ignored, installing multiple versions of the same package is silently broken

[Liliana Marie Prikler]

Am Samstag, dem 12.03.2022 um 10:45 +0100 schrieb Maxime Devos:
> Hi guix,
> 
> Try
> 
> $ guix shell --pure bash pango@1.42 pango@1.48
> $ ~/.guix-profile/bin/ls -l $GUIX_ENVIRONMENT/lib/libpango*.so
> 
> Result:
> [...]/lib/libpango-1.0.so -> [...]-pango-1.42.4/lib/libpango-1.0.so
> [...]/lib/libpangocairo-1.0.so -> [...]-pango-
> 1.42.4/lib/libpangocairo-1.0.so
> [...]/lib/libpangoft2-1.0.so -> [...]-1.42.4/lib/libpangoft2-1.0.so
> [...]/lib/libpangoxft-1.0.so -> [...]-pango-1.42.4/lib/libpangoxft-
> 1.0.so
> 
> Question: how does this even work?  Aren't error messages about
> profile collisions a thing? 
They ought to be, but perhaps these are only checked for propagated
packages, not for explicitly specified ones.  Would still be weird, if
you e.g. put pango@1.48 and something propagating pango@1.42 together,
it ought to error.  Perhaps an oversight? 

> Another question: if this use case was supported, shouldn't it point
> to the newer pango instead of the older pango?
union-build implicitly takes the first file it finds -- this is
actually sane, but sadly undocumented behaviour.  In other words, if
you specify pango@1.48 first, it should be the one being picked.

Cheers

bug#54350: Profile collisions are ignored, installing multiple versions of the same package is silently broken

[Ludovic Courtès]

Hi,

Maxime Devos <maximedevos@telenet.be> skribis:

> $ guix shell --pure bash pango@1.42 pango@1.48
> $ ~/.guix-profile/bin/ls -l $GUIX_ENVIRONMENT/lib/libpango*.so
>
> Result:
> [...]/lib/libpango-1.0.so -> [...]-pango-1.42.4/lib/libpango-1.0.so
> [...]/lib/libpangocairo-1.0.so -> [...]-pango-1.42.4/lib/libpangocairo-1.0.so
> [...]/lib/libpangoft2-1.0.so -> [...]-1.42.4/lib/libpangoft2-1.0.so
> [...]/lib/libpangoxft-1.0.so -> [...]-pango-1.42.4/lib/libpangoxft-1.0.so
>
> Question: how does this even work?  Aren't error messages about profile
> collisions a thing?

Unlike ‘guix package’, ‘guix environment/shell’ explicitly disables
collision checks—see #:allow-collisions? in environment.scm.

The rationale back then was that if you do:

  guix shell -D guile

then you likely end up with profile collisions, like so (I modified the
code to do #:allow-collisions? #f):

--8<---------------cut here---------------start------------->8---
$ ./pre-inst-env guix shell -D guile -n
guix shell: error: profile contains conflicting entries for bash-minimal
guix shell: error:   first entry: bash-minimal@5.1.8 /gnu/store/chfwin3a4qp1znnpsjbmydr2jbzk0d6y-bash-minimal-5.1.8
guix shell: error:   second entry: bash-minimal@5.1.8 /gnu/store/4y5m9lb8k3qkb1y9m02sw9w9a6hacd16-bash-minimal-5.1.8
hint: You cannot have two different versions or variants of `bash-minimal' in the same profile.
--8<---------------cut here---------------end--------------->8---

These collisions are harmless but frequent in package development
environments, which is why ‘guix environment’ had to turn off those
checks (commit afd06f605bf88a796acefc7ed598b43879346a6b).

That’s a bit of a hack though.

What we could do, as a mitigation, is to enable collision checks unless
there’s at least one ‘-D’ flag.  We’d also need to introduce
‘--allow-collisions’ for ‘guix shell’.

WDYT?

Thanks,
Ludo’.

bug#54350: Profile collisions are ignored, installing multiple versions of the same package is silently broken

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 2048 bytes --]

Ludovic Courtès schreef op di 15-03-2022 om 14:50 [+0100]:
> $ ./pre-inst-env guix shell -D guile -n
> guix shell: error: profile contains conflicting entries for bash-minimal
> guix shell: error:   first entry: bash-minimal@5.1.8 /gnu/store/chfwin3a4qp1znnpsjbmydr2jbzk0d6y-bash-minimal-5.1.8
> guix shell: error:   second entry: bash-minimal@5.1.8 /gnu/store/4y5m9lb8k3qkb1y9m02sw9w9a6hacd16-bash-minimal-5.1.8
> hint: You cannot have two different versions or variants of `bash-minimal' in the same profile.
> --8<---------------cut here---------------end--------------->8---
> 
> These collisions are harmless but frequent in package development
> environments, which is why ‘guix environment’ had to turn off those
> checks (commit afd06f605bf88a796acefc7ed598b43879346a6b).
> 
> That’s a bit of a hack though.

These collisions happen because the bag of a package contains of multiple
variants of bash: the default canonical bash-minimal, from (gnu packages
commencement), not directly visible (i.e., with "guix show" and
'specification->package'), which is overidden by the bash-minimal from
native-inputs, which is visible from specification->package and as a public
variable, and as I understand it, the bash-minimal supposed to end up
in the references of non-bootstrap packages.

For these kind of collisions, I think the non-canonical package (that's
actually visible with "guix show FOO") is preferred here, so perhaps
the canonical variants can be filtered out whenever that would resolve
a collision?

That seems also a bit of a hack though.

> What we could do, as a mitigation, is to enable collision checks
> unless there’s at least one ‘-D’ flag.  We’d also need to introduce
> ‘--allow-collisions’ for ‘guix shell’.

Special-casing -D like this also seems like a hack to me.  I'm not sure
which hack is better, though I prefer this one and the ‘filter out
canonical packages’ above the original ‘just disable collision
checking’.

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

bug#54350: Profile collisions are ignored, installing multiple versions of the same package is silently broken

[Ludovic Courtès]

Hi,

Maxime Devos <maximedevos@telenet.be> skribis:

> Ludovic Courtès schreef op di 15-03-2022 om 14:50 [+0100]:
>> $ ./pre-inst-env guix shell -D guile -n
>> guix shell: error: profile contains conflicting entries for bash-minimal
>> guix shell: error:   first entry: bash-minimal@5.1.8 /gnu/store/chfwin3a4qp1znnpsjbmydr2jbzk0d6y-bash-minimal-5.1.8
>> guix shell: error:   second entry: bash-minimal@5.1.8 /gnu/store/4y5m9lb8k3qkb1y9m02sw9w9a6hacd16-bash-minimal-5.1.8
>> hint: You cannot have two different versions or variants of `bash-minimal' in the same profile.
>> --8<---------------cut here---------------end--------------->8---

I wonder whether this is the exception rather than the rule.  Here’s an
unscientific not-quite-random sample:

--8<---------------cut here---------------start------------->8---
$ ./pre-inst-env guix shell -D guix
guix shell: error: profile contains conflicting entries for bzip2
guix shell: error:   first entry: bzip2@1.0.8 /gnu/store/gzmwrl6cb8nxmc68hpgqzblng2kamkgc-bzip2-1.0.8
guix shell: error:   second entry: bzip2@1.0.8 /gnu/store/s3hl12jxz9ybs7nsy7kq7ybzz7qnzmsg-bzip2-1.0.8
hint: You cannot have two different versions or variants of `bzip2' in the same profile.

$ ./pre-inst-env guix shell -D inkscape --no-grafts -n
guix shell: error: profile contains conflicting entries for libsigc++
guix shell: error:   first entry: libsigc++@3.0.6 /gnu/store/ng4k2yl94d758p5vnashd4nvyb1aw8s1-libsigc++-3.0.6
guix shell: error:    ... propagated from cairomm@1.14.2
guix shell: error:    ... propagated from gtkmm@3.24.5
guix shell: error:   second entry: libsigc++@2.9.3 /gnu/store/iad8jg1fm7jsq0pqj547f3n5s2jn9rp0-libsigc++-2.9.3
hint: Try upgrading both `gtkmm' and `libsigc++', or remove one of them from the profile.

$ ./pre-inst-env guix shell -D gimp --no-grafts -n
guix shell: error: profile contains conflicting entries for librsvg
guix shell: error:   first entry: librsvg@2.50.7 /gnu/store/wj3sigq6pwjc2z68qgbpv8pxcv8haklf-librsvg-2.50.7
guix shell: error:   second entry: librsvg@2.50.7 /gnu/store/q3vymxinnp6cxqq1cpz1rdqmnwq5x1ni-librsvg-2.50.7
hint: You cannot have two different versions or variants of `librsvg' in the same profile.

$ ./pre-inst-env guix shell -D openmpi --no-grafts -n
The following derivation would be built:
   /gnu/store/fpj74z159bf6v7cxsxs5rxm3xm4w2amz-profile.drv

$ ./pre-inst-env guix shell -D hwloc --no-grafts -n
The following derivation would be built:
   /gnu/store/ikf40sfmk3y7r3frxm9fbkzmma4ym8ay-profile.drv

$ ./pre-inst-env guix shell -D guile-gcrypt --no-grafts -n
The following derivation would be built:
   /gnu/store/3ip5kq50b97bsm6zhygz0fjx54w0fqa0-profile.drv

$ ./pre-inst-env guix shell -D libgit2 --no-grafts -n
The following derivation would be built:
   /gnu/store/8f83l9p88g40mrnl5l4wrzqg5zwgn6cm-profile.drv

$ ./pre-inst-env guix shell -D python-pytorch --no-grafts -n
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 100.0%
The following derivation would be built:
   /gnu/store/aly6xd3vi5j0qimjzj2hj0wry5zma376-profile.drv

37.2 MB would be downloaded
$ ./pre-inst-env guix shell -D python-itsdangerous --no-grafts -n
The following derivation would be built:
   /gnu/store/cgxgvj73wgx263lzd3f82vvjl39vn4pn-profile.drv

--8<---------------cut here---------------end--------------->8---

We should check automatically the fraction of packages for which
‘package->development-manifest’ contains colliding entries.

If it’s rather rare, we might as well have ‘guix shell’ default to
#:allow-collisions? #f.

(The issue for GIMP comes from librsvg vs. librsvg-bootstrap.  The one
for inkscape is a genuine problem.)

If we take that route, we could also add a linter that checks for
collisions in package dependencies.

>> These collisions are harmless but frequent in package development
>> environments, which is why ‘guix environment’ had to turn off those
>> checks (commit afd06f605bf88a796acefc7ed598b43879346a6b).
>> 
>> That’s a bit of a hack though.
>
> These collisions happen because the bag of a package contains of multiple
> variants of bash: the default canonical bash-minimal, from (gnu packages
> commencement), not directly visible (i.e., with "guix show" and
> 'specification->package'), which is overidden by the bash-minimal from
> native-inputs, which is visible from specification->package and as a public
> variable, and as I understand it, the bash-minimal supposed to end up
> in the references of non-bootstrap packages.
>
> For these kind of collisions, I think the non-canonical package (that's
> actually visible with "guix show FOO") is preferred here, so perhaps
> the canonical variants can be filtered out whenever that would resolve
> a collision?

I’m not fond of filtering out, because that can make it hard to
understand what the inputs of a package really are.

However, we could tweak the collision detection code to ignore
collisions with hidden packages or something along these lines.

Thoughts?

Ludo’.

bug#54350: Profile collisions are ignored, installing multiple versions of the same package is silently broken

[Greg Hogan]

[-- Attachment #1: Type: text/plain, Size: 295 bytes --]

On Tue, Mar 15, 2022 at 9:51 AM Ludovic Courtès <ludo@gnu.org> wrote:

> What we could do, as a mitigation, is to enable collision checks unless
> there’s at least one ‘-D’ flag.  We’d also need to introduce
> ‘--allow-collisions’ for ‘guix shell’.
>

Also 'guix pack'.

[-- Attachment #2: Type: text/html, Size: 576 bytes --]

bug#54350: Profile collisions in "guix shell"

[Konrad Hinsen]

Hi everyone,

I ran into this issue when I was trying to turn a much-used shell
environment into a profile for persisting the binaries in the store.
I had been using it for several months, believing it to be OK.
Fortunately it was easy to fix.

Background: as part of a reproducible computation workflow (see my talk
at the ten-year meeting,
https://10years.guix.gnu.org/program/#guix-as-a-tool-for-computational-science),
I run everything that is part of a research project in containers,
meaning "guix shell -C". Since I rely on Python 2 software, I end up
using package transformations frequently to get old software to work. As
it turns out now, package transformations are also an excellent way to
create profile collisions: three out of five of my commonly used
containers have collisions, which I wasn't aware of until a few days
ago.

From that perspective, the current behavior of "guix shell" is... let's
say "suboptimal".

I do understand the motivation, as explained in this thread, but I don't
quite understand (1) why profile collisions are so frequent in
development environments and (2) why ignoring them in that use case
doesn't cause any trouble.

When I run "guix shell -D inkscape", I expect to have, as much as
possible, the same environment as the one the daemon uses when build
inkscape. Does the daemon also switch off collision detection?

Cheers,
  Konrad.

bug#54350: Profile collisions in "guix shell"

[Konrad Hinsen]

Konrad Hinsen <konrad.hinsen@fastmail.net> writes:

> I run everything that is part of a research project in containers,
> meaning "guix shell -C".

Related: HPC users on clusters using NFS are also advised to use
"guix shell" rather than profiles:

  https://lists.gnu.org/archive/html/guix-science/2023-12/msg00007.html
  https://lists.gnu.org/archive/html/guix-science/2023-12/msg00008.html

Cheers,
  Konrad.

bug#54350: Profile collisions in "guix shell"

[Suhail]

Hello Guix,

I came across this issue the other day when attempting to create a
profile that could act as a development environment for Guix. In
addition to noting my observations below, I'd like to understand why
there are collisions in a *new* profile tasked with installing the
dependencies of guix?  Is this a bug?

The [documentation] notes the below command to start "a new shell where
all the dependencies and appropriate environment variables are set up to
hack on Guix":

#+begin_src sh
  guix shell -D guix -CPW
#+end_src

Using guix-shell to export the manifest corresponding to the above one
gets:

#+header: :tangle ./manifest.scm
#+begin_src scheme
  ;; What follows is a "manifest" equivalent to the command line you gave.
  ;; You can store it in a file that you may then pass to any 'guix' command
  ;; that accepts a '--manifest' (or '-m') option.
  (package->development-manifest
    (specification->package "guix"))
#+end_src

Saving the above to manifest.scm and evaluating the below:

#+begin_src sh
  mkdir -p ${GUIX_EXTRA_PROFILES}/guix-build-2
  guix package -p ${GUIX_EXTRA_PROFILES}/guix-build-2/guix-build-2 -m ./manifest.scm
#+end_src

Results in the following error:

#+begin_example
  ...
  guix package: error: profile contains conflicting entries for bzip2
  guix package: error:   first entry: bzip2@1.0.8 /gnu/store/4jkvfh9hp9zf9plrcw490ci6vpvc033d-bzip2-1.0.8
  guix package: error:   second entry: bzip2@1.0.8 /gnu/store/0ngy7fmrp9g3q35wwccnjxzaaqq9f8wi-bzip2-1.0.8
  hint: You cannot have two different versions or variants of `bzip2' in the same profile.
#+end_example

Adding --allow-collisions to match guix-shell behaviour makes the
installation succeed, as expected.

[documentation]: <https://guix.gnu.org/en/manual/devel/en/guix.html#Building-from-Git>

-- 
Suhail

This email is not an offer capable of acceptance, does not evidence an
intention to enter into an agreement, has no operative effect until a
definitive agreement is signed in writing by both parties, and that no
party should act in reliance on the email or any representations of the
sender until a definitive agreement is signed in writing by both
parties.