bug#46569: python can't find internal pip modules in environment from manifest

bug#46569: python can't find internal pip modules in environment from manifest

[Jelle Licht]

Python can not load all pip-related python modules when loaded from a
manifest file, yet seems to work fine when loaded 'directly' as part of
a `guix environment'-invocation.


Provided that we have a file in `hi.py':
--8<---------------cut here---------------start------------->8---
import pip
print("halfwaypoint")
import pip._internal.index.package_finder
print("I made it!")
--8<---------------cut here---------------end--------------->8---

and a `manifest.scm' with:

--8<---------------cut here---------------start------------->8---
(use-modules (gnu) (guix packages)
	     (gnu packages python)
	     (gnu packages python-xyz))

(packages->manifest (list python python-pip))
--8<---------------cut here---------------end--------------->8---

I can get the expected output by running:
`guix environment --pure --ad-hoc python python-pip -- python3 hi.py' =>
> halfwaypoint
> I made it!

Yet, when I run the following:
`guix environment --pure -m manifest.scm -- python3 hi.py' =>

--8<---------------cut here---------------start------------->8---
halfwaypoint
Traceback (most recent call last):
  File "hi.py", line 4, in <module>
    import pip._internal.index.package_finder
  File "/gnu/store/4m5vhlq61qnj36rq60l83xcmgj3mx92j-profile/lib/python3.8/site-packages/pip/_internal/__init__.py", line 40, in <module>
    from pip._internal.cli.autocompletion import autocomplete
  File "/gnu/store/4m5vhlq61qnj36rq60l83xcmgj3mx92j-profile/lib/python3.8/site-packages/pip/_internal/cli/autocompletion.py", line 8, in <module>
    from pip._internal.cli.main_parser import create_main_parser
  File "/gnu/store/4m5vhlq61qnj36rq60l83xcmgj3mx92j-profile/lib/python3.8/site-packages/pip/_internal/cli/main_parser.py", line 11, in <module>
    from pip._internal.commands import (
  File "/gnu/store/4m5vhlq61qnj36rq60l83xcmgj3mx92j-profile/lib/python3.8/site-packages/pip/_internal/commands/__init__.py", line 6, in <module>
    from pip._internal.commands.completion import CompletionCommand
  File "/gnu/store/4m5vhlq61qnj36rq60l83xcmgj3mx92j-profile/lib/python3.8/site-packages/pip/_internal/commands/completion.py", line 6, in <module>
    from pip._internal.cli.base_command import Command
  File "/gnu/store/4m5vhlq61qnj36rq60l83xcmgj3mx92j-profile/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 26, in <module>
    from pip._internal.index import PackageFinder
ImportError: cannot import name 'PackageFinder' from 'pip._internal.index' (/gnu/store/4m5vhlq61qnj36rq60l83xcmgj3mx92j-profile/lib/python3.8/site-packages/pip/_internal/index/__init__.py)
--8<---------------cut here---------------end--------------->8---

Why is there a difference in the first place? Shouldn't both approaches work?

- Jelle

bug#46569: python can't find internal pip modules in environment from manifest

[Jelle Licht]

Jelle Licht <jlicht@fsfe.org> writes:

> Python can not load all pip-related python modules when loaded from a
> manifest file, yet seems to work fine when loaded 'directly' as part of
> a `guix environment'-invocation.
>
>
> Provided that we have a file in `hi.py':
> --8<---------------cut here---------------start------------->8---
> import pip
> print("halfwaypoint")
> import pip._internal.index.package_finder
> print("I made it!")
> --8<---------------cut here---------------end--------------->8---
>
> and a `manifest.scm' with:
>
> --8<---------------cut here---------------start------------->8---
> (use-modules (gnu) (guix packages)
> 	     (gnu packages python)
> 	     (gnu packages python-xyz))
>
> (packages->manifest (list python python-pip))
> --8<---------------cut here---------------end--------------->8---
>
> I can get the expected output by running:
> `guix environment --pure --ad-hoc python python-pip -- python3 hi.py' =>
>> halfwaypoint
>> I made it!
>
> Yet, when I run the following:
> `guix environment --pure -m manifest.scm -- python3 hi.py' =>
>
> --8<---------------cut here---------------start------------->8---
> halfwaypoint
> Traceback (most recent call last):
>   File "hi.py", line 4, in <module>
>     import pip._internal.index.package_finder
>   File "/gnu/store/4m5vhlq61qnj36rq60l83xcmgj3mx92j-profile/lib/python3.8/site-packages/pip/_internal/__init__.py", line 40, in <module>
>     from pip._internal.cli.autocompletion import autocomplete
>   File "/gnu/store/4m5vhlq61qnj36rq60l83xcmgj3mx92j-profile/lib/python3.8/site-packages/pip/_internal/cli/autocompletion.py", line 8, in <module>
>     from pip._internal.cli.main_parser import create_main_parser
>   File "/gnu/store/4m5vhlq61qnj36rq60l83xcmgj3mx92j-profile/lib/python3.8/site-packages/pip/_internal/cli/main_parser.py", line 11, in <module>
>     from pip._internal.commands import (
>   File "/gnu/store/4m5vhlq61qnj36rq60l83xcmgj3mx92j-profile/lib/python3.8/site-packages/pip/_internal/commands/__init__.py", line 6, in <module>
>     from pip._internal.commands.completion import CompletionCommand
>   File "/gnu/store/4m5vhlq61qnj36rq60l83xcmgj3mx92j-profile/lib/python3.8/site-packages/pip/_internal/commands/completion.py", line 6, in <module>
>     from pip._internal.cli.base_command import Command
>   File "/gnu/store/4m5vhlq61qnj36rq60l83xcmgj3mx92j-profile/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 26, in <module>
>     from pip._internal.index import PackageFinder
> ImportError: cannot import name 'PackageFinder' from 'pip._internal.index' (/gnu/store/4m5vhlq61qnj36rq60l83xcmgj3mx92j-profile/lib/python3.8/site-packages/pip/_internal/index/__init__.py)
> --8<---------------cut here---------------end--------------->8---
>
> Why is there a difference in the first place? Shouldn't both approaches work?

Using diffoscope, I found the (practical) difference:

$GUIX_ENVIRONMENT/bin/pip3 is
│ │ -destination: /gnu/store/rz42ba0my9vrgbkjpkzr2drmnjk5ah50-python-3.8.2/bin/pip3
│ │ +destination: /gnu/store/d75bwzbla5ybhs0mdw80qy94mawnhhsw-python-pip-20.2.4/bin/pip3

If you make sure that `python-pip' precedes the `python' package in the
manifest, things work out fine. Likewise for the following --ad-hoc
example:

This fails:
guix environment --ad-hoc python-pip python -- python3 hi.py

This works:
guix environment --ad-hoc python python-pip -- python3 hi.py

So it seems the order in which guix sees packages when building profiles
determines how collisions are resolved, good to know :-).

Is the python-included pip package supposed to be broken like this, though?
 - Jelle

bug#46569: python can't find internal pip modules in environment from manifest

[Jelle Licht]

Jelle Licht <jlicht@fsfe.org> writes:

> Jelle Licht <jlicht@fsfe.org> writes:
> This fails:
> guix environment --ad-hoc python-pip python -- python3 hi.py
>
> This works:
> guix environment --ad-hoc python python-pip -- python3 hi.py

This should be the other way around, pasting error on my end.

bug#46569: python can't find internal pip modules in environment from manifest

[Maxim Cournoyer]

Hi Jelle,

Jelle Licht <jlicht@fsfe.org> writes:

> Python can not load all pip-related python modules when loaded from a
> manifest file, yet seems to work fine when loaded 'directly' as part of
> a `guix environment'-invocation.
>
>
> Provided that we have a file in `hi.py':
>
> import pip
> print("halfwaypoint")
> import pip._internal.index.package_finder
> print("I made it!")
>
>
> and a `manifest.scm' with:
>
> (use-modules (gnu) (guix packages)
> 	     (gnu packages python)
> 	     (gnu packages python-xyz))
>
> (packages->manifest (list python python-pip))
>
>
> I can get the expected output by running:
> `guix environment --pure --ad-hoc python python-pip -- python3 hi.py' =>
>> halfwaypoint
>> I made it!

It seems that two pip versions don't cohabit very well.  The same
happens if you install pip via 'pip install --user pip' (e.g., to
~/.local/bin) and try to use it.

You could try using the pip version that comes bundled with Python
itself (e.g, not installing python-pip in the same profile); if it's the
only one, it should work.

HTH,

Maxim

bug#46569: python can't find internal pip modules in environment from manifest

[Jelle Licht]

Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:

> Hi Jelle,
>
> Jelle Licht <jlicht@fsfe.org> writes:
>
>> Python can not load all pip-related python modules when loaded from a
>> manifest file, yet seems to work fine when loaded 'directly' as part of
>> a `guix environment'-invocation.
>>
>>
>> Provided that we have a file in `hi.py':
>>
>> import pip
>> print("halfwaypoint")
>> import pip._internal.index.package_finder
>> print("I made it!")
>>
>>
>> and a `manifest.scm' with:
>>
>> (use-modules (gnu) (guix packages)
>> 	     (gnu packages python)
>> 	     (gnu packages python-xyz))
>>
>> (packages->manifest (list python python-pip))
>>
>>
>> I can get the expected output by running:
>> `guix environment --pure --ad-hoc python python-pip -- python3 hi.py' =>
>>> halfwaypoint
>>> I made it!
>
> It seems that two pip versions don't cohabit very well.  The same
> happens if you install pip via 'pip install --user pip' (e.g., to
> ~/.local/bin) and try to use it.
>
> You could try using the pip version that comes bundled with Python
> itself (e.g, not installing python-pip in the same profile); if it's the
> only one, it should work.

It doesn't, though, with the exact same error message as above.

Does my example (hi.py) work for you?

bug#46569: python can't find internal pip modules in environment from manifest

[Maxim Cournoyer]

Hello Jelle,

Jelle Licht <jlicht@fsfe.org> writes:

> Python can not load all pip-related python modules when loaded from a
> manifest file, yet seems to work fine when loaded 'directly' as part of
> a `guix environment'-invocation.
>
>
> Provided that we have a file in `hi.py':
>
> import pip
> print("halfwaypoint")
> import pip._internal.index.package_finder
> print("I made it!")

The problem seems to be related to the version of pip used.  The one
bundled with our Python on master is:

$ guix environment --pure --ad-hoc python -- pip3 --version
pip 19.2.3 from
/gnu/store/pw65c67dmzr2m9wbkj5mskhm6ljk6xal-profile/lib/python3.8/site-packages/pip
(python 3.8)

and

$ guix environment --pure --ad-hoc python -- python3 -c 'import pip._internal.index.package_finder'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named
'pip._internal.index.package_finder'; 'pip._internal.index' is not a
package

but

$ guix environment --pure --ad-hoc python python-pip -- pip3 --version
pip 20.2.4 from
/gnu/store/d75bwzbla5ybhs0mdw80qy94mawnhhsw-python-pip-20.2.4/lib/python3.8/site-packages/pip
(python 3.8)

and

$ guix environment --pure --ad-hoc python python-pip -- python3 -c 'import pip._internal.index.package_finder'
$ echo $?
0

HTH!

Closing.

Maxim