unofficial mirror of help-guix@gnu.org 
 help / color / mirror / Atom feed
* Avoiding PYTHONPATH - latest?
@ 2020-11-30  0:22 Phil
  2020-11-30  9:41 ` Phil
  2020-11-30 13:05 ` zimoun
  0 siblings, 2 replies; 9+ messages in thread
From: Phil @ 2020-11-30  0:22 UTC (permalink / raw)
  To: help-guix

Hi all,

I've been having an argument with myself over the last 4 days about if
Guix's use of PYTHONPATH is a necessary evil or avoidable on a foreign OS.

I've found references to a similar discussion last year, and reference
to using a 'fake virtual environment' (does anyone have a reference to
the other thread referenced where fake venvs are demonstrated as not
working):

https://lists.gnu.org/archive/html/guix-devel/2019-06/msg00204.html
https://lists.gnu.org/archive/html/guix-devel/2019-06/msg00221.html

I had what I think is a similar idea over the weekend and tried it out.

Reading the mechanics of venvs as per PEP:
https://www.python.org/dev/peps/pep-0405

I decided to see if I added a fake pyvenv.cfg to the python.scm package
could allow for us to drop the use of PYTHONPATH completely.

Now the results are very preliminary, and I am essentially replacing
one hack with another (I'm not creating a venv after-all) - it does seem
to work just adding this to the python-3.8's phases: 

(add-after 'install 'add-pyvenv-cfg
           (lambda* (#:key outputs #:allow-other-keys)
                     ;; guix-daemon --build-users-group=guixbuild --chroot-directory=/path/to/guix_chroot_extra
                     (let ((out (assoc-ref outputs "out")))
                       (display (string-append "\nOutput Directory: \n" out "\n"))
                       (install-file "/path/to/guix_chroot_extra/pyvenv.cfg" out)
                       (substitute* (string-append out "/pyvenv.cfg")
                                    (("^home.*") (string-append "home = " out "\n"))
                                    (("^version.*") (string-append "version = 3.8.2\n"))))))

The native-search-paths can then have PYTHONPATH removed (I emulated
this by unset'ing it after creating a profile with the modified version
of the python package).

Starting python and looking at sys.path I could see this worked, both in
the cases where lib and bin symlink their counterpart
directories in the store, and also work when real lib and bin
directories in the profile contain symlinks to the store at the file level.

It seems so simple I'm sure it's been tried and found to misbehave, as I
hands-up accept it's misuse venv's funtionality (using pyvenv.cfg
without setting VIRTUAL_ENV is not defined behaviour).

But the current PYTHONPATH implementation immediately runs into problems
when you use venvs for development, via the cannonical "pip install -e
.".  The use of PYTHONPATH means that venvs are not created pure, they
inherit Guix's python env which is the exact opposite to what any
non-Guix python developer would expect.

Ironically when sharing environment setup responsibilities between
profiles and venvs like this, inheriting the profile's PYTHONPATH is
often useful - but it's still unexpected.

The PYTHONPATH, if I've understood correctly, is for user-level tweaks,
so hijacking it for system use may cause issues.

I was wondering what the current thoughts were on how Guix could
workaround using PYTHONPATH to set base system paths?

Thanks,
Phil.


^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: Avoiding PYTHONPATH - latest?
  2020-11-30  0:22 Avoiding PYTHONPATH - latest? Phil
@ 2020-11-30  9:41 ` Phil
  2020-12-03 11:06   ` 宋文武
  2020-11-30 13:05 ` zimoun
  1 sibling, 1 reply; 9+ messages in thread
From: Phil @ 2020-11-30  9:41 UTC (permalink / raw)
  To: help-guix

I should have added the pyvenv.cfg file should look like this:

$ cat pyvenv.cfg 
home = /replace/with/out/variable/in/guix
include-system-site-packages = true
version = replace.with.guix.version.variable

We include system packages to ensure we take site-packages from both the
Python 3.8 store's and the profile's site-packages.

----

One thing I have noticed is that if you want to further nest a venv
(using the regular: python3 -m venv) inside the setup below - that will
NOT work if you include --system-site-packages in the nested venv.  The
profile's site-packages are not included, because it would seem that
python does not respect the outer pyvenv.cfg in nested venvs.

If this can't be solved it pours cold water on the idea somewhat.


Phil writes:

> Hi all,
>
> I've been having an argument with myself over the last 4 days about if
> Guix's use of PYTHONPATH is a necessary evil or avoidable on a foreign OS.
>
> I've found references to a similar discussion last year, and reference
> to using a 'fake virtual environment' (does anyone have a reference to
> the other thread referenced where fake venvs are demonstrated as not
> working):
>
> https://lists.gnu.org/archive/html/guix-devel/2019-06/msg00204.html
> https://lists.gnu.org/archive/html/guix-devel/2019-06/msg00221.html
>
> I had what I think is a similar idea over the weekend and tried it out.
>
> Reading the mechanics of venvs as per PEP:
> https://www.python.org/dev/peps/pep-0405
>
> I decided to see if I added a fake pyvenv.cfg to the python.scm package
> could allow for us to drop the use of PYTHONPATH completely.
>
> Now the results are very preliminary, and I am essentially replacing
> one hack with another (I'm not creating a venv after-all) - it does seem
> to work just adding this to the python-3.8's phases: 
>
> (add-after 'install 'add-pyvenv-cfg
>            (lambda* (#:key outputs #:allow-other-keys)
>                      ;; guix-daemon --build-users-group=guixbuild --chroot-directory=/path/to/guix_chroot_extra
>                      (let ((out (assoc-ref outputs "out")))
>                        (display (string-append "\nOutput Directory: \n" out "\n"))
>                        (install-file "/path/to/guix_chroot_extra/pyvenv.cfg" out)
>                        (substitute* (string-append out "/pyvenv.cfg")
>                                     (("^home.*") (string-append "home = " out "\n"))
>                                     (("^version.*") (string-append "version = 3.8.2\n"))))))
>
> The native-search-paths can then have PYTHONPATH removed (I emulated
> this by unset'ing it after creating a profile with the modified version
> of the python package).
>
> Starting python and looking at sys.path I could see this worked, both in
> the cases where lib and bin symlink their counterpart
> directories in the store, and also work when real lib and bin
> directories in the profile contain symlinks to the store at the file level.
>
> It seems so simple I'm sure it's been tried and found to misbehave, as I
> hands-up accept it's misuse venv's funtionality (using pyvenv.cfg
> without setting VIRTUAL_ENV is not defined behaviour).
>
> But the current PYTHONPATH implementation immediately runs into problems
> when you use venvs for development, via the cannonical "pip install -e
> .".  The use of PYTHONPATH means that venvs are not created pure, they
> inherit Guix's python env which is the exact opposite to what any
> non-Guix python developer would expect.
>
> Ironically when sharing environment setup responsibilities between
> profiles and venvs like this, inheriting the profile's PYTHONPATH is
> often useful - but it's still unexpected.
>
> The PYTHONPATH, if I've understood correctly, is for user-level tweaks,
> so hijacking it for system use may cause issues.
>
> I was wondering what the current thoughts were on how Guix could
> workaround using PYTHONPATH to set base system paths?
>
> Thanks,
> Phil.



^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: Avoiding PYTHONPATH - latest?
  2020-11-30  0:22 Avoiding PYTHONPATH - latest? Phil
  2020-11-30  9:41 ` Phil
@ 2020-11-30 13:05 ` zimoun
  2020-11-30 15:04   ` Phil
  1 sibling, 1 reply; 9+ messages in thread
From: zimoun @ 2020-11-30 13:05 UTC (permalink / raw)
  To: Phil, help-guix

Hi,

I have not re-read all the discussion about PYTHONPATH and co.

On Mon, 30 Nov 2020 at 00:22, Phil <phil@beadling.co.uk> wrote:

> But the current PYTHONPATH implementation immediately runs into problems
> when you use venvs for development, via the cannonical "pip install -e
> .".  The use of PYTHONPATH means that venvs are not created pure, they
> inherit Guix's python env which is the exact opposite to what any
> non-Guix python developer would expect.

Just to understand, is your point to be able to mix both “pip install“
and “guix install”?  If yes, it appears to me a bad idea, choose one or
the other.


> I was wondering what the current thoughts were on how Guix could
> workaround using PYTHONPATH to set base system paths?

Well, I do not remember more than the attempt of GUIX_PYTHONPATH.


All the best,
simon


^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: Avoiding PYTHONPATH - latest?
  2020-11-30 13:05 ` zimoun
@ 2020-11-30 15:04   ` Phil
  2020-12-01 12:39     ` zimoun
  0 siblings, 1 reply; 9+ messages in thread
From: Phil @ 2020-11-30 15:04 UTC (permalink / raw)
  To: zimoun, help-guix

Hi

zimoun writes:

> Just to understand, is your point to be able to mix both “pip install“
> and “guix install”?  If yes, it appears to me a bad idea, choose one or
> the other.

I suppose my point is - system-level use of PYTHONPATH is ill-advised, because
it's prescribed use in Python is for user-level sys.path modifications. An
example of when this causes surprise is creating an environment using
'pip install -e .' will not give a pure environment if PYTHONPATH is set.

I 100% agree it's a bad idea to mix packagers, but pip install's use-cases go
beyond the need to do fixed installs of package@version, provided by Guix.

The use of 'pip install -e .' to create a development environment is
widely (but not exclusively) used in the python community, and
integrates nicely with many python tools (eg pytest).  It's not
essential of course.

Given that Guix profile's can't be edited from within a Guix profile
(for obvious reasons), if I want to edit my code whilst it is installed,
I see the only way to do this is by having a virtual env on top of a
Guix profile that contains only my own module - with the Guix profile
holding Python and all the imports I depend on.

It's not pretty.  So much so that I'm considering if I can live without
'-e', but the workarounds of using sys.append, PYTHONPATH, or even pth
files are, at least in some people's minds, not as clean.

If anyone has worked-out a better setup with Guix/Python, I'm keen to
get advice (hence me floating the (half-baked) idea of using pyvenv.cfg
to create Guix profiles).

There's a 2 year long discussion on a similar topic for the Poetry package
(which also aims to replace pip, for different reasons than Guix):
https://github.com/python-poetry/poetry/issues/34

Things get a bit heated, which isn't helpful, but it does illustrate how
dearly some python fans take their ability to use pip's developer setup
- I'm more pragmatic and looking for workable solutions, preferably without 
fighting Python stock features just to keep Guix happy ;-)

Perhaps the simple answer here is never mix pip and guix, and setup your
development environment so there is never a requirement on pip.  But this is
a limitation to people used to working this way.

If nothing else, personally, it was a good off-the-deep-end way to get under the
covers of Guix's treatment of python and python packages :-)


^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: Avoiding PYTHONPATH - latest?
  2020-11-30 15:04   ` Phil
@ 2020-12-01 12:39     ` zimoun
  2020-12-01 13:28       ` Phil
  0 siblings, 1 reply; 9+ messages in thread
From: zimoun @ 2020-12-01 12:39 UTC (permalink / raw)
  To: Phil, help-guix

Hi,
On Mon, 30 Nov 2020 at 15:04, Phil <phil@beadling.co.uk> wrote:

> I 100% agree it's a bad idea to mix packagers, but pip install's use-cases go
> beyond the need to do fixed installs of package@version, provided by
> Guix.

I am not sure to understand what you mean.  Installing always means
“fixed at package@version”.  I should miss something with your
workflow.


> Given that Guix profile's can't be edited from within a Guix profile
> (for obvious reasons), if I want to edit my code whilst it is installed,
> I see the only way to do this is by having a virtual env on top of a
> Guix profile that contains only my own module - with the Guix profile
> holding Python and all the imports I depend on.

This is a bad practise and should not be encouraged.  All in all, you
end up with the big mess that Guix is trying to fix.

For example, I am using Emacs and before I was using ’use-package’ which
allows to locally tweak the packages that I depend on.  Then, I wanted
to do the same with Guix.  At the end, it is wrong.  The right thing is
inspect the package and if you need to fix, do “guix build -S <pkg>”,
fix and create a variant using package transformation, for instance.
This way, you have something reproducible, easy to share and/or deploy.
IMHO.


> Perhaps the simple answer here is never mix pip and guix, and setup your
> development environment so there is never a requirement on pip.  But this is
> a limitation to people used to working this way.

My point of view is use pip or Guix, not both.  I was using Conda to
manage my Python stuff.  And same, I wanted to reproduce my Conda habits
with Guix.

Yeah, Guix changes the working habits, and as any changing habits, we
often do not like at first.  But, somehow it is only an habit.  And as
sugar in coffee, you need 3 weeks of complaints and then you will even
forget the taste of sugar. ;-)


> If nothing else, personally, it was a good off-the-deep-end way to get under the
> covers of Guix's treatment of python and python packages :-)

We discussed at the recent Guix Days that documentation “Getting
starting with X” is lacking–where X is Python, R, C, Haskell, etc.

Feel free to send a Cookbook recipe [1] about your Python setup. :-)

1: <https://guix.gnu.org/cookbook/en>


All the best,
simon



^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: Avoiding PYTHONPATH - latest?
  2020-12-01 12:39     ` zimoun
@ 2020-12-01 13:28       ` Phil
  2020-12-01 14:54         ` zimoun
  0 siblings, 1 reply; 9+ messages in thread
From: Phil @ 2020-12-01 13:28 UTC (permalink / raw)
  To: zimoun, help-guix

Thanks again zimoun for your comments.

zimoun writes:


> I am not sure to understand what you mean.  Installing always means
> “fixed at package@version”.  I should miss something with your
> workflow.

So using pip in 'editable' mode installs your git clone via softlinks as a package.
It doesn't really have a version - it's the code you're currently
working on and will change every time you save a file without having to re-package.

There is no equivalent to this in Guix - by design, of course.

> This is a bad practise and should not be encouraged.  All in all, you
> end up with the big mess that Guix is trying to fix.

What I like about 'pip install -e' is that you are testing the actual
package that end-users are going to install - without the hassle of
explicitly installing it.

I could instead run my unit tests against my git clone, but then you're
not testing what you will deliver, you're testing the source code that
will be packaged, not the package itself.  The unit tests may fail if
run on the package itself.

The way around this in Guix, I imagine, is to rebuild a Guix package
containing your own Python code each time you want to test it and then
have the unit tests run as part of that package definition.

This is what I'm going to try to do, but it might end-up being a fairly
custom setup which may not integrate so well.


>
> For example, I am using Emacs and before I was using ’use-package’ which
> allows to locally tweak the packages that I depend on.  Then, I wanted
> to do the same with Guix.  At the end, it is wrong.  The right thing is
> inspect the package and if you need to fix, do “guix build -S <pkg>”,
> fix and create a variant using package transformation, for instance.
> This way, you have something reproducible, easy to share and/or deploy.
> IMHO.

Yes this makes total sense when I'm installing someone else package
rather than developing my own using pip.


> We discussed at the recent Guix Days that documentation “Getting
> starting with X” is lacking–where X is Python, R, C, Haskell, etc.

Yes this would be great - I'm happy to chip away at it, and even change
my setup to make it more Guix-sensible.... but as a launch-pad to get
people setup quickly with an explicit list of do this, avoid this -
would be great!

>
> Feel free to send a Cookbook recipe [1] about your Python setup. :-)
>
> 1: <https://guix.gnu.org/cookbook/en>

Will keep this in mind - once I have my setup more Guix-complaint.  I'd
be keen to get feedback on it too - as per my last e-mail the Python
community have a wide range of development setups, so it will probably
be community effort to write a cookbook that caters to, or at least
rationalises the various approaches that work in Guix.


^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: Avoiding PYTHONPATH - latest?
  2020-12-01 13:28       ` Phil
@ 2020-12-01 14:54         ` zimoun
  2020-12-01 18:46           ` Phil
  0 siblings, 1 reply; 9+ messages in thread
From: zimoun @ 2020-12-01 14:54 UTC (permalink / raw)
  To: Phil, help-guix

Hi,

On Tue, 01 Dec 2020 at 13:28, Phil <phil@beadling.co.uk> wrote:

> I could instead run my unit tests against my git clone, but then you're
> not testing what you will deliver, you're testing the source code that
> will be packaged, not the package itself.  The unit tests may fail if
> run on the package itself.
>
> The way around this in Guix, I imagine, is to rebuild a Guix package
> containing your own Python code each time you want to test it and then
> have the unit tests run as part of that package definition.

You might be interested by the Efraim’s talk «Just build it with Guix»
from the online Guix Day:

  <https://guix.gnu.org/en/blog/2020/online-guix-day-announce-2/>

Roughly speaking, Guix allows an undocumented file usually named
“guix.scm” or “.guix.scm“ where the point is to easily do the “build and
test what is delivered” part.


Thanks,
simon


^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: Avoiding PYTHONPATH - latest?
  2020-12-01 14:54         ` zimoun
@ 2020-12-01 18:46           ` Phil
  0 siblings, 0 replies; 9+ messages in thread
From: Phil @ 2020-12-01 18:46 UTC (permalink / raw)
  To: zimoun, help-guix


zimoun writes:

> You might be interested by the Efraim’s talk «Just build it with Guix»
> from the online Guix Day:

Yes this is extremely useful - thanks!


^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: Avoiding PYTHONPATH - latest?
  2020-11-30  9:41 ` Phil
@ 2020-12-03 11:06   ` 宋文武
  0 siblings, 0 replies; 9+ messages in thread
From: 宋文武 @ 2020-12-03 11:06 UTC (permalink / raw)
  To: Phil; +Cc: help-guix

Hello!

Phil <phil@beadling.co.uk> writes:
>> I've been having an argument with myself over the last 4 days about if
>> Guix's use of PYTHONPATH is a necessary evil or avoidable on a foreign OS.
>>
>> I've found references to a similar discussion last year, and reference
>> to using a 'fake virtual environment' (does anyone have a reference to
>> the other thread referenced where fake venvs are demonstrated as not
>> working):

I remembered did that too, I think it works in most cases.
<https://lists.gnu.org/archive/html/guix-devel/2018-03/msg00223.html>

>>
>> https://lists.gnu.org/archive/html/guix-devel/2019-06/msg00204.html
>> https://lists.gnu.org/archive/html/guix-devel/2019-06/msg00221.html
>>
>> I had what I think is a similar idea over the weekend and tried it out.
>>
>> Reading the mechanics of venvs as per PEP:
>> https://www.python.org/dev/peps/pep-0405
>>
>> I decided to see if I added a fake pyvenv.cfg to the python.scm package
>> could allow for us to drop the use of PYTHONPATH completely.

I had came up with adding another environment variable
‘GUIX_PYTHON_X_Y_SITE_PACKAGES’:
  <https://lists.gnu.org/archive/html/guix-devel/2018-03/msg00226.html>

But didn’t made it in, feel free take it if useful, thanks!


^ permalink raw reply	[flat|nested] 9+ messages in thread

end of thread, other threads:[~2020-12-03 11:27 UTC | newest]

Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-11-30  0:22 Avoiding PYTHONPATH - latest? Phil
2020-11-30  9:41 ` Phil
2020-12-03 11:06   ` 宋文武
2020-11-30 13:05 ` zimoun
2020-11-30 15:04   ` Phil
2020-12-01 12:39     ` zimoun
2020-12-01 13:28       ` Phil
2020-12-01 14:54         ` zimoun
2020-12-01 18:46           ` Phil

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).