bug#62448: Inconsistent guile enviornment from guix shell

bug#62448: Inconsistent guile enviornment from guix shell

[Skyler via Bug reports for GNU Guix]

[-- Attachment #1.1.1: Type: text/plain, Size: 5464 bytes --]

Hello,

The modules available in a guile REPL seem to be inconsistent after running the guix shell command, where using the -D flag to create a development environment behaves as expected, but leaving out that flag to add the package itself to the environment does not. I am experiencing this issue with some random code on my machine, but I will use guile-pubstrate defined in guix code (https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/guile-xyz.scm?id=04724e59971b03f86a410285653d24005c62b924#n340) as an example for reproduction. Note that guile-pubstrate lists guile-webutils as a propagated input.

When I use `guix shell -D guile-pubstrate`, I am able to import the `(webutils bytevectors)` module, as I would expect. I am not able to import `(pubstrate apclient)`, also as I would expect. Inspecting GUILE_LOAD_PATH shows that a new profile path was added which contains the dependencies:

$ guix shell -D guile-pubstrate --checkguix shell: checking the environment variables visible from shell '/gnu/store/d99ykvj3axzzidygsmdmzxah4lvxd6hw-bash-5.1.8/bin/bash'...
guix shell: All is good!  The shell gets correct environment variables.
[env]$ guile
GNU Guile 3.0.7
Copyright (C) 1995-2021 Free Software Foundation, Inc.

Guile comes with ABSOLUTELY NO WARRANTY; for details type `,show w'.
This program is free software, and you are welcome to redistribute it
under certain conditions; type `,show c' for details.

Enter `,help' for help.
scheme@(guile-user)> (use-modules (webutils bytevectors))
scheme@(guile-user)> (use-modules (pubstrate apclient))
While compiling expression:
no code for module (pubstrate apclient)
scheme@(guile-user)> ,q
[env]$ echo $GUILE_LOAD_PATH
/gnu/store/pr4vyrz2xsziv25hrdfyim0wy331p4z5-profile/share/guile/site/3.0:/run/current-system/profile/share/guile/site/3.0:/home/skyler/.guix-profile/share/guile/site/3.0
$ ls /gnu/store/pr4vyrz2xsziv25hrdfyim0wy331p4z5-profile/share/guile/site/3.0
8sync      apicheck.scm  config     debugging  fibers      gcrypt    gnutls      graph         io       match-bind.scm  md5.scm  rx      search  sjson.scm  term     text           web
8sync.scm  compat        container  fash.scm   fibers.scm  gdbm.scm  gnutls.scm  htmlprag.scm  logging  math            os       scheme  sjson   string     texinfo  unit-test.scm  webutils

However, when I use guix shell guile-pubstrate, neither (webutils bytevectors) nor (pubstrate apclient) are available. I would expect both of them to be, the former because it is a propagated input and the latter because it is the package that was directly added. If guile was able to locate pubstrate's modules, I would expect loading to fail because its runtime dependencies are not available. Inspecting GUILE_LOAD_PATH shows that nothing was added to it:

$ guix shell guile-pubstrate --check
guix shell: checking the environment variables visible from shell '/gnu/store/d99ykvj3axzzidygsmdmzxah4lvxd6hw-bash-5.1.8/bin/bash'...
guix shell: All is good!  The shell gets correct environment variables.
[env]$ guile
GNU Guile 3.0.9
Copyright (C) 1995-2023 Free Software Foundation, Inc.

Guile comes with ABSOLUTELY NO WARRANTY; for details type `,show w'.
This program is free software, and you are welcome to redistribute it
under certain conditions; type `,show c' for details.

Enter `,help' for help.
scheme@(guile-user)> (use-modules (webutils bytevectors))
While compiling expression:
no code for module (webutils bytevectors)
scheme@(guile-user)> (use-modules (pubstrate apclient))
While compiling expression:
no code for module (pubstrate apclient)
scheme@(guile-user)> ,q
[env]$ echo $GUILE_LOAD_PATH
/run/current-system/profile/share/guile/site/3.0:/home/skyler/.guix-profile/share/guile/site/3.0


This is not an issue with the guile-pubstrate definition in particular, because guix shell guile-webutils also fails to add its modules to my environment:


$ guix shell --check guile-webutilsThe following derivation will be built:
  /gnu/store/ca8gv3yk2khs0h60kc5v682g7gvfqd23-profile.drv

building CA certificate bundle...
listing Emacs sub-directories...
building fonts directory...
building directory of Info manuals...
building profile with 1 package...
guix shell: checking the environment variables visible from shell '/gnu/store/d99ykvj3axzzidygsmdmzxah4lvxd6hw-bash-5.1.8/bin/bash'...
guix shell: All is good!  The shell gets correct environment variables.
[env]$ guile
GNU Guile 3.0.9
Copyright (C) 1995-2023 Free Software Foundation, Inc.

Guile comes with ABSOLUTELY NO WARRANTY; for details type `,show w'.
This program is free software, and you are welcome to redistribute it
under certain conditions; type `,show c' for details.

Enter `,help' for help.
scheme@(guile-user)> (use-modules (webutils bytevectors))
While compiling expression:
no code for module (webutils bytevectors)

My machine is up-to-date as of today:

Generation 32 Mar 25 2023 11:32:38 (current)  guix 04724e5
    repository URL: https://git.savannah.gnu.org/git/guix.git
    branch: master
    commit: 04724e59971b03f86a410285653d24005c62b924
  rde a7b5944
    repository URL: https://git.sr.ht/~abcdw/rde
    branch: master
    commit: a7b59443405169600a00f0b295a3fb1de360cb0b


Let me know if there is anything I can do to help diagnose or fix this issue.


Thanks,
Skyler

[-- Attachment #1.1.2.1: Type: text/html, Size: 13650 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 249 bytes --]

bug#62448: Inconsistent guile enviornment from guix shell

[Skyler via Bug reports for GNU Guix]

[-- Attachment #1: Type: text/plain, Size: 7173 bytes --]

Hello again,

I did some more digging, and it looks like the issue is that guile also has to be specified in the shell environment. I suppose this is reasonable, because the result of the shell environment should not depend on the calling environment, and the existence of guile in the environment is what tells guix that the variables need to be set. The load path is set with the -D flag because guile is a non-propagated input to the package, so it exists in the development shell environment automatically, but not in an ad-hoc environment.

I determined this by trying variations of the shell command in test users with clean environments, one of them including a home environment because I thought that might be a factor. There was no change based on whether the user had a home environment installed, but both users experienced the problem when guile was not included in the shell command. Incidentally, using the pure or container flags always works, because guile must be specified regardless of the calling environment in those cases, but I don't think it is immediately obvious why those flags "make it work".

In spite of being reasonable, it is not particularly intuitive. Could I recommend an addition to the manual in section "7.1 Invoking guix-shell", adding text like the below after the example invocation `guix shell python python-numpy -- python3​`?

"Note that it is necessary to include the main python package in this command even if it is already installed into your environment. This is so that the shell environment knows to set PYTHONPATH and other related variables. The shell environment cannot check the previously installed environment, because then it would be non-deterministic."

Thanks,
Skyler
------- Original Message -------
On Saturday, March 25th, 2023 at 12:10 PM, Skyler <skyvine@protonmail.com> wrote:

> Hello,
>
> The modules available in a guile REPL seem to be inconsistent after running the guix shell command, where using the -D flag to create a development environment behaves as expected, but leaving out that flag to add the package itself to the environment does not. I am experiencing this issue with some random code on my machine, but I will use guile-pubstrate defined in guix code (https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/guile-xyz.scm?id=04724e59971b03f86a410285653d24005c62b924#n340) as an example for reproduction. Note that guile-pubstrate lists guile-webutils as a propagated input.
>
> When I use guix shell -D guile-pubstrate​, I am able to import the (webutils bytevectors)​ module, as I would expect. I am not able to import (pubstrate apclient)​, also as I would expect. Inspecting GUILE_LOAD_PATH shows that a new profile path was added which contains the dependencies:
>
> $ guix shell -D guile-pubstrate --check
> guix shell: checking the environment variables visible from shell '/gnu/store/d99ykvj3axzzidygsmdmzxah4lvxd6hw-bash-5.1.8/bin/bash'...
> guix shell: All is good! The shell gets correct environment variables.
> [env]$ guile
> GNU Guile 3.0.7
> Copyright (C) 1995-2021 Free Software Foundation, Inc.
>
> Guile comes with ABSOLUTELY NO WARRANTY; for details type `,show w'.
> This program is free software, and you are welcome to redistribute it
> under certain conditions; type `,show c' for details.
>
> Enter `,help' for help.
> scheme@(guile-user)> (use-modules (webutils bytevectors))
> scheme@(guile-user)> (use-modules (pubstrate apclient))
> While compiling expression:
> no code for module (pubstrate apclient)
> scheme@(guile-user)> ,q
> [env]$ echo $GUILE_LOAD_PATH
> /gnu/store/pr4vyrz2xsziv25hrdfyim0wy331p4z5-profile/share/guile/site/3.0:/run/current-system/profile/share/guile/site/3.0:/home/skyler/.guix-profile/share/guile/site/3.0
> $ ls /gnu/store/pr4vyrz2xsziv25hrdfyim0wy331p4z5-profile/share/guile/site/3.0
> 8sync apicheck.scm config debugging fibers gcrypt gnutls graph io match-bind.scm md5.scm rx search sjson.scm term text web
> 8sync.scm compat container fash.scm fibers.scm gdbm.scm gnutls.scm htmlprag.scm logging math os scheme sjson string texinfo unit-test.scm webutils
>
> However, when I use guix shell guile-pubstrate, neither (webutils bytevectors) nor (pubstrate apclient) are available. I would expect both of them to be, the former because it is a propagated input and the latter because it is the package that was directly added. If guile was able to locate pubstrate's modules, I would expect loading to fail because its runtime dependencies are not available. Inspecting GUILE_LOAD_PATH shows that nothing was added to it:
>
> $ guix shell guile-pubstrate --check
> guix shell: checking the environment variables visible from shell '/gnu/store/d99ykvj3axzzidygsmdmzxah4lvxd6hw-bash-5.1.8/bin/bash'...
> guix shell: All is good! The shell gets correct environment variables.
> [env]$ guile
> GNU Guile 3.0.9
> Copyright (C) 1995-2023 Free Software Foundation, Inc.
>
> Guile comes with ABSOLUTELY NO WARRANTY; for details type `,show w'.
> This program is free software, and you are welcome to redistribute it
> under certain conditions; type `,show c' for details.
>
> Enter `,help' for help.
> scheme@(guile-user)> (use-modules (webutils bytevectors))
> While compiling expression:
> no code for module (webutils bytevectors)
> scheme@(guile-user)> (use-modules (pubstrate apclient))
> While compiling expression:
> no code for module (pubstrate apclient)
> scheme@(guile-user)> ,q
> [env]$ echo $GUILE_LOAD_PATH
> /run/current-system/profile/share/guile/site/3.0:/home/skyler/.guix-profile/share/guile/site/3.0
>
> This is not an issue with the guile-pubstrate definition in particular, because guix shell guile-webutils also fails to add its modules to my environment:
>
> $ guix shell --check guile-webutils
> The following derivation will be built:
> /gnu/store/ca8gv3yk2khs0h60kc5v682g7gvfqd23-profile.drv
>
> building CA certificate bundle...
> listing Emacs sub-directories...
> building fonts directory...
> building directory of Info manuals...
> building profile with 1 package...
> guix shell: checking the environment variables visible from shell '/gnu/store/d99ykvj3axzzidygsmdmzxah4lvxd6hw-bash-5.1.8/bin/bash'...
> guix shell: All is good! The shell gets correct environment variables.
> [env]$ guile
> GNU Guile 3.0.9
> Copyright (C) 1995-2023 Free Software Foundation, Inc.
>
> Guile comes with ABSOLUTELY NO WARRANTY; for details type `,show w'.
> This program is free software, and you are welcome to redistribute it
> under certain conditions; type `,show c' for details.
>
> Enter `,help' for help.
> scheme@(guile-user)> (use-modules (webutils bytevectors))
> While compiling expression:
> no code for module (webutils bytevectors)
>
> My machine is up-to-date as of today:
>
> Generation 32 Mar 25 2023 11:32:38 (current)
> guix 04724e5
> repository URL: https://git.savannah.gnu.org/git/guix.git
> branch: master
> commit: 04724e59971b03f86a410285653d24005c62b924
> rde a7b5944
> repository URL: https://git.sr.ht/~abcdw/rde
> branch: master
> commit: a7b59443405169600a00f0b295a3fb1de360cb0b
>
> Let me know if there is anything I can do to help diagnose or fix this issue.
>
> Thanks,
> Skyler

[-- Attachment #2: Type: text/html, Size: 16771 bytes --]

bug#62448: Inconsistent guile enviornment from guix shell

[Josselin Poiret via Bug reports for GNU Guix]

[-- Attachment #1: Type: text/plain, Size: 868 bytes --]

Hi Skyler,

It's a shame that nobody had pointed out this (quite common) missing
piece earlier.

Skyler via Bug reports for GNU Guix <bug-guix@gnu.org> writes:

> In spite of being reasonable, it is not particularly intuitive. Could I recommend an addition to the manual in section "7.1 Invoking guix-shell", adding text like the below after the example invocation `guix shell python python-numpy -- python3​`?
>
> "Note that it is necessary to include the main python package in this command even if it is already installed into your environment. This is so that the shell environment knows to set PYTHONPATH and other related variables. The shell environment cannot check the previously installed environment, because then it would be non-deterministic."

Yes, this would be a good idea.  I can try to do this right now.

Best,
-- 
Josselin Poiret

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 682 bytes --]

bug#62448: [PATCH] doc: Note that `guix shell` should contain base language packages.

[Josselin Poiret via Bug reports for GNU Guix]

* doc/guix.texi (Invoking guix shell): Explain that python must be included in
the invocation even if it is already available in the external environment.

Suggested-by: Skyler <skyvine@protonmail.com>
---
 doc/guix.texi | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/doc/guix.texi b/doc/guix.texi
index c49e51b72e..879a526b77 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -96,7 +96,7 @@ Copyright @copyright{} 2021 Domagoj Stolfa@*
 Copyright @copyright{} 2021 Hui Lu@*
 Copyright @copyright{} 2021 pukkamustard@*
 Copyright @copyright{} 2021 Alice Brenon@*
-Copyright @copyright{} 2021, 2022 Josselin Poiret@*
+Copyright @copyright{} 2021, 2022, 2023 Josselin Poiret@*
 Copyright @copyright{} 2021 muradm@*
 Copyright @copyright{} 2021, 2022 Andrew Tropin@*
 Copyright @copyright{} 2021 Sarah Morgensen@*
@@ -5959,6 +5959,14 @@ building or downloading any missing package, and runs the
 guix shell python python-numpy -- python3
 @end example
 
+Note that it is necessary to include the main @code{python} package in
+this command even if it is already installed into your environment. This
+is so that the shell environment knows to set @env{PYTHONPATH} and other
+related variables. The shell environment cannot check the previously
+installed environment, because then it would be non-deterministic.  This
+is true for most libraries: their corresponding language package should
+be included in the shell invocation.
+
 @quotation Note
 @cindex shebang, for @command{guix shell}
 @command{guix shell} can be also be used as a script interpreter, also

base-commit: be5e280e5fe26f93bd5a6e3f76e4502edb913a94
-- 
2.39.2

bug#62448: [PATCH] doc: Note that `guix shell` should contain base language packages.

[Skyler via Bug reports for GNU Guix]

Is there a normal timeline for patch review or is it mor of an "as resources allow" type of deal?

------- Original Message -------
On Thursday, March 30th, 2023 at 1:42 PM, Josselin Poiret <dev@jpoiret.xyz> wrote:


> * doc/guix.texi (Invoking guix shell): Explain that python must be included in
> the invocation even if it is already available in the external environment.
> 
> Suggested-by: Skyler skyvine@protonmail.com
> 
> ---
> doc/guix.texi | 10 +++++++++-
> 1 file changed, 9 insertions(+), 1 deletion(-)
> 
> diff --git a/doc/guix.texi b/doc/guix.texi
> index c49e51b72e..879a526b77 100644
> --- a/doc/guix.texi
> +++ b/doc/guix.texi
> @@ -96,7 +96,7 @@ Copyright @copyright{} 2021 Domagoj Stolfa@*
> Copyright @copyright{} 2021 Hui Lu@*
> Copyright @copyright{} 2021 pukkamustard@*
> Copyright @copyright{} 2021 Alice Brenon@*
> -Copyright @copyright{} 2021, 2022 Josselin Poiret@*
> +Copyright @copyright{} 2021, 2022, 2023 Josselin Poiret@*
> Copyright @copyright{} 2021 muradm@*
> Copyright @copyright{} 2021, 2022 Andrew Tropin@*
> Copyright @copyright{} 2021 Sarah Morgensen@*
> @@ -5959,6 +5959,14 @@ building or downloading any missing package, and runs the
> guix shell python python-numpy -- python3
> @end example
> 
> +Note that it is necessary to include the main @code{python} package in
> +this command even if it is already installed into your environment. This
> +is so that the shell environment knows to set @env{PYTHONPATH} and other
> +related variables. The shell environment cannot check the previously
> +installed environment, because then it would be non-deterministic. This
> +is true for most libraries: their corresponding language package should
> +be included in the shell invocation.
> +
> @quotation Note
> @cindex shebang, for @command{guix shell}
> @command{guix shell} can be also be used as a script interpreter, also
> 
> base-commit: be5e280e5fe26f93bd5a6e3f76e4502edb913a94
> --
> 2.39.2

bug#62448: [PATCH] doc: Note that `guix shell` should contain base language packages.

[Josselin Poiret via Bug reports for GNU Guix]

[-- Attachment #1: Type: text/plain, Size: 465 bytes --]

Hi Skyler,

Skyler <skyvine@protonmail.com> writes:

> Is there a normal timeline for patch review or is it mor of an "as resources allow" type of deal?

Unfortunately no, there are no guarantees: someone needs to take some
time to look at this, and we can't force anyone to do it.  The fact that
it's in a bug report instead of a patch submission probably doesn't
help, let me see if I can fix this with a debbugs command.

Best,
-- 
Josselin Poiret

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 682 bytes --]

bug#62448: [PATCH] doc: Note that `guix shell` should contain base language packages.

[Simon Tournier]

Hi,

On jeu., 30 mars 2023 at 22:42, Josselin Poiret via Bug reports for GNU Guix <bug-guix@gnu.org> wrote:

> +Note that it is necessary to include the main @code{python} package in
> +this command even if it is already installed into your environment. This
> +is so that the shell environment knows to set @env{PYTHONPATH} and other
> +related variables. The shell environment cannot check the previously
> +installed environment, because then it would be non-deterministic.  This
> +is true for most libraries: their corresponding language package should
> +be included in the shell invocation.

Nitpick: Missing double space for sentences.

Otherwise, LGTM.


Cheers,
simon

bug#62448: [PATCH] doc: Note that `guix shell` should contain base language packages.

[Maxim Cournoyer]

Hello,

Skyler via Bug reports for GNU Guix <bug-guix@gnu.org> writes:

> Is there a normal timeline for patch review or is it mor of an "as resources allow" type of deal?

The later, I'd say; we're all volunteers :-).

-- 
Thanks,
Maxim

bug#62448: [PATCH] doc: Note that `guix shell` should contain base language packages.

[Skyler via Bug reports for GNU Guix]

> The later, I'd say; we're all volunteers :-).

Fair enough!

bug#62448: [PATCH] doc: Note that `guix shell` should contain base language packages.

[Josselin Poiret via Bug reports for GNU Guix]

[-- Attachment #1: Type: text/plain, Size: 238 bytes --]

Hi everyone,

Simon Tournier <zimon.toutoune@gmail.com> writes:

> Nitpick: Missing double space for sentences.
>
> Otherwise, LGTM.

Pushed as 9f68a2a9c41166ec5ac24c082bcd96c433dd2ede.

Closing.

Best,
-- 
Josselin Poiret

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 682 bytes --]