bug#72849: [PATCH] Keep project's exec-path during with-temp-buffer call

[Evgenii Klimov via "Bug reports for GNU Emacs, the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>]

[-- Attachment #1: Type: text/plain, Size: 3162 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Wed, 28 Aug 2024 00:13:25 +0100
>> From:  Evgenii Klimov via "Bug reports for GNU Emacs,
>>  the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>
>> 
>> `with-temp-buffer' doesn't respect buffer-local environment variables,
>> `exec-path' in this case.  Which results in executables not being found,
>> or the wrong versions of executables being picked up.  E.g. if
>> environment variable is modified via .dir-local file or direnv/envrc
>> package.
>
> Hmm, this doesn't look clean to me: exec-path is just one variable,
> what makes it special here?
>
> Moreover, it sounds like python-shell-with-environment, which
> python-shell-prompt-detect calls, already attempts to have
> buffer-local value of exec-path to be available to Python, so why
> isn't that working for you?  And if it isn't work, I think we should
> amend python-shell-with-environment to do this, so we don't need to do
> it "by hand".

Indeed, my initial approach is too manual.

Here the problem that I have: I don't use Python's "venv" module to
create virtual environment for the project.  Instead, I use GNU Guix's
"guix shell" command [1] which provides augmented PATH and PYTHONPATH,
etc. to link project's dependencies.  Then, envrc.el package picks up
these environment variables and makes them buffer-local project-wise
(`exec-path' and `process-environment').

You are correct that `python-shell-with-environment' provides
buffer-local variables, but `with-temp-buffer' treats `exec-path' and
`process-environment' variables very specially.

I didn't find this behavior in documentation, but look at this example:

  (setq-default exec-path (list "global" "list"))
  (setq-local exec-path (cons "local"
                              (default-value 'exec-path)))
  (setq-default myvar (list "global" "list"))
  (setq-local myvar (cons "local" (default-value 'myvar)))

  (let ((exec-path exec-path) 		; takes buffer-local
        (myvar myvar)) 			; takes buffer-local
    (with-temp-buffer
      (insert (car exec-path) 		; uses global
              "\n"
              (car myvar)) 		; uses `let'-binded
      (buffer-string)))
  ;; => "global
  ;;     local"

  (require 'cl-lib)
  (let ((myvar myvar))
    ;; temporarily binds buffer-local value to global symbol
    (cl-letf (((default-value 'exec-path) exec-path))
      (with-temp-buffer
        ;; global variable is used, but it's value is temporarily equal
        ;; to buffer-local value
        (insert (car exec-path)
                "\n"
                (car myvar))
        (buffer-string))))
  ;; => "local
  ;;     local"

It's a simplified and expanded version of
`python-shell-with-environment' and `python-shell-prompt-detect'.  As
you can see, `exec-path' is treated differently inside of
`with-temp-buffer' and `cl-letf' is needed to force `with-temp-buffer'
to use buffer-local value of `exec-path'.

In the new patch attached I show how this can be overcome.  Don't know
if you'll consider my use case too narrow and specific, but I'll be glad
to hear your thoughts on this.

[1] https://guix.gnu.org/en/manual/en/guix.html#Invoking-guix-environment


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Make-python-shell-with-environment-respect-buffer-lo.patch --]
[-- Type: text/x-patch, Size: 1449 bytes --]

From 6db90f6087d0d3270e1d7a7d77d8830eac1f0154 Mon Sep 17 00:00:00 2001
From: Evgenii Klimov <eugene.dev@lipklim.org>
Date: Thu, 29 Aug 2024 23:36:12 +0100
Subject: [PATCH] Make python-shell--with-environment respect buffer-local
 variables

* lisp/progmodes/python.el (python-shell--with-environment): Make
`with-temp-buffer' respect buffer-local values of
`process-environment' and `exec-path', if set.
---
 lisp/progmodes/python.el | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el
index 7193cc19425..3e543442dba 100644
--- a/lisp/progmodes/python.el
+++ b/lisp/progmodes/python.el
@@ -3030,11 +3030,11 @@ (defun python-shell--with-environment (extraenv bodyfun)
                  (tramp-dissect-file-name default-directory 'noexpand)))))
     (if vec
         (python-shell--tramp-with-environment vec extraenv bodyfun)
-      (let ((process-environment
-             (append extraenv process-environment))
-            (exec-path
-             ;; FIXME: This is still Python-specific.
-             (python-shell-calculate-exec-path)))
+      (cl-letf (((default-value 'process-environment)
+		 (append extraenv process-environment))
+		((default-value 'exec-path)
+		 ;; FIXME: This is still Python-specific.
+		 (python-shell-calculate-exec-path)))
         (funcall bodyfun)))))
 
 (defun python-shell--tramp-with-environment (vec extraenv bodyfun)
-- 
2.45.2