Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Wed, 28 Aug 2024 00:13:25 +0100
>> From:  Evgenii Klimov via "Bug reports for GNU Emacs,
>>  the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>
>> 
>> `with-temp-buffer' doesn't respect buffer-local environment variables,
>> `exec-path' in this case.  Which results in executables not being found,
>> or the wrong versions of executables being picked up.  E.g. if
>> environment variable is modified via .dir-local file or direnv/envrc
>> package.
>
> Hmm, this doesn't look clean to me: exec-path is just one variable,
> what makes it special here?
>
> Moreover, it sounds like python-shell-with-environment, which
> python-shell-prompt-detect calls, already attempts to have
> buffer-local value of exec-path to be available to Python, so why
> isn't that working for you?  And if it isn't work, I think we should
> amend python-shell-with-environment to do this, so we don't need to do
> it "by hand".

Indeed, my initial approach is too manual.

Here the problem that I have: I don't use Python's "venv" module to
create virtual environment for the project.  Instead, I use GNU Guix's
"guix shell" command [1] which provides augmented PATH and PYTHONPATH,
etc. to link project's dependencies.  Then, envrc.el package picks up
these environment variables and makes them buffer-local project-wise
(`exec-path' and `process-environment').

You are correct that `python-shell-with-environment' provides
buffer-local variables, but `with-temp-buffer' treats `exec-path' and
`process-environment' variables very specially.

I didn't find this behavior in documentation, but look at this example:

  (setq-default exec-path (list "global" "list"))
  (setq-local exec-path (cons "local"
                              (default-value 'exec-path)))
  (setq-default myvar (list "global" "list"))
  (setq-local myvar (cons "local" (default-value 'myvar)))

  (let ((exec-path exec-path) 		; takes buffer-local
        (myvar myvar)) 			; takes buffer-local
    (with-temp-buffer
      (insert (car exec-path) 		; uses global
              "\n"
              (car myvar)) 		; uses `let'-binded
      (buffer-string)))
  ;; => "global
  ;;     local"

  (require 'cl-lib)
  (let ((myvar myvar))
    ;; temporarily binds buffer-local value to global symbol
    (cl-letf (((default-value 'exec-path) exec-path))
      (with-temp-buffer
        ;; global variable is used, but it's value is temporarily equal
        ;; to buffer-local value
        (insert (car exec-path)
                "\n"
                (car myvar))
        (buffer-string))))
  ;; => "local
  ;;     local"

It's a simplified and expanded version of
`python-shell-with-environment' and `python-shell-prompt-detect'.  As
you can see, `exec-path' is treated differently inside of
`with-temp-buffer' and `cl-letf' is needed to force `with-temp-buffer'
to use buffer-local value of `exec-path'.

In the new patch attached I show how this can be overcome.  Don't know
if you'll consider my use case too narrow and specific, but I'll be glad
to hear your thoughts on this.

[1] https://guix.gnu.org/en/manual/en/guix.html#Invoking-guix-environment