Environment variables in dynamic modules

Environment variables in dynamic modules

[Spencer Baugh]

In Emacs, process-environment (read by Lisp getenv) is distinct from C
environment variables (read by C getenv).

This means that a dynamic module which links against a library which
reads environment variables will not be affected by changes to
process-environment.

For example, if a user calls (setenv "VAR" "value") or binds
process-environment to (cons (cons "VAR" "value" process-environment)),
a getenv("VAR") in the dynamic module library won't return "value".
Likewise, if a dynamic module spawns subprocesses, they will inherit the
environment that the Emacs process started with, not the current
environment in process-environment.

This is usually unexpected, and causes difficult-to-track-down bugs,
especially for dynamic modules that spawn subprocesses or for large
dynamic modules with lots of functionality.

There are a number of possible ways to solve it:

A. Carefully track down every place that a library reads environment
   variables or spawns subprocesses, and pass in the Emacs environment
   instead.
   (but this is intractable in modules which call other libraries)

B. Advise Elisp setenv to also change the C environment
   (but this doesn't work with let-bindings of process-environment)

C. Set all variables in the C environment to match process-environment
   every time we call into the dynamic module
   (but this is slow and hurts performance)

D. Use linker tricks to replace C getenv with a version which calls back
   into Emacs.
   (but this doesn't work on other threads, since we can only call into
   Emacs from the main thread)

None of these are particularly satisfying.  I have implemented D, but
since my module uses multiple threads, it doesn't really solve the
problem for me.

Any suggestions?

Personally, I think some variation on B or C would be nicest, if it
could be done in a performant way which also works with let-bindings of
process-environment.  But I don't know how to do that.

Re: Environment variables in dynamic modules

[Eli Zaretskii]

> From: Spencer Baugh <sbaugh@janestreet.com>
> Date: Thu, 11 Jan 2024 11:03:25 -0500
> 
> 
> In Emacs, process-environment (read by Lisp getenv) is distinct from C
> environment variables (read by C getenv).
> 
> This means that a dynamic module which links against a library which
> reads environment variables will not be affected by changes to
> process-environment.
> 
> For example, if a user calls (setenv "VAR" "value") or binds
> process-environment to (cons (cons "VAR" "value" process-environment)),
> a getenv("VAR") in the dynamic module library won't return "value".
> Likewise, if a dynamic module spawns subprocesses, they will inherit the
> environment that the Emacs process started with, not the current
> environment in process-environment.
> 
> This is usually unexpected, and causes difficult-to-track-down bugs,
> especially for dynamic modules that spawn subprocesses or for large
> dynamic modules with lots of functionality.
> 
> There are a number of possible ways to solve it:
> 
> A. Carefully track down every place that a library reads environment
>    variables or spawns subprocesses, and pass in the Emacs environment
>    instead.
>    (but this is intractable in modules which call other libraries)
> 
> B. Advise Elisp setenv to also change the C environment
>    (but this doesn't work with let-bindings of process-environment)
> 
> C. Set all variables in the C environment to match process-environment
>    every time we call into the dynamic module
>    (but this is slow and hurts performance)
> 
> D. Use linker tricks to replace C getenv with a version which calls back
>    into Emacs.
>    (but this doesn't work on other threads, since we can only call into
>    Emacs from the main thread)
> 
> None of these are particularly satisfying.  I have implemented D, but
> since my module uses multiple threads, it doesn't really solve the
> problem for me.
> 
> Any suggestions?

The only correct solution is C, and you should only do that in the
module when the module calls something that really needs the
environment.  D is not portable, since a library that was linked to
use getenv from libc will not necessarily call getenv that you
provided, at least not in all systems.  I don't know why you say C is
slow; did you time it?  You could, of course, set only some of the
environment variables, those that matter, instead of setting all of
them.

Libraries that want to allow their callers to manipulate the
environment should do that themselves where needed.  Only the library
and its callers know when it's TRT to modify the environment and when
it is not, so no 3rd-part code (such as something in Emacs) can do
that for them.  For example, how Emacs know that some library called
by some other library that your module calls needs to see the modified
environment that Emacs uses?

It makes little sense to me to do this in Emacs, since most modules
don't care about the changes to process-environment done by Emacs.

Re: Environment variables in dynamic modules

[Spencer Baugh]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Spencer Baugh <sbaugh@janestreet.com>
>> Date: Thu, 11 Jan 2024 11:03:25 -0500
>> 
>> 
>> In Emacs, process-environment (read by Lisp getenv) is distinct from C
>> environment variables (read by C getenv).
>> 
>> This means that a dynamic module which links against a library which
>> reads environment variables will not be affected by changes to
>> process-environment.
>> 
>> For example, if a user calls (setenv "VAR" "value") or binds
>> process-environment to (cons (cons "VAR" "value" process-environment)),
>> a getenv("VAR") in the dynamic module library won't return "value".
>> Likewise, if a dynamic module spawns subprocesses, they will inherit the
>> environment that the Emacs process started with, not the current
>> environment in process-environment.
>> 
>> This is usually unexpected, and causes difficult-to-track-down bugs,
>> especially for dynamic modules that spawn subprocesses or for large
>> dynamic modules with lots of functionality.
>> 
>> There are a number of possible ways to solve it:
>> 
>> A. Carefully track down every place that a library reads environment
>>    variables or spawns subprocesses, and pass in the Emacs environment
>>    instead.
>>    (but this is intractable in modules which call other libraries)
>> 
>> B. Advise Elisp setenv to also change the C environment
>>    (but this doesn't work with let-bindings of process-environment)
>> 
>> C. Set all variables in the C environment to match process-environment
>>    every time we call into the dynamic module
>>    (but this is slow and hurts performance)
>> 
>> D. Use linker tricks to replace C getenv with a version which calls back
>>    into Emacs.
>>    (but this doesn't work on other threads, since we can only call into
>>    Emacs from the main thread)
>> 
>> None of these are particularly satisfying.  I have implemented D, but
>> since my module uses multiple threads, it doesn't really solve the
>> problem for me.
>> 
>> Any suggestions?
>
> The only correct solution is C,

Fair.

> and you should only do that in the module when the module calls
> something that really needs the environment.

Determining when that happens is the same task as A.  It's not possible
to know in a large module which calls many libraries all the places
which might read environment variables.  Some library might run a
subprocess in any function, which needs the environment.

So therefore, in a large dynamic module, C needs to be done every time
we call into the dynamic module.

> I don't know why you say C is slow; did you time it?  You could, of
> course, set only some of the environment variables, those that matter,
> instead of setting all of them.

It is not too slow if it is done only when needed.  It is too slow to do
on every call into the dynamic module.

On the other hand, maybe it could be optimized.  On every call into the
dynamic module I could check whether process-environment is eq to the
last process-environment value, and if it's not, only then synchronize
the environment.  Except setenv updates process-environment with setcdr,
so I can't just check eq.  Maybe I could also advise setenv to catch
this case.

Re: Environment variables in dynamic modules

[Andreas Schwab]

On Jan 11 2024, Spencer Baugh wrote:

> This means that a dynamic module which links against a library which
> reads environment variables will not be affected by changes to
> process-environment.

That is true for any use of getenv (the C function), dynamic modules nor
not.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: Environment variables in dynamic modules

[Psionic K]

[-- Attachment #1: Type: text/plain, Size: 1039 bytes --]

>> C. Set all variables in the C environment to match process-environment
>>    every time we call into the dynamic module
>>    (but this is slow and hurts performance)

I'm going to preface a solution to excessive redundant passing by pointing
out that the provenance of a piece of data is a requisite but something we
don't have.

Encoding input provenance in vector clocks is a solution to avoiding
unnecessary passing and maintaining caller and callee input synchronization
even when the callee is performing asynchronous work.

While we can use watch-variable for updates from Elisp, such as those done
by a package like envrc that creates buffer local environments, updates
from C don't have this convenience.

Protecting inputs from arbitrary mutations is a requirement to implement
this technique conveniently.  More likely, there need to be interfaces that
permit a logical clock and the value it versions to remain consistent.
Since the data needs to be immutable or else requires locking, coordination
with the GC is required.

[-- Attachment #2: Type: text/html, Size: 1207 bytes --]