bug#10980: 23.4; Variable init_environment incorrectly set

bug#10980: 23.4; Variable init_environment incorrectly set

[Bo Johansson]

[-- Attachment #1: Type: text/plain, Size: 6953 bytes --]

Hej!

--------------------------- Background
I am used to "live" in Emacs and want to get the same result doing something from within Emacs as doing it from a Windows command prompt.
For example (compile "dmake" nil) in Emacs and executing dmake in a Windows command window should give the same result.

--------------------------- Example of Problems
Emacs replaces the environment variable SHELL at start up in Windows to something like "C:/Program Files (x86)/GNU Emacs 23.4/bin/cmdproxy.exe". Processes started by Emacs will inherit this value.

Many utilities uses SHELL. One example is:  With " ... cmdproxy.exe" as value of SHELL dmake is configured to use a Kornshell compatible shell.

--------------------------- Solution ideas
A long time goal should be to avoid changing the environment variable within Emacs. Emacs should (internally) at least not change the value of SHELL.

A short time solution is to make the initial environment variable available as stated in http://www.gnu.org/software/emacs/manual/html_node/emacs/Environment.html: "The variable initial-environment stores the initial environment inherited by Emacs".

This make it possible in
emacs-git/lisp/progmodes/compile.el:1479
(defun compilation-start (command &optional mode name-function highlight-regexp)
to use the variable compilation-environment to temporary override the in emacs internal used environment variables.

In emacs-git/lisp/progmodes/simple.el:2088
(defun shell-command (command &optional output-buffer error-buffer)
the corresponding possibility is however not available.

--------------------------- Description of the Bug
Below follows a list of steps done at start up. The problem is that step 2) changes the environment variables before step 4) sets the variable initial_environment.

1) The function init_environment is called

emacs-git/src/emacs.c:1416
#ifdef MSDOS
  /* Call early 'cause init_environment needs it.  */
  init_dosfns ();
  /* Set defaults for several environment variables.  */
  if (initialized)
    init_environment (argc, argv, skip_args);
  else
    tzset ();
#endif /* MSDOS */

#ifdef WINDOWSNT
  globals_of_w32 ();
  /* Initialize environment from registry settings.  */
  init_environment (argv);
  init_ntproc ();    /* must precede init_editfns.  */
#endif

2) The function init_environment changes the environment variables

T:/emacs-git/src/w32.c:1528:init_environment (char ** argv)
Do a lot of changes to the environment variables.

3) The function set_initial_environment is called to save the "initial_environment"

emacs-git/src/emacs.c:1440
  /* Initialize and GC-protect Vinitial_environment and
     Vprocess_environment before set_initial_environment fills them
     in.  */
  if (!initialized)
    syms_of_callproc ();
  /* egetenv is a pretty low-level facility, which may get called in
     many circumstances; it seems flimsy to put off initializing it
     until calling init_callproc.  Do not do it when dumping.  */
  if (initialized || ((strcmp (argv[argc-1], "dump") != 0
               && strcmp (argv[argc-1], "bootstrap") != 0)))
    set_initial_environment ();

4) The variable initial_environment is set by set_initial_environment

emacs-git/src/callproc.c:1616
set_initial_environment (void)
emacs-git/src/callproc.c:1624
Vinitial_environment = Fcopy_sequence (Vprocess_environment);


-----------------------------------------------------
In GNU Emacs 23.4.1 (i386-mingw-nt6.1.7601)
of 2012-02-04 on MARVIN
Windowing system distributor `Microsoft Corp.', version 6.1.7601
configured using `configure --with-gcc (4.4) --cflags -ID:/devel/emacs/libs/libXpm-3.5.8/include -ID:/devel/emacs/libs/libXpm-3.5.8/src -ID:/devel/emacs/libs/libpng-dev_1.4.3-1/include -ID:/devel/emacs/libs/zlib-dev_1.2.5-2/include -ID:/devel/emacs/libs/giflib-4.1.4-1/include -ID:/devel/emacs/libs/jpeg-6b-4/include -ID:/devel/emacs/libs/tiff-3.8.2-1/include'

Important settings:
  value of $LC_ALL: nil
  value of $LC_COLLATE: nil
  value of $LC_CTYPE: nil
  value of $LC_MESSAGES: nil
  value of $LC_MONETARY: nil
  value of $LC_NUMERIC: nil
  value of $LC_TIME: nil
  value of $LANG: SVE
  value of $XMODIFIERS: nil
  locale-coding-system: cp1252
  default enable-multibyte-characters: t

Major mode: Lisp Interaction

Minor modes in effect:
  diff-auto-refine-mode: t
  shell-dirtrack-mode: t
  delete-selection-mode: t
  show-paren-mode: t
  cua-mode: t
  tooltip-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t
  auto-fill-mode: t

Recent input:
<escape> x r e p o <tab> r <tab> <return>

Recent messages:
Loading cua-base...done
Loading paren...done
userDotEmacs Done
Finding all versions of R on your system...
Sorry, no version of R could be found on your system.
Making completion list...

Load-path shadows:
None found.

Features:
(shadow sort mail-extr message ecomplete rfc822 mml mml-sec
password-cache mm-decode mm-bodies mm-encode mailcap mail-parse rfc2231
rfc2047 rfc2045 qp ietf-drums mailabbrev nnheader gnus-util netrc
time-date mm-util mail-prsvr gmm-utils mailheader canlock sha1 hex-util
hashcash mail-utils emacsbug help-mode view psvn byte-opt warnings
bytecomp byte-compile derived edmacro kmacro wid-edit log-edit pcvs-util
add-log diff-mode easy-mmode pp elp ediff-merg ediff-diff ediff-wind
ediff-mult ediff-help ediff-init ediff-util dired tex-mik preview-latex
tex-site auto-loads ess-eldoc ess-toolbar ess-mouse mouseme thingatpt
browse-url ess-menu ess-swv ess-noweb noweb-font-lock-mode ess-bugs-l
essd-els ess-sas-d ess-sas-l ess-sas-a executable shell ess-arc-d
ess-vst-d ess-xls-d ess-lsp-l ess-sta-d ess-sta-l cc-vars cc-defs
make-regexp ess-sp6w-d ess-sp4-d ess-sp3-d ess-r-d ess-r-args assoc
ess-s-l ess-inf ess-utils comint ring ess-mode noweb-mode ess ess-custom
regexp-opt ess-compat ess-site htmlize-view easymenu w32-winprint
htmlize cl cl-19 delsel server advice help-fns advice-preload paren
cua-base cus-start cus-load tooltip ediff-hook vc-hooks lisp-float-type
mwheel dos-w32 disp-table ls-lisp w32-win w32-vars tool-bar dnd fontset
image fringe lisp-mode register page menu-bar rfn-eshadow timer select
scroll-bar mldrag mouse jit-lock font-lock syntax facemenu font-core
frame cham georgian utf-8-lang misc-lang vietnamese tibetan thai
tai-viet lao korean japanese hebrew greek romanian slovak czech european
ethiopic indian cyrillic chinese case-table epa-hook jka-cmpr-hook help
simple abbrev loaddefs button minibuffer faces cus-face files
text-properties overlay md5 base64 format env code-pages mule custom
widget hashtable-print-readable backquote make-network-process multi-tty
emacs)

[-- Attachment #2: Type: text/html, Size: 9695 bytes --]

bug#10980: 23.4; Variable initial-environment incorrectly set

[Bo Johansson]

[-- Attachment #1: Type: text/plain, Size: 286 bytes --]

In the mail topic is “init_environment” corrected to  “initial-environment”.

Summary of bug:

On Windows the Emacs variable initial-environment is NOT set according to the documentation:
"The variable initial-environment stores the initial environment inherited by Emacs".

[-- Attachment #2: Type: text/html, Size: 526 bytes --]

bug#10980: 23.4; Variable initial-environment incorrectly set

[Eli Zaretskii]

> From: "Bo Johansson" <bo.johansson@lsn.se>
> Date: Fri, 23 Mar 2012 07:30:42 +0100
> 
> On Windows the Emacs variable initial-environment is NOT set according to the documentation:
> "The variable initial-environment stores the initial environment inherited by Emacs".

I'm sorry, I'm not sure I understand the essence of your report.  What
exactly would you like the documentation to say that it doesn't say
now?  Or, to put it differently, what in the way initial-environment
is set on Windows contradicts the documentation?

Thanks.

bug#10980: 23.4; Variable initial-environment incorrectly set

[Bo Johansson]

>From: Eli Zaretskii
>Date:     Fri, 23 Mar 2012 18:06:57 +0200
>> From: "Bo Johansson" <address@hidden>
>> Date: Fri, 23 Mar 2012 07:30:42 +0100
>>
>> On Windows the Emacs variable initial-environment is NOT set according to 
>> the
>> documentation:
>> "The variable initial-environment stores the initial environment 
>> inherited by
>> Emacs".
>
>I'm sorry, I'm not sure I understand the essence of your report.  What
>exactly would you like the documentation to say that it doesn't say
>now?  Or, to put it differently, what in the way initial-environment
>is set on Windows contradicts the documentation?
>
>Thanks.

Hej Eli!

My goal is:
I want to get the same result doing something from within Emacs as doing it 
from a Windows command prompt. To do "(compile "make" nil)" within Emacs 
should give the same result as doing "make" from the Windows command prompt.

The problem is:
The environment variables inherited by Emacs is changed. The changes in the 
environment leads to diffren results doing something from within Emacs 
compared to doing it from a Windows command prompt.

The BUG in Emacs is:
In Windows the variable initial-environment is set first after "the initial 
environment inherited by Emacs" is changed. The function init_environment 
(char ** argv) (in emacs-git/src/w32.c at line 1528)  changes the 
environment variables. The CHANGED "initial environment" is stored in the 
variable initial_environment by the function set_initial_environment (in 
emacs-git/src/callproc.c at line 1616).
However the documentation says: "The variable initial-environment stores the 
initial environment inherited by Emacs".

My goal can be achieved if the bug is corrected:
A correct value in the variable initial-environment is need to be able to 
know "the environment inherited by Emacs". This make it possible to use 
initial-environment in the lisp function (defun compilation-start (command 
&optional mode name-function highlight-regexp) in 
emacs-git/lisp/progmodes/simple.el:2088. The in emacs internal used 
environment variables can temporary be override by those in the variable 
initial-environment.
(It is of course also possible to use a new variable which contains "the 
initial environment inherited by Emacs".)

Summary: A value in the variable initial-environment according to the 
documentation makes it possible to "get the same result doing something from 
within Emacs as doing it from a Windows command prompt".

(See also 
http://lists.gnu.org/archive/html/bug-gnu-emacs/2012-03/msg00178.html.)

Best regards
Bosse 

bug#10980: 23.4; Variable initial-environment incorrectly set

[Eli Zaretskii]

> From: "Bo Johansson" <bo.johansson@lsn.se>
> Cc: <10980@debbugs.gnu.org>
> Date: Fri, 23 Mar 2012 21:20:23 +0100
> 
> Summary: A value in the variable initial-environment according to the 
> documentation makes it possible to "get the same result doing something from 
> within Emacs as doing it from a Windows command prompt".
> 
> (See also 
> http://lists.gnu.org/archive/html/bug-gnu-emacs/2012-03/msg00178.html.)

Is this only about SHELL, and only with dmake, or are there other
variables that get in the way of other programs?  Please be specific;
the few changes done to the variables outside Emacs are done for a
good reason, so it's important to know the exact adverse effects of
that to judge how justified would it be to change the current code.

bug#10980: 23.4; Variable initial-environment incorrectly set

[Bo Johansson]

>From: Eli Zaretskii
>Sent: Friday, March 23, 2012 10:05 PM
>> From: "Bo Johansson" <bo.johansson@lsn.se>
>> Cc: <10980@debbugs.gnu.org>
>> Date: Fri, 23 Mar 2012 21:20:23 +0100
>>
>> Summary: A value in the variable initial-environment according to the
>> documentation makes it possible to "get the same result doing something 
>> from
>> within Emacs as doing it from a Windows command prompt".
>>
>> (See also
>> http://lists.gnu.org/archive/html/bug-gnu-emacs/2012-03/msg00178.html.)
>
>Is this only about SHELL, and only with dmake, or are there other
>variables that get in the way of other programs?  Please be specific;
>the few changes done to the variables outside Emacs are done for a
>good reason, so it's important to know the exact adverse effects of
>that to judge how justified would it be to change the current code.

It is not only about SHELL. But SHELL is the most frequent problem.
Dmake and other make programs is just one among many other cases.

I searched internet for "emacs" and "warning: extra args ignored after". 
There was +1400 hits.
Many of theese depends on that SHELL is set to .../cmdproxy.exe. 
Adding -dmake to the search, resulted in 1280 hits. This indicates that 
dmake is only one case among others.

In the function compilation-start (lisp/progmodes/compile.el:1166) is the 
variable process-environment changed. This is done TEMPORARY by using "(let 
((process-environment ...".  However, in many other places are "persistent" 
changes made by using the lisp function setenv. This is in many cases not 
good. Activating a function can then result in an unexpected "persistent" 
change of the environment.

The last 20 years I have seen many unexpected things using "(compile 
"something" nil)", but it is first lately I have understood that one of the 
reasons are the changed environment.

At least the functions compile and shell-command should use an as far as 
possible unmodified "initial environment inherited by Emacs". I am missing a 
strategy how to handle the environment within Emacs.

I will try to find examples of other variables then SHELL!

bug#10980: 23.4; Variable initial-environment incorrectly set

[Bo Johansson]

>From: Eli Zaretskii
>Sent: Friday, March 23, 2012 10:05 PM
>> From: "Bo Johansson" <bo.johansson@lsn.se>
>> Cc: <10980@debbugs.gnu.org>
>> Date: Fri, 23 Mar 2012 21:20:23 +0100
>>
>> Summary: A value in the variable initial-environment according to the
>> documentation makes it possible to "get the same result doing something 
>> from
>> within Emacs as doing it from a Windows command prompt".
>>
>Is this only about SHELL, and only with dmake, or are there other
>variables that get in the way of other programs?  Please be specific;
>the few changes done to the variables outside Emacs are done for a
>good reason, so it's important to know the exact adverse effects of
>that to judge how justified would it be to change the current code.

Hej Eli!

Here is some more information.

I have looked in makefile using http://code.google.com/codesearch. SHELL and 
COMSPEC are frequently used. In some cases is even ComSpec used.
To set SHELL to c:/Program Files (x86)/GNU Emacs 23.4/bin/cmdproxy.exe 
causes a lot of problems.
The resulting errors are often difficult to find. This partly because the 
warning messages given by cmdproxy.exe is often given without any context.

A program developed to run in Windows, expects a “Windows environment”!?

I have compared the environment variables in different situations. The 
differences between the environment in Windows cmd.exe and the Emacs 
variable initial-environment are:

-ComSpec=C:\Windows\system32\cmd.exe
+COMSPEC=C:\Windows\system32\cmd.exe
+EMACSDATA=c:/Program Files (x86)/GNU Emacs 23.4/etc
+EMACSDOC=c:/Program Files (x86)/GNU Emacs 23.4/etc
+EMACSLOADPATH=c:/Program Files (x86)/GNU Emacs 23.4/site-lisp;c:/Program 
Files (x86)/GNU Emacs 23.4/../site-lisp;c:/Program Files (x86)/GNU Emacs 
23.4/lisp;c:/Program Files (x86)/GNU Emacs 23.4/leim
+EMACSPATH=c:/Program Files (x86)/GNU Emacs 23.4/bin
+LANG=SVE
-Path=...
+PATH=...
+SHELL=c:/Program Files (x86)/GNU Emacs 23.4/bin/cmdproxy.exe
+TMPDIR=C:\Users\bo\AppData\Local\Temp
+emacs_dir=c:/Program Files (x86)/GNU Emacs 23.4

Comspec and Path are renamed. SHELL is added and set to the cmdproxy.exe.

Differences between initial-environment to shell-command environment are:

+EM_PARENT_PROCESS_ID=...


Differences initial-environment to compile environment are:

+DISPLAY=
+EMACS=t
+EM_PARENT_PROCESS_ID=...
+INSIDE_EMACS=t
-TERM=dumb
+TERM=emacs
+TERMCAP=emacs:co#80:tc=unknown:

This comment in emacs-git/src/w32.c:1778:
/* Another special case: on NT, the PATH variable is actually named
     "Path" although cmd.exe (perhaps NT itself) arranges for
     environment variable lookup and setting to be case insensitive.
     However, Emacs assumes a fully case sensitive environment, so we
     need to change "Path" to "PATH" to match the expectations of
     various elisp packages.  We do this by the sneaky method of
     modifying the string in the C runtime environ entry.

     The same applies to COMSPEC.  */
indicates that changes to the environment are mainly? made for Emacs, NOT 
for the processes started by Emacs. Emacs should separate the "internal 
environment" used internally by Emacs and the external used to set up 
external processes!?

My idea to get a read-only "inherited environment" is:
1) To save the "inherited environment" early in "c-code" at start up of 
Emacs
2) Implement a lisp function which can return the saved "inherited 
environment".

The new read-only "inherited environment" can then later be used to start 
external processes with a more "transparent" environment.
To start to change the current handling of the variable initial-environment 
is probably difficult and error prone.