bug#11959: 24.1.50; Warning: Lisp directory `C:/Emacs-24-2012-07-16/../site-lisp' does not exist.

bug#11959: 24.1.50; Warning: Lisp directory `C:/Emacs-24-2012-07-16/../site-lisp' does not exist.

[Drew Adams]

I do not see this warning message with emacs -Q, but I see it with my
setup.  I do not find anywhere in my setup where I refer to site-lisp,
but perhaps I do somewhere.
 
When I start Emacs this warning is the first thing in buffer *Messages*:
 
Warning: Lisp directory `C:/Emacs-24-2012-07-16/../site-lisp' does not exist.
 
My directory structure is this:
 
C:/Emacs-24-2012-07-16 contains the following, all of which comes
from the Windows binary I downloaded from GNU:
 
 bin/
 etc/
 info/
 leim/
 lisp/
 site-lisp/
 .gdbinit
 BUGS
 COPYING
 README
 README.W32
 
I made no changes to the directory content or structure, apart from
renaming the top-level directory to Emacs-24-2012-07-16.
 
So why the warning?  Why is Emacs looking for a site-lisp directory above
directory Emacs-24-2012-07-16?  And what does this all mean?

In GNU Emacs 24.1.50.1 (i386-mingw-nt5.1.2600)
 of 2012-07-16 on MARVIN
Bzr revision: 109106 fabian@anue.biz-20120716171839-0dv19ib9h6vfggfn
Windowing system distributor `Microsoft Corp.', version 5.1.2600
Configured using:
 `configure --with-gcc (4.6) --no-opt --enable-checking --cflags
 -ID:/devel/emacs/libs/libXpm-3.5.8/include
 -ID:/devel/emacs/libs/libXpm-3.5.8/src
 -ID:/devel/emacs/libs/libpng-dev_1.4.3-1/include
 -ID:/devel/emacs/libs/zlib-dev_1.2.5-2/include
 -ID:/devel/emacs/libs/giflib-4.1.4-1/include
 -ID:/devel/emacs/libs/jpeg-6b-4/include
 -ID:/devel/emacs/libs/tiff-3.8.2-1/include
 -ID:/devel/emacs/libs/gnutls-3.0.9/include
 -ID:/devel/emacs/libs/libiconv-1.13.1-1-dev/include
 -ID:/devel/emacs/libs/libxml2-2.7.8/include/libxml2'
 

bug#11959: 24.1.50; Warning: Lisp directory `C:/Emacs-24-2012-07-16/../site-lisp' does not exist.

[Eli Zaretskii]

> From: "Drew Adams" <drew.adams@oracle.com>
> Date: Tue, 17 Jul 2012 09:40:31 -0700
> 
> I do not see this warning message with emacs -Q, but I see it with my
> setup.  I do not find anywhere in my setup where I refer to site-lisp,
> but perhaps I do somewhere.
>  
> When I start Emacs this warning is the first thing in buffer *Messages*:
>  
> Warning: Lisp directory `C:/Emacs-24-2012-07-16/../site-lisp' does not exist.

It's a bug, I've seen it since a few days ago and will get to fixing
it when I have time (unless someone beats me to it).

For now, just create that directory, to get it to shut up.

bug#11959: 24.1.50; Warning: Lisp directory `C:/Emacs-24-2012-07-16/../site-lisp' does not exist.

[Drew Adams]

> It's a bug, I've seen it since a few days ago and will get to fixing
> it when I have time (unless someone beats me to it).

OK, thanks.

> For now, just create that directory, to get it to shut up.

I don't mind it for now.

bug#11959: 24.1.50; Warning: Lisp directory `C:/Emacs-24-2012-07-16/../site-lisp' does not exist.

[Eli Zaretskii]

> Date: Tue, 17 Jul 2012 20:40:10 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: 11959@debbugs.gnu.org
> 
> > From: "Drew Adams" <drew.adams@oracle.com>
> > Date: Tue, 17 Jul 2012 09:40:31 -0700
> > 
> > I do not see this warning message with emacs -Q, but I see it with my
> > setup.  I do not find anywhere in my setup where I refer to site-lisp,
> > but perhaps I do somewhere.
> >  
> > When I start Emacs this warning is the first thing in buffer *Messages*:
> >  
> > Warning: Lisp directory `C:/Emacs-24-2012-07-16/../site-lisp' does not exist.
> 
> It's a bug, I've seen it since a few days ago and will get to fixing
> it when I have time (unless someone beats me to it).

The reason for this are the changes made by Glenn in revision 108939.
Prior to those changes, when load-path was determined by the value of
EMACSLOADPATH environment variable, the resulting list was never
checked to verify that every directory there exists.  Now load-path
determined that way _is_ checked.

The other part of the puzzle is that when Emacs on Windows starts, it
always defines EMACSLOADPATH and puts 2 site-lisp directories into it:
one that is in the same tree as the binary, which is supposed to be
version dependent, and another one that is a sibling of the root of
the installed tree, which is supposed to be version-independent.  This
is for compatibility with Emacs on Posix platforms, where we also push
2 directories onto load-path: /usr/local/share/emacs/XY.Z/site-lisp
and /usr/local/share/emacs/site-lisp, respectively.  However, unlike
on Unix, on Windows there's no test whether these directories actually
exist, because this was never a problem with the old code.

But now, that the value of EMACSLOADPATH is being checked, we are
seeing these warnings (unless Emacs is invoked with -Q), and I'm
guessing that most Emacs users on Windows don't have a
version-independent site-lisp directory, so most of them will be
affected.

It would be easy enough to make sure the site-lisp directories exist
before adding them to EMACSLOADPATH on Windows.  But before doing so,
I'd like to understand why was the behavior in init_lread changed so
as to check EMACSLOADPATH in the first place?  And if we do want to
check that, why not exempt the site-lisp directories from the need to
exist, like we do in the case where EMACSLOADPATH is not set?

bug#11959: 24.1.50; Warning: Lisp directory `C:/Emacs-24-2012-07-16/../site-lisp' does not exist.

[Drew Adams]

> The reason for this are the changes made by Glenn in revision 108939.
> Prior to those changes, when load-path was determined by the value of
> EMACSLOADPATH environment variable, the resulting list was never
> checked to verify that every directory there exists.  Now load-path
> determined that way _is_ checked.

May I ask why?  What was the problem with allowing non-existing directories in
`load-path'?  A priori, that sounds like the _right_ thing to do.  And what is
done now whenever one such is encountered - just print a warning?

> The other part of the puzzle is that when Emacs on Windows starts, it
> always defines EMACSLOADPATH and puts 2 site-lisp directories into it:
> one that is in the same tree as the binary, which is supposed to be
> version dependent, and another one that is a sibling of the root of
> the installed tree, which is supposed to be version-independent.

You mean that it does so systematically, in a hard-coded fashion?  Or does it
put those there only if such directories exist?

> This is for compatibility with Emacs on Posix platforms, where we
> also push 2 directories onto load-path:
> /usr/local/share/emacs/XY.Z/site-lisp and
> /usr/local/share/emacs/site-lisp, respectively.  However, unlike
> on Unix, on Windows there's no test whether these directories actually
> exist, because this was never a problem with the old code.
> 
> But now, that the value of EMACSLOADPATH is being checked, we are
> seeing these warnings (unless Emacs is invoked with -Q), and I'm
> guessing that most Emacs users on Windows don't have a
> version-independent site-lisp directory, so most of them will be
> affected.
> 
> It would be easy enough to make sure the site-lisp directories exist
> before adding them to EMACSLOADPATH on Windows.  But before doing so,
> I'd like to understand why was the behavior in init_lread changed so
> as to check EMACSLOADPATH in the first place?

That was my question also.  Just what problem is that trying to solve?

> And if we do want to check that, why not exempt the site-lisp
> directories from the need to exist, like we do in the case where
> EMACSLOADPATH is not set?

Such exceptionalism smacks of fragile design.  What applies to site-lisp might
apply to some other directory as well (tomorrow, if not today).

But I don't have a puppy in this critter crawl - just curious why the change was
made.

bug#11959: 24.1.50; Warning: Lisp directory `C:/Emacs-24-2012-07-16/../site-lisp' does not exist.

[Eli Zaretskii]

> From: "Drew Adams" <drew.adams@oracle.com>
> Cc: <11959@debbugs.gnu.org>
> Date: Wed, 18 Jul 2012 08:20:15 -0700
> 
> > The other part of the puzzle is that when Emacs on Windows starts, it
> > always defines EMACSLOADPATH and puts 2 site-lisp directories into it:
> > one that is in the same tree as the binary, which is supposed to be
> > version dependent, and another one that is a sibling of the root of
> > the installed tree, which is supposed to be version-independent.
> 
> You mean that it does so systematically, in a hard-coded fashion?  Or does it
> put those there only if such directories exist?

On Windows, there's no test for their existence.

> > And if we do want to check that, why not exempt the site-lisp
> > directories from the need to exist, like we do in the case where
> > EMACSLOADPATH is not set?
> 
> Such exceptionalism smacks of fragile design.  What applies to site-lisp might
> apply to some other directory as well (tomorrow, if not today).

Not exactly.  The other directories, 'lisp' and 'leim', are _required_
by Emacs, otherwise it's an incomplete package, i.e. a broken
installation.  By contrast, 'site-lisp' is not required for Emacs
operation, it's a convenience for the user to put her local stuff in.
On Unix, Emacs always tested whether site-lisp exists before adding it
to load-path, with the rationale that it's not required.

bug#11959: 24.1.50; Warning: Lisp directory `C:/Emacs-24-2012-07-16/../site-lisp' does not exist.

[Glenn Morris]

Eli Zaretskii wrote:

> It would be easy enough to make sure the site-lisp directories exist
> before adding them to EMACSLOADPATH on Windows. 

That's probably the simplest solution.
Or make the install create them, as the POSIX installation does.

Actually, my recommendation would be to stop setting EMACSLOADPATH (and
the other EMACS* environment variables...) on MS Windows, similar to
what I recently did for the NS port. The only time I ever hear about
EMACSLOADPATH is it causing subtle problems (eg building one version of
Emacs within another, the recent several uni-mirror related startup
failures, I'm guessing).

> But before doing so, I'd like to understand why was the behavior in
> init_lread changed so as to check EMACSLOADPATH in the first place?

Because I saw no reason to treat EMACSLOADPATH directories differently
to the normal directories. Users should still be warned if they are
missing. (I guess very few people are using EMACSLOADPATH
intentionally though.)

> And if we do want to check that, why not exempt the site-lisp
> directories from the need to exist, like we do in the case where
> EMACSLOADPATH is not set?

Because I assumed people setting EMACSLOADPATH would only include
existing directories, and would want to be warned about missing ones.
I overlooked that MS Windows adds site-lisp directories without checking
they exist. Also there's no clean way to detect what a site-lisp
directory is in the EMACSLOADPATH case (simply checking for site-lisp in
the name is not robust).

bug#11959: 24.1.50; Warning: Lisp directory `C:/Emacs-24-2012-07-16/../site-lisp' does not exist.

[Glenn Morris]

Glenn Morris wrote:

> Eli Zaretskii wrote:
>
>> It would be easy enough to make sure the site-lisp directories exist
>> before adding them to EMACSLOADPATH on Windows. 
>
> That's probably the simplest solution.

PS or stick the warning back (effectively) inside #ifndef WINDOWSNT if
you prefer.

bug#11959: 24.1.50; Warning: Lisp directory `C:/Emacs-24-2012-07-16/../site-lisp' does not exist.

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Cc: 11959@debbugs.gnu.org
> Date: Sun, 29 Jul 2012 19:50:28 -0400
> 
> Eli Zaretskii wrote:
> 
> > It would be easy enough to make sure the site-lisp directories exist
> > before adding them to EMACSLOADPATH on Windows. 
> 
> That's probably the simplest solution.

I will do that, if this is the consensus.  Stefan, Chong, what say
you?

> Or make the install create them, as the POSIX installation does.

The problem happens when you run the uninstalled binary as well.

> Actually, my recommendation would be to stop setting EMACSLOADPATH (and
> the other EMACS* environment variables...) on MS Windows, similar to
> what I recently did for the NS port.

I don't see how this is possible.  Emacs on Windows is built to be
relocatable, because many users install precompiled binaries in any
place they feel like.  So Emacs on Windows must determine its
load-path at run time.  By contrast, the mainline code relies on file
names hardwired into the executable at configure/build time, which is
a non-starter.  What other devices do we have for forcing load-path to
have a specific value, except setting EMACSLOADPATH?  I could, of
course, ifdef away the entire code that does that on lread.c, and put
there a Windows specific code instead, but is that really a better
alternative?

I would encourage to rewrite that code on all platforms to allow
relocation of the binary, since that cannot be bad on any platform.
But until that happens, do you see a better way than set
EMACSLOADPATH?

> > And if we do want to check that, why not exempt the site-lisp
> > directories from the need to exist, like we do in the case where
> > EMACSLOADPATH is not set?
> 
> Because I assumed people setting EMACSLOADPATH would only include
> existing directories, and would want to be warned about missing ones.
> I overlooked that MS Windows adds site-lisp directories without checking
> they exist. Also there's no clean way to detect what a site-lisp
> directory is in the EMACSLOADPATH case (simply checking for site-lisp in
> the name is not robust).

Checking for site-lisp is better than nothing, though.

bug#11959: 24.1.50; Warning: Lisp directory `C:/Emacs-24-2012-07-16/../site-lisp' does not exist.

[Jan Djärv]

30 jul 2012 kl. 05:34 skrev Eli Zaretskii:

>> From: Glenn Morris <rgm@gnu.org>
>> Cc: 11959@debbugs.gnu.org
>> Date: Sun, 29 Jul 2012 19:50:28 -0400
>> 
>> Eli Zaretskii wrote:
>> 
>>> It would be easy enough to make sure the site-lisp directories exist
>>> before adding them to EMACSLOADPATH on Windows. 
>> 
>> That's probably the simplest solution.
> 
> I will do that, if this is the consensus.  Stefan, Chong, what say
> you?
> 
>> Or make the install create them, as the POSIX installation does.
> 
> The problem happens when you run the uninstalled binary as well.
> 
>> Actually, my recommendation would be to stop setting EMACSLOADPATH (and
>> the other EMACS* environment variables...) on MS Windows, similar to
>> what I recently did for the NS port.
> 
> I don't see how this is possible.  Emacs on Windows is built to be
> relocatable, because many users install precompiled binaries in any
> place they feel like.  So Emacs on Windows must determine its
> load-path at run time.  By contrast, the mainline code relies on file
> names hardwired into the executable at configure/build time, which is
> a non-starter.  What other devices do we have for forcing load-path to
> have a specific value, except setting EMACSLOADPATH?  I could, of
> course, ifdef away the entire code that does that on lread.c, and put
> there a Windows specific code instead, but is that really a better
> alternative?

The NS-port is also fully relocatable and determines load-path at runtime.

	Jan D.

bug#11959: 24.1.50; Warning: Lisp directory `C:/Emacs-24-2012-07-16/../site-lisp' does not exist.

[Eli Zaretskii]

> From: Jan Djärv <jan.h.d@swipnet.se>
> Date: Mon, 30 Jul 2012 08:05:04 +0200
> Cc: Glenn Morris <rgm@gnu.org>,
>  Stefan Monnier <monnier@iro.umontreal.ca>,
>  Chong Yidong <cyd@gnu.org>,
>  11959@debbugs.gnu.org
> 
> 
> >> Actually, my recommendation would be to stop setting EMACSLOADPATH (and
> >> the other EMACS* environment variables...) on MS Windows, similar to
> >> what I recently did for the NS port.
> > 
> > I don't see how this is possible.  Emacs on Windows is built to be
> > relocatable, because many users install precompiled binaries in any
> > place they feel like.  So Emacs on Windows must determine its
> > load-path at run time.  By contrast, the mainline code relies on file
> > names hardwired into the executable at configure/build time, which is
> > a non-starter.  What other devices do we have for forcing load-path to
> > have a specific value, except setting EMACSLOADPATH?  I could, of
> > course, ifdef away the entire code that does that on lread.c, and put
> > there a Windows specific code instead, but is that really a better
> > alternative?
> 
> The NS-port is also fully relocatable and determines load-path at runtime.

Yeah, with "#ifdef HAVE_NS" around it.  That's what I meant by
"Windows specific code" above.

bug#11959: 24.1.50; Warning: Lisp directory `C:/Emacs-24-2012-07-16/../site-lisp' does not exist.

[Glenn Morris]

Eli Zaretskii wrote:

> Emacs on Windows is built to be relocatable, because many users
> install precompiled binaries in any place they feel like. So Emacs on
> Windows must determine its load-path at run time. By contrast, the
> mainline code relies on file names hardwired into the executable at
> configure/build time, which is a non-starter. What other devices do we
> have for forcing load-path to have a specific value, except setting
> EMACSLOADPATH?

Define MS functions analogous to ns_etc_directory, ns_exec_path, and
ns_load_path. For simplicity/consistency, we can rename these to
something like "platform_etc_directory" etc (or reloc_, or whatever_),
then NS and MS Windows can use the same function names. Then we can just
replace the #ifdef HAVE_NS in callproc.c with #if defined HAVS_NS ||
defined WINDOWSNT.

bug#11959: 24.1.50; Warning: Lisp directory `C:/Emacs-24-2012-07-16/../site-lisp' does not exist.

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>,  Chong Yidong <cyd@gnu.org>,  11959@debbugs.gnu.org
> Date: Mon, 30 Jul 2012 02:43:47 -0400
> 
> Eli Zaretskii wrote:
> 
> > Emacs on Windows is built to be relocatable, because many users
> > install precompiled binaries in any place they feel like. So Emacs on
> > Windows must determine its load-path at run time. By contrast, the
> > mainline code relies on file names hardwired into the executable at
> > configure/build time, which is a non-starter. What other devices do we
> > have for forcing load-path to have a specific value, except setting
> > EMACSLOADPATH?
> 
> Define MS functions analogous to ns_etc_directory, ns_exec_path, and
> ns_load_path. For simplicity/consistency, we can rename these to
> something like "platform_etc_directory" etc (or reloc_, or whatever_),
> then NS and MS Windows can use the same function names. Then we can just
> replace the #ifdef HAVE_NS in callproc.c with #if defined HAVS_NS ||
> defined WINDOWSNT.

That's "Windows specific code" I alluded to.  If that's what we want,
I have no problems with that.  I just thought #ifdef's would be ugly.

bug#11959: 24.1.50; Warning: Lisp directory `C:/Emacs-24-2012-07-16/../site-lisp' does not exist.

[Glenn Morris]

Eli Zaretskii wrote:

> That's "Windows specific code" I alluded to.  If that's what we want,
> I have no problems with that.  I just thought #ifdef's would be ugly.

I think it's a lot less ugly than one part of a program's startup
communicating with another part by means of a permanent change to the
program's environment.

bug#11959: 24.1.50; Warning: Lisp directory `C:/Emacs-24-2012-07-16/../site-lisp' does not exist.

[Glenn Morris]

Eli Zaretskii wrote:

> I would encourage to rewrite that code on all platforms to allow
> relocation of the binary, since that cannot be bad on any platform.

I think this would be pretty straightforward to implement (bascially
just copy what the NS port does), if considered generally desirable.

bug#11959: 24.1.50; Warning: Lisp directory `C:/Emacs-24-2012-07-16/../site-lisp' does not exist.

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>,  Chong Yidong <cyd@gnu.org>,  11959@debbugs.gnu.org
> Date: Mon, 30 Jul 2012 11:11:09 -0400
> 
> Eli Zaretskii wrote:
> 
> > I would encourage to rewrite that code on all platforms to allow
> > relocation of the binary, since that cannot be bad on any platform.
> 
> I think this would be pretty straightforward to implement (bascially
> just copy what the NS port does), if considered generally desirable.

I'm waiting for Stefan and Chong to chime in.

bug#11959: 24.1.50; Warning: Lisp directory `C:/Emacs-24-2012-07-16/../site-lisp' does not exist.

[Stefan Monnier]

>> > I would encourage to rewrite that code on all platforms to allow
>> > relocation of the binary, since that cannot be bad on any platform.
>> I think this would be pretty straightforward to implement (bascially
>> just copy what the NS port does), if considered generally desirable.
> I'm waiting for Stefan and Chong to chime in.

Sounds fine to me,


        Stefan

bug#11959: 24.1.50; Warning: Lisp directory `C:/Emacs-24-2012-07-16/../site-lisp' does not exist.

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Cc: Glenn Morris <rgm@gnu.org>, cyd@gnu.org, 11959@debbugs.gnu.org
> Date: Wed, 01 Aug 2012 16:32:01 -0400
> 
> >> > I would encourage to rewrite that code on all platforms to allow
> >> > relocation of the binary, since that cannot be bad on any platform.
> >> I think this would be pretty straightforward to implement (bascially
> >> just copy what the NS port does), if considered generally desirable.
> > I'm waiting for Stefan and Chong to chime in.
> 
> Sounds fine to me,

Glenn, are you working on this, or should I fix the w32 build in the
meantime?

bug#11959: 24.1.50; Warning: Lisp directory `C:/Emacs-24-2012-07-16/../site-lisp' does not exist.

[Glenn Morris]

Eli Zaretskii wrote:

> Glenn, are you working on this, or should I fix the w32 build in the
> meantime?

I'm not planning to work on it any time soon (and I don't think it would
affect w32 anyway).

bug#11959: 24.1.50; Warning: Lisp directory `C:/Emacs-24-2012-07-16/../site-lisp' does not exist.

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Cc: Glenn Morris <rgm@gnu.org>, cyd@gnu.org, 11959@debbugs.gnu.org
> Date: Wed, 01 Aug 2012 16:32:01 -0400
> 
> >> > I would encourage to rewrite that code on all platforms to allow
> >> > relocation of the binary, since that cannot be bad on any platform.
> >> I think this would be pretty straightforward to implement (bascially
> >> just copy what the NS port does), if considered generally desirable.
> > I'm waiting for Stefan and Chong to chime in.
> 
> Sounds fine to me,

Upon studying the NS code, I didn't like it.  It's too
platform-specific (so would require implementing the same stuff at
least one more time), and it bypasses most of the code in emacs.c,
callproc.c, and lread.c that sets up the various VFOO_directory and
VBAR_path variables on Posix platforms, and IMO for no good reason,
since the logic of that code is generally correct and useful.

So instead I made a few changes in decode_env_path that allow use of a
"magical" prefix in strings defined in epaths.h.  That prefix is
expanded at startup time into the root of the Emacs tree, either
installation tree or the source tree.  (Some changes to support these
were needed in w32.c and nt/paths.h, as well.)  This allows to leave
the logic in init_lread and other similar places intact, and still
DTRT both in installed and uninstalled Emacs.  The advantage of this
is that making this work on all the other platforms is (I hope)
_really_ trivial; we just need to agree on some suitable replacement
for the %emacs_dir% thingy.

The changes I installed (in trunk revision 109429) are for MS-Windows
only for now.

This eliminates the annoying warning that started this bug report, and
I'm therefore marking this bug as done.