bug#50666: 28.0.50; Fix native compilation on Cygwin

[Eli Zaretskii <eliz@gnu.org>]

> Cc: Stromeko@nexgo.de, 50666@debbugs.gnu.org
> From: Ken Brown <kbrown@cornell.edu>
> Date: Thu, 23 Sep 2021 10:20:19 -0400
> 
> > Is it really necessary to rebase the *.eln files before each startup?
> > Isn't it enough to rebase each of the .eln files just once, when it is
> > produced?  If indeed this is needed every time, can you explain why?
> 
> We have to distinguish between system libraries and user libraries.  Libraries 
> installed by Cygwin packages are system libraries.  The *.eln files in 
> ~/.emacs.d/eln-cache are user libraries.  Cygwin maintains a database of all 
> system libraries and their base addresses.  Whenever a Cygwin package is 
> installed or updated, Cygwin rebases all system libraries and updates the database.
> 
> Each time Emacs starts, it has no way of knowing whether the system libraries 
> have been rebased since the last time Emacs was run.  So the user's *.eln files 
> could now have base address conflicts with system libraries.  Rebasing the *.eln 
> files fixes this problem.

What do you mean by "system libraries" here?  Does it, for example,
include the DLLs distributed in the Cygwin port of libpng or libjpeg?
Or does that include only the basic libraries: the Cygwin DLL, the C
runtime, etc.?

> > The non-preloaded *.eln files are all loaded by native-elisp-load, so
> > I guess the rebase should be launched from there?  The preloaded *.eln
> > files are loaded in pdumper.c:dump_do_dump_relocation, but do we need
> > to support non-rebased preloaded *.eln files?
> 
> The preloaded *.eln files will be installed by Cygwin's package manager when 
> emacs is installed.  They are therefore system libraries and are automatically 
> rebased as needed.

If the only problem is with non-preloaded *.eln files, why not rebase
them on the fly, when they are loaded.  That is, run the 'rebase'
command from the native-elisp-load function, before it actually loads
the file.  User libraries are never loaded during startup, only when
some Lisp requires them.

> > When an updated .eln file is produced for .eln that is loaded into
> > some running Emacs, Emacs on Windows renames the original .eln to
> > avoid a similar problem.  
> 
> I hadn't thought of this issue.  We may have to use a similar technique...
> 
> > Can't you use the same technique, to avoid
> > the need of rebasing on each start?
> 
> ...but it wouldn't eliminate the need for rebasing at each start for the reasons 
> explained above.

Perhaps that need could be eliminated after all, see above.

> >  Please note that users could
> > place *.eln files in unusual locations (and customize
> > native-comp-eln-load-path to reflect that), so finding _all_ of the
> > relevant *.eln files from a shell script might not be easy.  In fact,
> > even without customizing the load-path, I don't think I understand how
> > will that script you propose know where to find all the *.eln files.
> 
> The current proposal that Achim and I are looking at would require each user to 
> maintain a file ~/.config/rebase/dynpath.d/emacs containing a list of 
> directories where the .eln files can be found.  By default, this file would 
> contain one line, which is the path to the standard eln-cache directory.  Users 
> who customize native-comp-eln-load-path would have to modify that file accordingly.

That's tough on users, because Emacs by default automatically compiles
every .el file it loads into .eln, if there's no up-to-date .eln file
already, and the compilation runs in the background (by forking
additional Emacs sub-processes that run in batch mode).  In addition,
the native-compilation process sometimes decides that it needs to
create a special "trampoline" .eln file (if you want to know why, I'm
sure Andrea can explain) that correspond to parts of the *.el files.
The upshot is that users may not even be aware that new *.eln files
have been created as part of their session, and may not know their
names.  Unless the automatic rebase process, which runs from
native-elisp-load, will also update the file in dynpath.d, I don't see
how users could maintain such a database by hand in practice.

There's one more aspect that you should be aware of.  A single file
FOO.el could give birth to several different .eln files, for example
if they are compiled by different Emacs binaries and/or from different
source directories and/or from somewhat different versions of FOO.el.
For example, I now have 3 different versions of .eln files
corresponding to window.el, in the same directory:

   window-0d1b8b93-3370bedb.eln
   window-0d1b8b93-7d08b7b4.eln
   window-0d1b8b93-f8fc9683.eln

This makes the job of maintaining the database by hand even harder and
more error-prone.

> Finally, as a side note, I don't think it would be a tragedy if this just turns 
> out to be too complicated and we have to disable native compilation on 32-bit 
> Cygwin.  The Cygwin home page at https://cygwin.com/ already contains the following:
> 
>    Address space is a very limiting factor for Cygwin. These days, a full
>    32 bit Cygwin distro is not feasible anymore, and will in all likelihood
>    fail in random places due to an issue with the fork(2) system call.
> 
>    Therefore we recommend using 32 bit Cygwin only in limited scenarios, with
>    only a minimum of necessary packages installed, and only if there's no way
>    to run 64 bit Cygwin instead.

My point is that maybe we should make that decision already, before
burning too much time and energy on it.  Maybe you should ask on the
Cygwin list whether somebody will object to making 32-bit Cygwin Emacs
a second-class citizen.

Thanks.