bug#58318: 28.2; Emacs installed from package won't work with MinGW

bug#58318: 28.2; Emacs installed from package won't work with MinGW

[Bartosz Bubak]

[-- Attachment #1: Type: text/plain, Size: 5586 bytes --]

When installing Emacs using the installer (emacs-28.2-installer.exe),
by default it adds the compiler and libraries (gcc and libgccjit)
to the ./bin directory without prompting the user for his opinion.

The problem appears when another compiler is already installed in the
system, eg. from MinGW. Emacs then tries to use the libraries from PATH
instead of those installed in its subdirectories:

Warning (comp): c: /tools/emacs/share/emacs/28.2/lisp/org/org-entities.el:
Error: Internal native compiler error failed to compile
Warning (comp): C: \\ ProgramData \\ chocolatey \\ lib \\ mingw \\ tools \\
install \\ mingw64 \\ bin \\ libgccjit-0.dll: error: error invoking gcc
driver

When I tried to find a solution to the problem, I found something like
this:

M-: (executable-find "gcc") RET
"c:/ProgramData/chocolatey/bin/gcc.exe
M-: (executable-find "as") RET
"c:/ProgramData/chocolatey/bin/as.exe"
etc, etc

But  should be:
C:\Program Files\Emacs\emacs-28.2\bin\gcc.exe
C:\Program Files\Emacs\emacs-28.2\bin\as.exe

The only solution I have found so far is to uninstall the "global"
MinGW, then emacs uses the embedded libraries and everything is fine.


In GNU Emacs 28.2 (build 2, x86_64-w64-mingw32)
 of 2022-09-13 built on AVALON
Windowing system distributor 'Microsoft Corp.', version 10.0.22000
System Description: Microsoft Windows 10 Pro (v10.0.2009.22000.978)

Configured using:
 'configure --with-modules --without-dbus --with-native-compilation
 --without-compress-install CFLAGS=-O2'

Configured features:
ACL GIF GMP GNUTLS HARFBUZZ JPEG JSON LCMS2 LIBXML2 MODULES NATIVE_COMP
NOTIFY W32NOTIFY PDUMPER PNG RSVG SOUND THREADS TIFF TOOLKIT_SCROLL_BARS
XPM ZLIB

(NATIVE_COMP present but libgccjit not available)

Important settings:
  value of $LANG: PLK
  locale-coding-system: cp1250

Major mode: Org

Minor modes in effect:
  override-global-mode: t
  global-eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  indent-tabs-mode: t
  transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug sendmail org-mobile ffap tmm diary-lib
diary-loaddefs cal-iso face-remap org-agenda org-refile org-clock
mule-util cal-move org-element avl-tree generator ol-eww eww xdg
url-queue mm-url ol-rmail ol-mhe ol-irc ol-info ol-gnus nnselect
gnus-search eieio-opt speedbar ezimage dframe gnus-art mm-uu mml2015
mm-view mml-smime smime dig gnus-sum shr kinsoku svg dom gnus-group
gnus-undo gnus-start gnus-dbus dbus xml gnus-cloud nnimap nnmail
mail-source utf7 netrc nnoo parse-time gnus-spec gnus-int gnus-range
message rfc822 mml mml-sec epa derived mm-decode mm-bodies mm-encode
mail-parse rfc2231 mailabbrev gmm-utils mailheader gnus-win gnus
nnheader gnus-util rmail rmail-loaddefs rfc2047 rfc2045 ietf-drums
mail-utils mm-util mail-prsvr ol-docview doc-view jka-compr image-mode
exif ol-bibtex ol-bbdb ol-w3m ol-doi org-link-doi dired-aux dired
dired-loaddefs which-key lsp-pyright lsp-mode comp comp-cstr warnings
lsp-protocol yasnippet xref project tree-widget wid-edit spinner pcase
network-stream puny nsm rmc markdown-mode color thingatpt lv inline
imenu ht filenotify f f-shortdoc shortdoc s ewoc epg rfc6068 epg-config
dash compile text-property-search org ob ob-tangle ob-ref ob-lob
ob-table ob-exp org-macro org-footnote org-src ob-comint org-pcomplete
pcomplete comint ansi-color ring org-list org-faces org-entities
noutline outline org-version ob-emacs-lisp ob-core ob-eval org-table
oc-basic bibtex iso8601 time-date ol rx org-keys oc org-compat advice
org-macs org-loaddefs format-spec find-func cal-menu calendar
cal-loaddefs command-log-mode cl-extra help-mode use-package
use-package-ensure use-package-delight use-package-diminish
use-package-bind-key bind-key easy-mmode use-package-core finder-inf
edmacro kmacro wombat-theme info package browse-url url url-proxy
url-privacy url-expand url-methods url-history url-cookie url-domsuf
url-util mailcap url-handlers url-parse auth-source cl-seq eieio
eieio-core cl-macs eieio-loaddefs password-cache json subr-x map
url-vars seq byte-opt gv bytecomp byte-compile cconv cl-loaddefs cl-lib
iso-transl tooltip eldoc paren electric uniquify ediff-hook vc-hooks
lisp-float-type elisp-mode mwheel dos-w32 ls-lisp disp-table
term/w32-win w32-win w32-vars term/common-win tool-bar dnd fontset image
regexp-opt fringe tabulated-list replace newcomment text-mode lisp-mode
prog-mode register page tab-bar menu-bar rfn-eshadow isearch easymenu
timer select scroll-bar mouse jit-lock font-lock syntax font-core
term/tty-colors frame minibuffer cl-generic cham georgian utf-8-lang
misc-lang vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms
cp51932 hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese composite emoji-zwj charscript charprop case-table
epa-hook jka-cmpr-hook help simple abbrev obarray cl-preloaded nadvice
button loaddefs faces cus-face macroexp files window text-properties
overlay sha1 md5 base64 format env code-pages mule custom widget
hashtable-print-readable backquote threads w32notify w32 lcms2 multi-tty
make-network-process native-compile emacs)

Memory information:
((conses 16 497677 139781)
 (symbols 48 38406 9)
 (strings 32 164117 43025)
 (string-bytes 1 4805171)
 (vectors 16 75523)
 (vector-slots 8 1608377 199020)
 (floats 8 434 411)
 (intervals 56 3060 1085)
 (buffers 992 27))

[-- Attachment #2: Type: text/html, Size: 7117 bytes --]

bug#58318: 28.2; Emacs installed from package won't work with MinGW

[Eli Zaretskii]

> From: Bartosz Bubak <bartosz.bubak@gmail.com>
> Date: Wed, 5 Oct 2022 18:01:00 +0200
> 
> When installing Emacs using the installer (emacs-28.2-installer.exe),
> by default it adds the compiler and libraries (gcc and libgccjit)
> to the ./bin directory without prompting the user for his opinion.
> 
> The problem appears when another compiler is already installed in the
> system, eg. from MinGW. Emacs then tries to use the libraries from PATH
> instead of those installed in its subdirectories:
> 
> Warning (comp): c: /tools/emacs/share/emacs/28.2/lisp/org/org-entities.el:
> Error: Internal native compiler error failed to compile
> Warning (comp): C: \\ ProgramData \\ chocolatey \\ lib \\ mingw \\ tools \\
> install \\ mingw64 \\ bin \\ libgccjit-0.dll: error: error invoking gcc
> driver
> 
> When I tried to find a solution to the problem, I found something like
> this:
> 
> M-: (executable-find "gcc") RET
> "c:/ProgramData/chocolatey/bin/gcc.exe
> M-: (executable-find "as") RET
> "c:/ProgramData/chocolatey/bin/as.exe"
> etc, etc
> 
> But  should be:
> C:\Program Files\Emacs\emacs-28.2\bin\gcc.exe
> C:\Program Files\Emacs\emacs-28.2\bin\as.exe
> 
> The only solution I have found so far is to uninstall the "global"
> MinGW, then emacs uses the embedded libraries and everything is fine.
> 
> In GNU Emacs 28.2 (build 2, x86_64-w64-mingw32)
>  of 2022-09-13 built on AVALON
> Windowing system distributor 'Microsoft Corp.', version 10.0.22000
> System Description: Microsoft Windows 10 Pro (v10.0.2009.22000.978)
> 
> Configured using:
>  'configure --with-modules --without-dbus --with-native-compilation
>  --without-compress-install CFLAGS=-O2'
> 
> Configured features:
> ACL GIF GMP GNUTLS HARFBUZZ JPEG JSON LCMS2 LIBXML2 MODULES NATIVE_COMP
> NOTIFY W32NOTIFY PDUMPER PNG RSVG SOUND THREADS TIFF TOOLKIT_SCROLL_BARS
> XPM ZLIB
> 
> (NATIVE_COMP present but libgccjit not available)
> 
> Important settings:
>   value of $LANG: PLK
>   locale-coding-system: cp1250

Corwin, are you reading this?

I admit I don't have a clear idea of why the problem happens.  Is the
libgccjit and/or GCC and/or Binutils distributed by chocolatey somehow
incompatible with the ones you include in the installer?  Because if
they are compatible, just removing the GCC/Binutils stuff bundled with
the Emacs installer should have solved the issue (and then providing
an option in the installer not to install the bundled GCC would be a
way towards solving this).  And yet the OP seems to say (AFAIU) that
this didn't help, and only uninstalling the chocolatey GCC/Binutils
did.  We cannot possibly ask users to uninstall their existing
development environment when installing Emacs.

I think someone should try installing the chocolatey distribution and
see whether the binaries from the GNU FTP site can work with its
libgccjit.  Because I'm not sure I understand what happens in this
case, even though I asked several times.  If indeed there's
incompatibility, I'd be interested to know why (I have no idea how
chocolatey builds its GCC).  If that is not solvable, we should
probably say that people with chocolatey installation should not
install Emacs binaries with native-compilation enabled.

bug#58318: 28.2; Emacs installed from package won't work with MinGW

[Corwin Brust]

On Thu, Oct 6, 2022 at 12:44 AM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: Bartosz Bubak <bartosz.bubak@gmail.com>
> > Date: Wed, 5 Oct 2022 18:01:00 +0200
> >
> > When installing Emacs using the installer (emacs-28.2-installer.exe),
> > by default it adds the compiler and libraries (gcc and libgccjit)
> > to the ./bin directory without prompting the user for his opinion.

I'm confused.  With one exception, I do not think that we provide GCC
with Emacs binaries for windows.

We *do* provide libgcc_s_seh-1.dll but we do not provide, e.g.
gcc.exe, as.exe, etc.

> >
> > The problem appears when another compiler is already installed in the
> > system, eg. from MinGW. Emacs then tries to use the libraries from PATH
> > instead of those installed in its subdirectories:
> >
> > Warning (comp): c: /tools/emacs/share/emacs/28.2/lisp/org/org-entities.el:
> > Error: Internal native compiler error failed to compile
> > Warning (comp): C: \\ ProgramData \\ chocolatey \\ lib \\ mingw \\ tools \\
> > install \\ mingw64 \\ bin \\ libgccjit-0.dll: error: error invoking gcc
> > driver

It does make sense to me that having two different versions of GCC
both findable on the Windows path could be problematic (although I
don't see how the Emacs installer could be responsible for installing
any of them).


> >
> > When I tried to find a solution to the problem, I found something like
> > this:
> >
> > M-: (executable-find "gcc") RET
> > "c:/ProgramData/chocolatey/bin/gcc.exe
> > M-: (executable-find "as") RET
> > "c:/ProgramData/chocolatey/bin/as.exe"
> > etc, etc
> >
> > But  should be:
> > C:\Program Files\Emacs\emacs-28.2\bin\gcc.exe
> > C:\Program Files\Emacs\emacs-28.2\bin\as.exe

That's very odd -- looking at the "install" folder created when I
built and packaged the emacs-28 binaries I can't find either of those
files:

  corwi@Avalon MINGW64 /d/emacs-build/install/emacs-28.2
  $ (cd /d/emacs-build/install/emacs-28.2; ls -Rl | grep 'as.exe')

  corwi@Avalon MINGW64 /d/emacs-build/install/emacs-28.2
  $ (cd /d/emacs-build/install/emacs-28.2; ls -Rl | grep 'gcc')
  -rwxr-xr-x 1 corwi corwi   84147 Feb 20  2022 libgcc_s_seh-1.dll
  -rw-r--r-- 1 corwi corwi  10282 Sep  6 16:31 gcc.el
  -rw-r--r-- 1 corwi corwi   4680 Sep  6 18:05 gcc.elc

> >
> > The only solution I have found so far is to uninstall the "global"
> > MinGW, then emacs uses the embedded libraries and everything is fine.
> >
> > In GNU Emacs 28.2 (build 2, x86_64-w64-mingw32)
> >  of 2022-09-13 built on AVALON
> > Windowing system distributor 'Microsoft Corp.', version 10.0.22000
> > System Description: Microsoft Windows 10 Pro (v10.0.2009.22000.978)
> >
> > Configured using:
> >  'configure --with-modules --without-dbus --with-native-compilation
> >  --without-compress-install CFLAGS=-O2'
> >
> > Configured features:
> > ACL GIF GMP GNUTLS HARFBUZZ JPEG JSON LCMS2 LIBXML2 MODULES NATIVE_COMP
> > NOTIFY W32NOTIFY PDUMPER PNG RSVG SOUND THREADS TIFF TOOLKIT_SCROLL_BARS
> > XPM ZLIB
> >
> > (NATIVE_COMP present but libgccjit not available)

It looks from this that Emacs doesn't find (a complete) GCC kit for
use with GCC, so your Emacs should be loading ELN files shipped with
the release but not trying to compile any new ones, I think?

Unfortunately, that just makes the "Error: Internal native compiler
error failed to compile" even more of a mystery:

Eli,
If, per the configuration reported when generating this bug report,
Emacs can see libgccjit is not available, then should Emacs still be
trying to native compile org-entitles in this case?

> >
> > Important settings:
> >   value of $LANG: PLK
> >   locale-coding-system: cp1250
>
> Corwin, are you reading this?

Yes, I am now.  Thank you.

>
> I admit I don't have a clear idea of why the problem happens.  Is the
> libgccjit and/or GCC and/or Binutils distributed by chocolatey somehow
> incompatible with the ones you include in the installer?  Because if
> they are compatible, just removing the GCC/Binutils stuff bundled with
> the Emacs installer should have solved the issue (and then providing
> an option in the installer not to install the bundled GCC would be a
> way towards solving this).  And yet the OP seems to say (AFAIU) that
> this didn't help, and only uninstalling the chocolatey GCC/Binutils
> did.  We cannot possibly ask users to uninstall their existing
> development environment when installing Emacs.

I'm not aware of any incompatibilities but I'm not a chocolatey user.
I'll need to do some experimentation.

I'd be happy to add/adjust installer options.  Would we (probably most
simply?) add an option where we can "uncheck" installing all of the
deps?  If not, what else would the new option suppress installing
(beside libgcc_s_seh-1.dll)?

>
> I think someone should try installing the chocolatey distribution and
> see whether the binaries from the GNU FTP site can work with its
> libgccjit.  Because I'm not sure I understand what happens in this
> case, even though I asked several times.  If indeed there's
> incompatibility, I'd be interested to know why (I have no idea how
> chocolatey builds its GCC).  If that is not solvable, we should
> probably say that people with chocolatey installation should not
> install Emacs binaries with native-compilation enabled.

I'll get the machine I'm using to test release binaries going on this
today after work and report back with any success I have
reproducing/researching. Others' findings would be most welcome if
anyone else is experimenting with this too.

bug#58318: 28.2; Emacs installed from package won't work with MinGW

[Lars Ingebrigtsen]

Corwin Brust <corwin@bru.st> writes:

> It looks from this that Emacs doesn't find (a complete) GCC kit for
> use with GCC, so your Emacs should be loading ELN files shipped with
> the release but not trying to compile any new ones, I think?
>
> Unfortunately, that just makes the "Error: Internal native compiler
> error failed to compile" even more of a mystery:

It shouldn't try to compile .el(c) files, but it needs the compiler to
make trampolines to redefine built-in functions.  So a nativecomp Emacs
isn't fully functional if a compiler isn't present.

To check whether this is what's happening in this case, try to say:

(fset 'yes-or-no-p 'y-or-n-p)

You should get a file called something like

subr--trampoline-7965732d6f722d6e6f2d70_yes_or_no_p_0.eln

in your eln-cache directory.  If this leads to that mysterious error
message, then that's probably what's happening here.

bug#58318: 28.2; Emacs installed from package won't work with MinGW

[Eli Zaretskii]

> From: Corwin Brust <corwin@bru.st>
> Date: Thu, 6 Oct 2022 08:09:16 -0500
> Cc: Bartosz Bubak <bartosz.bubak@gmail.com>, 58318@debbugs.gnu.org
> 
> If, per the configuration reported when generating this bug report,
> Emacs can see libgccjit is not available, then should Emacs still be
> trying to native compile org-entitles in this case?

No, it should disable that in startup.el:

    (when (featurep 'native-compile)
      (unless (native-comp-available-p)
        ;; Disable deferred async compilation and trampoline synthesis
        ;; in this session.  This is necessary if libgccjit is not
        ;; available on MS-Windows, but Emacs was built with
        ;; native-compilation support.
        (setq native-comp-deferred-compilation nil
              comp-enable-subr-trampolines nil))

But libgccjit DLL _is_ available in this case, so maybe the test
doesn't discover the problem.

bug#58318: 28.2; Emacs installed from package won't work with MinGW

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Eli Zaretskii <eliz@gnu.org>,  58318@debbugs.gnu.org,  Bartosz Bubak
>  <bartosz.bubak@gmail.com>
> Date: Thu, 06 Oct 2022 15:30:14 +0200
> 
> Corwin Brust <corwin@bru.st> writes:
> 
> > It looks from this that Emacs doesn't find (a complete) GCC kit for
> > use with GCC, so your Emacs should be loading ELN files shipped with
> > the release but not trying to compile any new ones, I think?
> >
> > Unfortunately, that just makes the "Error: Internal native compiler
> > error failed to compile" even more of a mystery:
> 
> It shouldn't try to compile .el(c) files, but it needs the compiler to
> make trampolines to redefine built-in functions.  So a nativecomp Emacs
> isn't fully functional if a compiler isn't present.

No, the last conclusion incorrect.  See my other mail in this thread.

bug#58318: 28.2; Emacs installed from package won't work with MinGW

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> It shouldn't try to compile .el(c) files, but it needs the compiler to
>> make trampolines to redefine built-in functions.  So a nativecomp Emacs
>> isn't fully functional if a compiler isn't present.
>
> No, the last conclusion incorrect.  See my other mail in this thread.

I'm sorry, I don't follow you.  If trampolines can't be installed, then
Emacs isn't fully functional, because you can't say

(fset 'yes-or-no-p 'y-or-n-p)

and have that be respected.  I.e., the non-functional bit is about
redefinitions of built-in functions, which is pretty basic functionality
in Emacs.

bug#58318: 28.2; Emacs installed from package won't work with MinGW

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: corwin@bru.st,  bartosz.bubak@gmail.com,  58318@debbugs.gnu.org
> Date: Fri, 07 Oct 2022 13:42:15 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> It shouldn't try to compile .el(c) files, but it needs the compiler to
> >> make trampolines to redefine built-in functions.  So a nativecomp Emacs
> >> isn't fully functional if a compiler isn't present.
> >
> > No, the last conclusion incorrect.  See my other mail in this thread.
> 
> I'm sorry, I don't follow you.  If trampolines can't be installed, then
> Emacs isn't fully functional, because you can't say
> 
> (fset 'yes-or-no-p 'y-or-n-p)
> 
> and have that be respected.  I.e., the non-functional bit is about
> redefinitions of built-in functions, which is pretty basic functionality
> in Emacs.

Maybe there's a misunderstanding of what you meant by "if a compiler
isn't present".  By "the compiler" do you mean libgccjit, or is it GCC
and Binutils (or maybe all 3 together)?  IOW, are you talking about
the ability to load existing *.eln files, or are you talking about the
ability to both load existing *.eln files and produce new ones?

The startup code currently detects that libgccjit is unavailable or
cannot be loaded, and if so, disables all the aspects of
native-compilation: both JIT compilation of *.el and production of the
trampolines.  I'm not aware that when we disable those two, we get
Emacs that is not "fully functional".

Andrea, am I missing something?

The problem in this bug is that libgccjit _is_ available, but somehow
is not functional when actually used.  (The details are still sketchy
and not understood well enough.)  This situation might not be
supported yet, but when we understand it well enough, we should make
Emacs behave the same as when libgccjit is unavailable (perhaps with
some more specific message in *Messages*), because nothing else makes
sense.

bug#58318: 28.2; Emacs installed from package won't work with MinGW

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> I'm sorry, I don't follow you.  If trampolines can't be installed, then
>> Emacs isn't fully functional, because you can't say
>> 
>> (fset 'yes-or-no-p 'y-or-n-p)
>> 
>> and have that be respected.  I.e., the non-functional bit is about
>> redefinitions of built-in functions, which is pretty basic functionality
>> in Emacs.
>
> Maybe there's a misunderstanding of what you meant by "if a compiler
> isn't present".  By "the compiler" do you mean libgccjit, or is it GCC
> and Binutils (or maybe all 3 together)?  IOW, are you talking about
> the ability to load existing *.eln files, or are you talking about the
> ability to both load existing *.eln files and produce new ones?

I'm talking about trampolines, nothing else.

> The startup code currently detects that libgccjit is unavailable or
> cannot be loaded, and if so, disables all the aspects of
> native-compilation: both JIT compilation of *.el and production of the
> trampolines.  I'm not aware that when we disable those two, we get
> Emacs that is not "fully functional".

If native compilation is disabled in a native-compiled Emacs, then 

(fset 'yes-or-no-p 'y-or-n-p)

doesn't work (for calls to `yes-or-no-p' in native-compiled code).
That's what I meant by "not fully functional".

bug#58318: 28.2; Emacs installed from package won't work with MinGW

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Andrea Corallo <akrl@sdf.org>,  corwin@bru.st,  bartosz.bubak@gmail.com,
>   58318@debbugs.gnu.org
> Date: Fri, 07 Oct 2022 14:04:57 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Maybe there's a misunderstanding of what you meant by "if a compiler
> > isn't present".  By "the compiler" do you mean libgccjit, or is it GCC
> > and Binutils (or maybe all 3 together)?  IOW, are you talking about
> > the ability to load existing *.eln files, or are you talking about the
> > ability to both load existing *.eln files and produce new ones?
> 
> I'm talking about trampolines, nothing else.

Trampoline generation requires all the 3 components to be present,
AFAIK.  Andrea, am I right?

> > The startup code currently detects that libgccjit is unavailable or
> > cannot be loaded, and if so, disables all the aspects of
> > native-compilation: both JIT compilation of *.el and production of the
> > trampolines.  I'm not aware that when we disable those two, we get
> > Emacs that is not "fully functional".
> 
> If native compilation is disabled in a native-compiled Emacs, then 
> 
> (fset 'yes-or-no-p 'y-or-n-p)
> 
> doesn't work (for calls to `yes-or-no-p' in native-compiled code).
> That's what I meant by "not fully functional".

If it indeed doesn't work (and I wasn't aware it didn't work), we
should try fixing it, if that is feasible.

bug#58318: 28.2; Emacs installed from package won't work with MinGW

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> If it indeed doesn't work (and I wasn't aware it didn't work), we
> should try fixing it, if that is feasible.

It's previously been suggested that we might pre-generate all the
trampolines for binary distributions like this (but of course not load
them until needed).

The trampolines are smallish (16K per file on Ubuntu of which most is
nul bytes, at least), so it should be feasible from a distribution point
of view.

I forget whether the conclusion was that is was feasible or not, though.

bug#58318: 28.2; Emacs installed from package won't work with MinGW

[Andrea Corallo]

[-- Attachment #1: Type: text/plain, Size: 2208 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Lars Ingebrigtsen <larsi@gnus.org>
>> Cc: Andrea Corallo <akrl@sdf.org>,  corwin@bru.st,  bartosz.bubak@gmail.com,
>>   58318@debbugs.gnu.org
>> Date: Fri, 07 Oct 2022 14:04:57 +0200
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > Maybe there's a misunderstanding of what you meant by "if a compiler
>> > isn't present".  By "the compiler" do you mean libgccjit, or is it GCC
>> > and Binutils (or maybe all 3 together)?  IOW, are you talking about
>> > the ability to load existing *.eln files, or are you talking about the
>> > ability to both load existing *.eln files and produce new ones?
>> 
>> I'm talking about trampolines, nothing else.
>
> Trampoline generation requires all the 3 components to be present,
> AFAIK.  Andrea, am I right?

AFAIU only libgccjit and Binutils are necessary, but libgccjit *is* GCC
(in the sense another frontend fo the GNU Compiler Collection).  I
*think* gcc the binary (read the C frontend) should not be required.
But I don't know how distros package libgccjit and gcc, there might be
some dendency I'm not aware of.

>> > The startup code currently detects that libgccjit is unavailable or
>> > cannot be loaded, and if so, disables all the aspects of
>> > native-compilation: both JIT compilation of *.el and production of the
>> > trampolines.  I'm not aware that when we disable those two, we get
>> > Emacs that is not "fully functional".
>> 
>> If native compilation is disabled in a native-compiled Emacs, then 
>> 
>> (fset 'yes-or-no-p 'y-or-n-p)
>> 
>> doesn't work (for calls to `yes-or-no-p' in native-compiled code).
>> That's what I meant by "not fully functional".
>
> If it indeed doesn't work (and I wasn't aware it didn't work), we
> should try fixing it, if that is feasible.

Yes because `yes-or-no-p' is a primitive, so with no trampolines its
redefinition is not functional.

A quick ad-hoc fix for `yes-or-no-p' is attached.  It does not have a
perf impact as `yes-or-no-p' will have to wait for the user input
anyway, if okay I can push it.

Oherwise another strategy would be to disable direct calls from lisp
native code into primitives on Windows, this indeed has a performance
impact.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Add-yes-or-no-p-to-native-comp-never-optimize-functi.patch --]
[-- Type: text/x-diff, Size: 975 bytes --]

From a6d736d532e20b6763a7ff1995f952fc293886dd Mon Sep 17 00:00:00 2001
From: Andrea Corallo <akrl@sdf.org>
Date: Fri, 7 Oct 2022 12:28:51 +0000
Subject: [PATCH] * Add `yes-or-no-p' to
 `native-comp-never-optimize-functions'.

* lisp/emacs-lisp/comp.el (native-comp-never-optimize-functions): Add
`yes-or-no-p'.
---
 lisp/emacs-lisp/comp.el | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lisp/emacs-lisp/comp.el b/lisp/emacs-lisp/comp.el
index 5ee10fcbca2..2812e53bcc2 100644
--- a/lisp/emacs-lisp/comp.el
+++ b/lisp/emacs-lisp/comp.el
@@ -104,7 +104,7 @@ native-comp-never-optimize-functions
   '(;; The following two are mandatory for Emacs to be working
     ;; correctly (see comment in `advice--add-function'). DO NOT
     ;; REMOVE.
-    macroexpand rename-buffer)
+    macroexpand rename-buffer yes-or-no-p)
   "Primitive functions to exclude from trampoline optimization."
   :type '(repeat symbol)
   :version "28.1")
-- 
2.35.1.577.g74cc1aa55f

bug#58318: 28.2; Emacs installed from package won't work with MinGW

[Lars Ingebrigtsen]

Andrea Corallo <akrl@sdf.org> writes:

> A quick ad-hoc fix for `yes-or-no-p' is attached.  It does not have a
> perf impact as `yes-or-no-p' will have to wait for the user input
> anyway, if okay I can push it.

There's nothing special about yes-or-no-p -- I only used that as an
example.  The same problem exists for (almost) all built-in functions.

> Oherwise another strategy would be to disable direct calls from lisp
> native code into primitives on Windows, this indeed has a performance
> impact.

Yes, that would not be ideal.

bug#58318: 28.2; Emacs installed from package won't work with MinGW

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: Lars Ingebrigtsen <larsi@gnus.org>, corwin@bru.st, bartosz.bubak@gmail.com,
>         58318@debbugs.gnu.org
> Date: Fri, 07 Oct 2022 12:35:52 +0000
> 
> >> > Maybe there's a misunderstanding of what you meant by "if a compiler
> >> > isn't present".  By "the compiler" do you mean libgccjit, or is it GCC
> >> > and Binutils (or maybe all 3 together)?  IOW, are you talking about
> >> > the ability to load existing *.eln files, or are you talking about the
> >> > ability to both load existing *.eln files and produce new ones?
> >> 
> >> I'm talking about trampolines, nothing else.
> >
> > Trampoline generation requires all the 3 components to be present,
> > AFAIK.  Andrea, am I right?
> 
> AFAIU only libgccjit and Binutils are necessary, but libgccjit *is* GCC
> (in the sense another frontend fo the GNU Compiler Collection).  I
> *think* gcc the binary (read the C frontend) should not be required.
> But I don't know how distros package libgccjit and gcc, there might be
> some dendency I'm not aware of.

I didn't mean gcc, I meant cc1.  But maybe libgccjit can play its
role, I don't know.

> > If it indeed doesn't work (and I wasn't aware it didn't work), we
> > should try fixing it, if that is feasible.
> 
> Yes because `yes-or-no-p' is a primitive, so with no trampolines its
> redefinition is not functional.
> 
> A quick ad-hoc fix for `yes-or-no-p' is attached.  It does not have a
> perf impact as `yes-or-no-p' will have to wait for the user input
> anyway, if okay I can push it.

What about other primitives? fset can be used for more than just this
one.

> Oherwise another strategy would be to disable direct calls from lisp
> native code into primitives on Windows, this indeed has a performance
> impact.

How is this relevant only to Windows?

And what do you mean by "disable direct calls from Lisp native code
into primitives"?  I don't think I understand what this would do in
practice.

bug#58318: 28.2; Emacs installed from package won't work with MinGW

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> How is this relevant only to Windows?

It's relevant to our binary Windows distribution.

GNU/Linux distributions will make gcc (and the rest) a prerequisite to
installing the emacs-nativecomp package (or whatever they'll be calling
it), so it's not an issue there.

bug#58318: 28.2; Emacs installed from package won't work with MinGW

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <akrl@sdf.org>
>> Cc: Lars Ingebrigtsen <larsi@gnus.org>, corwin@bru.st, bartosz.bubak@gmail.com,
>>         58318@debbugs.gnu.org
>> Date: Fri, 07 Oct 2022 12:35:52 +0000
>>
>> >> > Maybe there's a misunderstanding of what you meant by "if a compiler
>> >> > isn't present".  By "the compiler" do you mean libgccjit, or is it GCC
>> >> > and Binutils (or maybe all 3 together)?  IOW, are you talking about
>> >> > the ability to load existing *.eln files, or are you talking about the
>> >> > ability to both load existing *.eln files and produce new ones?
>> >>
>> >> I'm talking about trampolines, nothing else.
>> >
>> > Trampoline generation requires all the 3 components to be present,
>> > AFAIK.  Andrea, am I right?
>>
>> AFAIU only libgccjit and Binutils are necessary, but libgccjit *is* GCC
>> (in the sense another frontend fo the GNU Compiler Collection).  I
>> *think* gcc the binary (read the C frontend) should not be required.
>> But I don't know how distros package libgccjit and gcc, there might be
>> some dendency I'm not aware of.
>
> I didn't mean gcc, I meant cc1.  But maybe libgccjit can play its
> role, I don't know.
>
>> > If it indeed doesn't work (and I wasn't aware it didn't work), we
>> > should try fixing it, if that is feasible.
>>
>> Yes because `yes-or-no-p' is a primitive, so with no trampolines its
>> redefinition is not functional.
>>
>> A quick ad-hoc fix for `yes-or-no-p' is attached.  It does not have a
>> perf impact as `yes-or-no-p' will have to wait for the user input
>> anyway, if okay I can push it.
>
> What about other primitives? fset can be used for more than just this
> one.
>
>> Oherwise another strategy would be to disable direct calls from lisp
>> native code into primitives on Windows, this indeed has a performance
>> impact.
>
> How is this relevant only to Windows?

Windows is the only system where a native compiled Emacs can start even
if libgccjit is not present.  On GNU/Linux we get and error at load time
from the dynamic linker in case.  As a consequence on GNU/Linux Emacs is
always capable of producing trampolines when needed.

> And what do you mean by "disable direct calls from Lisp native code
> into primitives"?  I don't think I understand what this would do in
> practice.

Native compiled elisp calls directly into primitive functions not to go
through funcall.  For this reason when a primitive is redefined we need
to produce a trampoline in order to forward these calls to the funcall
machinery.  If we disable all of this optimization the issue disappears
but indeed that's not good from a performance point of view.

Indeed the other option is to precompile all trampoline AOT when we know
libgccjit is available.  It is actually very simple with something like:

(mapatoms (λ (f)
            (when (subr-primitive-p (symbol-function f))
              (or (comp-trampoline-search f)
                  (comp-trampoline-compile f)))))

It was not consired worth as trampoline production is very quick, but
might be worth at least for Windows platforms for the discussed reason.

  Andrea

bug#58318: 28.2; Emacs installed from package won't work with MinGW

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Andrea Corallo <akrl@sdf.org>,  corwin@bru.st,  bartosz.bubak@gmail.com,
>   58318@debbugs.gnu.org
> Date: Fri, 07 Oct 2022 15:02:03 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > How is this relevant only to Windows?
> 
> It's relevant to our binary Windows distribution.
> 
> GNU/Linux distributions will make gcc (and the rest) a prerequisite to
> installing the emacs-nativecomp package (or whatever they'll be calling
> it), so it's not an issue there.

I don't think I agree.  Problems with incompatible libgccjit can
happen on other systems as well, so the solution should not be limited
to Windows.

bug#58318: 28.2; Emacs installed from package won't work with MinGW

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> I don't think I agree.  Problems with incompatible libgccjit can
> happen on other systems as well, so the solution should not be limited
> to Windows.

If that happens, it's a packaging problem on those systems.

For self-built Emacs versions, it may be a problem, but the solution is
then to rebuild Emacs for the version of libgccjit you've installed now.

In any case, it's not something we have to worry about on our side.

bug#58318: 28.2; Emacs installed from package won't work with MinGW

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: larsi@gnus.org, corwin@bru.st, bartosz.bubak@gmail.com,
>         58318@debbugs.gnu.org
> Date: Fri, 07 Oct 2022 13:04:55 +0000
> 
> > How is this relevant only to Windows?
> 
> Windows is the only system where a native compiled Emacs can start even
> if libgccjit is not present.  On GNU/Linux we get and error at load time
> from the dynamic linker in case.  As a consequence on GNU/Linux Emacs is
> always capable of producing trampolines when needed.

It could be that libgccjit is loaded but is incompatible or
something.  So I'd prefer a general solution.

> > And what do you mean by "disable direct calls from Lisp native code
> > into primitives"?  I don't think I understand what this would do in
> > practice.
> 
> Native compiled elisp calls directly into primitive functions not to go
> through funcall.  For this reason when a primitive is redefined we need
> to produce a trampoline in order to forward these calls to the funcall
> machinery.  If we disable all of this optimization the issue disappears
> but indeed that's not good from a performance point of view.

How much will performance suffer if we use funcall?

> Indeed the other option is to precompile all trampoline AOT when we know
> libgccjit is available.  It is actually very simple with something like:
> 
> (mapatoms (λ (f)
>             (when (subr-primitive-p (symbol-function f))
>               (or (comp-trampoline-search f)
>                   (comp-trampoline-compile f)))))
> 
> It was not consired worth as trampoline production is very quick, but
> might be worth at least for Windows platforms for the discussed reason.

If calling through funcall is too expensive, I think pre-compiling all
the trampolines would indeed be the best solution, thanks.

bug#58318: 28.2; Emacs installed from package won't work with MinGW

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <akrl@sdf.org>
>> Cc: larsi@gnus.org, corwin@bru.st, bartosz.bubak@gmail.com,
>>         58318@debbugs.gnu.org
>> Date: Fri, 07 Oct 2022 13:04:55 +0000
>> 
>> > How is this relevant only to Windows?
>> 
>> Windows is the only system where a native compiled Emacs can start even
>> if libgccjit is not present.  On GNU/Linux we get and error at load time
>> from the dynamic linker in case.  As a consequence on GNU/Linux Emacs is
>> always capable of producing trampolines when needed.
>
> It could be that libgccjit is loaded but is incompatible or
> something.  So I'd prefer a general solution.
>
>> > And what do you mean by "disable direct calls from Lisp native code
>> > into primitives"?  I don't think I understand what this would do in
>> > practice.
>> 
>> Native compiled elisp calls directly into primitive functions not to go
>> through funcall.  For this reason when a primitive is redefined we need
>> to produce a trampoline in order to forward these calls to the funcall
>> machinery.  If we disable all of this optimization the issue disappears
>> but indeed that's not good from a performance point of view.
>
> How much will performance suffer if we use funcall?

This is the usual 1 milion dollar question, we can run benchmarks but we
are never sure of how much realistic they are.  That said IME this is
one of the most effective optimizations we have, funcall is a non
trivial and relatively slow machine when executed at each function
activation.

  Andrea

bug#58318: 28.2; Emacs installed from package won't work with MinGW

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: larsi@gnus.org, corwin@bru.st, bartosz.bubak@gmail.com,
>         58318@debbugs.gnu.org
> Date: Fri, 07 Oct 2022 13:54:05 +0000
> 
> > How much will performance suffer if we use funcall?
> 
> This is the usual 1 milion dollar question, we can run benchmarks but we
> are never sure of how much realistic they are.  That said IME this is
> one of the most effective optimizations we have, funcall is a non
> trivial and relatively slow machine when executed at each function
> activation.

OK, then let's go for precompiling all the trampolines AOT.

bug#58318: 28.2; Emacs installed from package won't work with MinGW

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <akrl@sdf.org>
>> Cc: larsi@gnus.org, corwin@bru.st, bartosz.bubak@gmail.com,
>>         58318@debbugs.gnu.org
>> Date: Fri, 07 Oct 2022 13:54:05 +0000
>> 
>> > How much will performance suffer if we use funcall?
>> 
>> This is the usual 1 milion dollar question, we can run benchmarks but we
>> are never sure of how much realistic they are.  That said IME this is
>> one of the most effective optimizations we have, funcall is a non
>> trivial and relatively slow machine when executed at each function
>> activation.
>
> OK, then let's go for precompiling all the trampolines AOT.

The only downside might be build time, compiling a trampoline is quick,
1000+ maybe not so much.

  Andrea

bug#58318: 28.2; Emacs installed from package won't work with MinGW

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: larsi@gnus.org, corwin@bru.st, bartosz.bubak@gmail.com,
>         58318@debbugs.gnu.org
> Date: Fri, 07 Oct 2022 14:35:15 +0000
> 
> > OK, then let's go for precompiling all the trampolines AOT.
> 
> The only downside might be build time, compiling a trampoline is quick,
> 1000+ maybe not so much.

Can you time this and see how long it takes for, say, 50 trampolines?

bug#58318: 28.2; Emacs installed from package won't work with MinGW

[Corwin Brust]

On Fri, Oct 7, 2022 at 10:27 AM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > The only downside might be build time, compiling a trampoline is quick,
> > 1000+ maybe not so much.
>
> Can you time this and see how long it takes for, say, 50 trampolines?

FWIW, It appeared to take around 12 minutes to build all trampolines
starting from the "installed" Emacs that was the source for the 28.2
binary zips.

I can't confirm if this resolves the problem; I haven't yet reproduced it.

bug#58318: 28.2; Emacs installed from package won't work with MinGW

[Eli Zaretskii]

> From: Corwin Brust <corwin@bru.st>
> Date: Fri, 7 Oct 2022 10:34:38 -0500
> Cc: Andrea Corallo <akrl@sdf.org>, larsi@gnus.org, bartosz.bubak@gmail.com, 
> 	58318@debbugs.gnu.org
> 
> FWIW, It appeared to take around 12 minutes to build all trampolines
> starting from the "installed" Emacs that was the source for the 28.2
> binary zips.

Is that serially, i.e. using a single Emacs process at a time?

bug#58318: 28.2; Emacs installed from package won't work with MinGW

[Corwin Brust]

On Fri, Oct 7, 2022 at 10:43 AM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: Corwin Brust <corwin@bru.st>
> > Date: Fri, 7 Oct 2022 10:34:38 -0500
> > Cc: Andrea Corallo <akrl@sdf.org>, larsi@gnus.org, bartosz.bubak@gmail.com,
> >       58318@debbugs.gnu.org
> >
> > FWIW, It appeared to take around 12 minutes to build all trampolines
> > starting from the "installed" Emacs that was the source for the 28.2
> > binary zips.
>
> Is that serially, i.e. using a single Emacs process at a time?

Yes, exactly.  No attempt of any kind to parallelize.

bug#58318: 28.2; Emacs installed from package won't work with MinGW

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <akrl@sdf.org>
>> Cc: larsi@gnus.org, corwin@bru.st, bartosz.bubak@gmail.com,
>>         58318@debbugs.gnu.org
>> Date: Fri, 07 Oct 2022 14:35:15 +0000
>> 
>> > OK, then let's go for precompiling all the trampolines AOT.
>> 
>> The only downside might be build time, compiling a trampoline is quick,
>> 1000+ maybe not so much.
>
> Can you time this and see how long it takes for, say, 50 trampolines?

Yeah, unfortunately is not so fast.  On my machine is about 0.2 seconds
each trampoline.  This would translate in about 5min to compile
sequentially all trampolines AOT if I'm not mistaken.  If someone else
can repeat a similar measure that would be interesting.

  Andrea

bug#58318: 28.2; Emacs installed from package won't work with MinGW

[Corwin Brust]

On Fri, Oct 7, 2022 at 9:35 AM Andrea Corallo <akrl@sdf.org> wrote:
>
> >
> > OK, then let's go for precompiling all the trampolines AOT.
>
> The only downside might be build time, compiling a trampoline is quick,
> 1000+ maybe not so much.
>

If this will mostly affect those building Windows binaries *for
redistribution*, perhaps it's not a big problem?  Based on my own very
limited testing, it's fine for me.

bug#58318: 28.2; Emacs installed from package won't work with MinGW

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Corwin Brust <corwin@bru.st>
>> Date: Fri, 7 Oct 2022 10:34:38 -0500
>> Cc: Andrea Corallo <akrl@sdf.org>, larsi@gnus.org, bartosz.bubak@gmail.com, 
>> 	58318@debbugs.gnu.org
>> 
>> FWIW, It appeared to take around 12 minutes to build all trampolines
>> starting from the "installed" Emacs that was the source for the 28.2
>> binary zips.
>
> Is that serially, i.e. using a single Emacs process at a time?

The other (last?) option is to have one single .eln containing all
trampolines.  It should be very quick to compile, the downside is some
more memory usage (we'd have to load and map this .eln when the first
trampoline is requested).

  Andrea

bug#58318: 28.2; Emacs installed from package won't work with MinGW

[Eli Zaretskii]

> From: Corwin Brust <corwin@bru.st>
> Date: Fri, 7 Oct 2022 10:47:00 -0500
> Cc: akrl@sdf.org, larsi@gnus.org, bartosz.bubak@gmail.com, 
> 	58318@debbugs.gnu.org
> 
> On Fri, Oct 7, 2022 at 10:43 AM Eli Zaretskii <eliz@gnu.org> wrote:
> >
> > > From: Corwin Brust <corwin@bru.st>
> > > Date: Fri, 7 Oct 2022 10:34:38 -0500
> > > Cc: Andrea Corallo <akrl@sdf.org>, larsi@gnus.org, bartosz.bubak@gmail.com,
> > >       58318@debbugs.gnu.org
> > >
> > > FWIW, It appeared to take around 12 minutes to build all trampolines
> > > starting from the "installed" Emacs that was the source for the 28.2
> > > binary zips.
> >
> > Is that serially, i.e. using a single Emacs process at a time?
> 
> Yes, exactly.  No attempt of any kind to parallelize.

So this roughly doubles the build time of a release tarball.  Not a
catastrophe, IMO.

bug#58318: 28.2; Emacs installed from package won't work with MinGW

[Eli Zaretskii]

> From: Corwin Brust <corwin@bru.st>
> Date: Fri, 7 Oct 2022 10:52:04 -0500
> Cc: Eli Zaretskii <eliz@gnu.org>, larsi@gnus.org, bartosz.bubak@gmail.com, 
> 	58318@debbugs.gnu.org
> 
> On Fri, Oct 7, 2022 at 9:35 AM Andrea Corallo <akrl@sdf.org> wrote:
> >
> > >
> > > OK, then let's go for precompiling all the trampolines AOT.
> >
> > The only downside might be build time, compiling a trampoline is quick,
> > 1000+ maybe not so much.
> >
> 
> If this will mostly affect those building Windows binaries *for
> redistribution*, perhaps it's not a big problem?  Based on my own very
> limited testing, it's fine for me.

Right, so I think we need a special Makefile target to produce those
compiled trampolines (something like "make trampolines"), and that
target should be only used manually when building a binary
distribution, not when building the release tarball for use on the
same machine where it is built.

Andrea, could you please come up with a patch for that?  Thanks.

bug#58318: 28.2; Emacs installed from package won't work with MinGW

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: Corwin Brust <corwin@bru.st>, larsi@gnus.org, bartosz.bubak@gmail.com,
>         58318@debbugs.gnu.org
> Date: Fri, 07 Oct 2022 17:15:36 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: Corwin Brust <corwin@bru.st>
> >> Date: Fri, 7 Oct 2022 10:34:38 -0500
> >> Cc: Andrea Corallo <akrl@sdf.org>, larsi@gnus.org, bartosz.bubak@gmail.com, 
> >> 	58318@debbugs.gnu.org
> >> 
> >> FWIW, It appeared to take around 12 minutes to build all trampolines
> >> starting from the "installed" Emacs that was the source for the 28.2
> >> binary zips.
> >
> > Is that serially, i.e. using a single Emacs process at a time?
> 
> The other (last?) option is to have one single .eln containing all
> trampolines.  It should be very quick to compile, the downside is some
> more memory usage (we'd have to load and map this .eln when the first
> trampoline is requested).

I'm not sure this is necessary, since the trampolines only need to be
built as part of preparing a binary distribution.

bug#58318: 28.2; Emacs installed from package won't work with MinGW

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Right, so I think we need a special Makefile target to produce those
> compiled trampolines (something like "make trampolines"), and that
> target should be only used manually when building a binary
> distribution, not when building the release tarball for use on the
> same machine where it is built.

Yup.

But I'm not sure this is something we should do, really -- it's extra
work for something that is only important on non-free systems, and it
will complicate the logic in general (since we'd probably need to add
another directory for pre-built trampolines and manage those, etc).

I think we should leave this up to people who do packaging.  That is, if
Cygwin (etc) distributes an Emacs with nativecomp, they will also
distribute libgccjit etc (i.e., all the prerequisites).

That leaves the question of what we should do with the Windows zip file
we (that is, Corwin) distributes, and I think we should avoid enabling
nativecomp in that build, so that it works on the widest range of
Windows machines.

bug#58318: 28.2; Emacs installed from package won't work with MinGW

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Corwin Brust <corwin@bru.st>,  akrl@sdf.org,  bartosz.bubak@gmail.com,
>   58318@debbugs.gnu.org
> Date: Sat, 08 Oct 2022 14:56:46 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Right, so I think we need a special Makefile target to produce those
> > compiled trampolines (something like "make trampolines"), and that
> > target should be only used manually when building a binary
> > distribution, not when building the release tarball for use on the
> > same machine where it is built.
> 
> Yup.
> 
> But I'm not sure this is something we should do, really -- it's extra
> work for something that is only important on non-free systems, and it
> will complicate the logic in general (since we'd probably need to add
> another directory for pre-built trampolines and manage those, etc).
> 
> I think we should leave this up to people who do packaging.  That is, if
> Cygwin (etc) distributes an Emacs with nativecomp, they will also
> distribute libgccjit etc (i.e., all the prerequisites).

It could be part of the scripts in admin/nt/dist-build instead, yes.

> That leaves the question of what we should do with the Windows zip file
> we (that is, Corwin) distributes, and I think we should avoid enabling
> nativecomp in that build, so that it works on the widest range of
> Windows machines.

The Windows build with nativecomp is supposed to be fully workable on
systems that don't have libgccjit, even if the libgccjit bundled with
the zip file is not installed or deleted.  If there are issues with
that, they should be fixed, because we want to allow users to move
Emacs from system top system without the optional libraries, and have
a functional Emacs, like is already the case with image libraries.

bug#58318: 28.2; Emacs installed from package won't work with MinGW

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> That leaves the question of what we should do with the Windows zip file
>> we (that is, Corwin) distributes, and I think we should avoid enabling
>> nativecomp in that build, so that it works on the widest range of
>> Windows machines.
>
> The Windows build with nativecomp is supposed to be fully workable on
> systems that don't have libgccjit, even if the libgccjit bundled with
> the zip file is not installed or deleted.  If there are issues with
> that, they should be fixed, because we want to allow users to move
> Emacs from system top system without the optional libraries, and have
> a functional Emacs, like is already the case with image libraries.

And my suggestion for achieving that is to not enable nativecomp in
this build.

Adding extra these extra mechanisms for Windows builds only seems to be
against the general GNU guidelines for non-free systems (as well as
adding an extra maintenance burden to an already complicated area,
because the code that finds and uses the extra pre-built trampolines
will have to be in the general comp.el code).

bug#58318: 28.2; Emacs installed from package won't work with MinGW

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Corwin Brust <corwin@bru.st>
>> Date: Fri, 7 Oct 2022 10:52:04 -0500
>> Cc: Eli Zaretskii <eliz@gnu.org>, larsi@gnus.org, bartosz.bubak@gmail.com, 
>> 	58318@debbugs.gnu.org
>> 
>> On Fri, Oct 7, 2022 at 9:35 AM Andrea Corallo <akrl@sdf.org> wrote:
>> >
>> > >
>> > > OK, then let's go for precompiling all the trampolines AOT.
>> >
>> > The only downside might be build time, compiling a trampoline is quick,
>> > 1000+ maybe not so much.
>> >
>> 
>> If this will mostly affect those building Windows binaries *for
>> redistribution*, perhaps it's not a big problem?  Based on my own very
>> limited testing, it's fine for me.
>
> Right, so I think we need a special Makefile target to produce those
> compiled trampolines (something like "make trampolines"), and that
> target should be only used manually when building a binary
> distribution, not when building the release tarball for use on the
> same machine where it is built.
>
> Andrea, could you please come up with a patch for that?  Thanks.

Right, will do.

  Andrea

bug#58318: 28.2; Emacs installed from package won't work with MinGW

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: corwin@bru.st,  akrl@sdf.org,  bartosz.bubak@gmail.com,
>   58318@debbugs.gnu.org
> Date: Sat, 08 Oct 2022 15:10:05 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> That leaves the question of what we should do with the Windows zip file
> >> we (that is, Corwin) distributes, and I think we should avoid enabling
> >> nativecomp in that build, so that it works on the widest range of
> >> Windows machines.
> >
> > The Windows build with nativecomp is supposed to be fully workable on
> > systems that don't have libgccjit, even if the libgccjit bundled with
> > the zip file is not installed or deleted.  If there are issues with
> > that, they should be fixed, because we want to allow users to move
> > Emacs from system top system without the optional libraries, and have
> > a functional Emacs, like is already the case with image libraries.
> 
> And my suggestion for achieving that is to not enable nativecomp in
> this build.

I don't agree.

> Adding extra these extra mechanisms for Windows builds only seems to be
> against the general GNU guidelines for non-free systems (as well as
> adding an extra maintenance burden to an already complicated area,
> because the code that finds and uses the extra pre-built trampolines
> will have to be in the general comp.el code).

The general mechanism already exists, and for a long time.  We are
just using it.  Adding an optional library is boilerplate and quite
easy.  It's basically a non-issue.

As for the specific issue of trampolines, I understand that compiling
them is a simple command, and so also a non-issue.

bug#58318: 28.2; Emacs installed from package won't work with MinGW

[Andrea Corallo]

Andrea Corallo <akrl@sdf.org> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> From: Corwin Brust <corwin@bru.st>
>>> Date: Fri, 7 Oct 2022 10:52:04 -0500
>>> Cc: Eli Zaretskii <eliz@gnu.org>, larsi@gnus.org, bartosz.bubak@gmail.com, 
>>> 	58318@debbugs.gnu.org
>>> 
>>> On Fri, Oct 7, 2022 at 9:35 AM Andrea Corallo <akrl@sdf.org> wrote:
>>> >
>>> > >
>>> > > OK, then let's go for precompiling all the trampolines AOT.
>>> >
>>> > The only downside might be build time, compiling a trampoline is quick,
>>> > 1000+ maybe not so much.
>>> >
>>> 
>>> If this will mostly affect those building Windows binaries *for
>>> redistribution*, perhaps it's not a big problem?  Based on my own very
>>> limited testing, it's fine for me.
>>
>> Right, so I think we need a special Makefile target to produce those
>> compiled trampolines (something like "make trampolines"), and that
>> target should be only used manually when building a binary
>> distribution, not when building the release tarball for use on the
>> same machine where it is built.
>>
>> Andrea, could you please come up with a patch for that?  Thanks.
>
> Right, will do.
>
>   Andrea

Okay 3744720904 adds 'trampolines' as target.  The trampolines are
deposed in the 'native-lisp' directory so it will do the job for
packaging the release.  Build time for this target is on my machine ~
1min 30sec.

  Andrea

bug#58318: 28.2; Emacs installed from package won't work with MinGW

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: larsi@gnus.org, Corwin Brust <corwin@bru.st>, 58318@debbugs.gnu.org,
>         bartosz.bubak@gmail.com
> Date: Tue, 11 Oct 2022 19:23:54 +0000
> 
> Okay 3744720904 adds 'trampolines' as target.  The trampolines are
> deposed in the 'native-lisp' directory so it will do the job for
> packaging the release.

Thanks.

> Build time for this target is on my machine ~ 1min 30sec.

How does this compare to the rest of the build?

bug#58318: 28.2; Emacs installed from package won't work with MinGW

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <akrl@sdf.org>
>> Cc: larsi@gnus.org, Corwin Brust <corwin@bru.st>, 58318@debbugs.gnu.org,
>>         bartosz.bubak@gmail.com
>> Date: Tue, 11 Oct 2022 19:23:54 +0000
>> 
>> Okay 3744720904 adds 'trampolines' as target.  The trampolines are
>> deposed in the 'native-lisp' directory so it will do the job for
>> packaging the release.
>
> Thanks.
>
>> Build time for this target is on my machine ~ 1min 30sec.
>
> How does this compare to the rest of the build?

make bootstrap on my dev machine (-j16) is ~ 3min 10 secs.

  Andrea

bug#58318: 28.2; Emacs installed from package won't work with MinGW

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: larsi@gnus.org, corwin@bru.st, 58318@debbugs.gnu.org,
>         bartosz.bubak@gmail.com
> Date: Tue, 11 Oct 2022 20:45:19 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: Andrea Corallo <akrl@sdf.org>
> >> Cc: larsi@gnus.org, Corwin Brust <corwin@bru.st>, 58318@debbugs.gnu.org,
> >>         bartosz.bubak@gmail.com
> >> Date: Tue, 11 Oct 2022 19:23:54 +0000
> >> 
> >> Okay 3744720904 adds 'trampolines' as target.  The trampolines are
> >> deposed in the 'native-lisp' directory so it will do the job for
> >> packaging the release.
> >
> > Thanks.
> >
> >> Build time for this target is on my machine ~ 1min 30sec.
> >
> > How does this compare to the rest of the build?
> 
> make bootstrap on my dev machine (-j16) is ~ 3min 10 secs.

I meant the time to build a release tarball, which should be much
shorter.  Sorry for being unclear.

bug#58318: 28.2; Emacs installed from package won't work with MinGW

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <akrl@sdf.org>
>> Cc: larsi@gnus.org, corwin@bru.st, 58318@debbugs.gnu.org,
>>         bartosz.bubak@gmail.com
>> Date: Tue, 11 Oct 2022 20:45:19 +0000
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> >> From: Andrea Corallo <akrl@sdf.org>
>> >> Cc: larsi@gnus.org, Corwin Brust <corwin@bru.st>, 58318@debbugs.gnu.org,
>> >>         bartosz.bubak@gmail.com
>> >> Date: Tue, 11 Oct 2022 19:23:54 +0000
>> >> 
>> >> Okay 3744720904 adds 'trampolines' as target.  The trampolines are
>> >> deposed in the 'native-lisp' directory so it will do the job for
>> >> packaging the release.
>> >
>> > Thanks.
>> >
>> >> Build time for this target is on my machine ~ 1min 30sec.
>> >
>> > How does this compare to the rest of the build?
>> 
>> make bootstrap on my dev machine (-j16) is ~ 3min 10 secs.
>
> I meant the time to build a release tarball, which should be much
> shorter.  Sorry for being unclear.

Apologies on my side, I'm not very much into release tarball generation.
If you specify what's the command you are referring to I can time it and
report.

BR

  Andrea

bug#58318: 28.2; Emacs installed from package won't work with MinGW

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: larsi@gnus.org, corwin@bru.st, 58318@debbugs.gnu.org,
>         bartosz.bubak@gmail.com
> Date: Wed, 12 Oct 2022 08:14:43 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> make bootstrap on my dev machine (-j16) is ~ 3min 10 secs.
> >
> > I meant the time to build a release tarball, which should be much
> > shorter.  Sorry for being unclear.
> 
> Apologies on my side, I'm not very much into release tarball generation.
> If you specify what's the command you are referring to I can time it and
> report.

It isn't too important, so feel free to disregard.  But if you have a
few minutes, then unpack the Emacs 28.2 release tarball and build it
with -j16.

Thanks.

bug#58318: 28.2; Emacs installed from package won't work with MinGW

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <akrl@sdf.org>
>> Cc: larsi@gnus.org, corwin@bru.st, 58318@debbugs.gnu.org,
>>         bartosz.bubak@gmail.com
>> Date: Wed, 12 Oct 2022 08:14:43 +0000
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> >> make bootstrap on my dev machine (-j16) is ~ 3min 10 secs.
>> >
>> > I meant the time to build a release tarball, which should be much
>> > shorter.  Sorry for being unclear.
>> 
>> Apologies on my side, I'm not very much into release tarball generation.
>> If you specify what's the command you are referring to I can time it and
>> report.
>
> It isn't too important, so feel free to disregard.  But if you have a
> few minutes, then unpack the Emacs 28.2 release tarball and build it
> with -j16.

Ok,

on my machine is  about 40secs the configure phase and ~1min make -j16.

BR

  Andrea

bug#58318: 28.2; Emacs installed from package won't work with MinGW

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: larsi@gnus.org, corwin@bru.st, 58318@debbugs.gnu.org,
>         bartosz.bubak@gmail.com
> Date: Wed, 12 Oct 2022 14:55:32 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: Andrea Corallo <akrl@sdf.org>
> >> Cc: larsi@gnus.org, corwin@bru.st, 58318@debbugs.gnu.org,
> >>         bartosz.bubak@gmail.com
> >> Date: Wed, 12 Oct 2022 08:14:43 +0000
> >> 
> >> Eli Zaretskii <eliz@gnu.org> writes:
> >> 
> >> >> make bootstrap on my dev machine (-j16) is ~ 3min 10 secs.
> >> >
> >> > I meant the time to build a release tarball, which should be much
> >> > shorter.  Sorry for being unclear.
> >> 
> >> Apologies on my side, I'm not very much into release tarball generation.
> >> If you specify what's the command you are referring to I can time it and
> >> report.
> >
> > It isn't too important, so feel free to disregard.  But if you have a
> > few minutes, then unpack the Emacs 28.2 release tarball and build it
> > with -j16.
> 
> Ok,
> 
> on my machine is  about 40secs the configure phase and ~1min make -j16.

Thanks, that confirms previous findings: compiling all the trampolines
takes roughly the same time as building the tarball.

bug#58318: 28.2; Emacs installed from package won't work with MinGW

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <akrl@sdf.org>
>> Cc: larsi@gnus.org, corwin@bru.st, 58318@debbugs.gnu.org,
>>         bartosz.bubak@gmail.com
>> Date: Wed, 12 Oct 2022 14:55:32 +0000
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> >> From: Andrea Corallo <akrl@sdf.org>
>> >> Cc: larsi@gnus.org, corwin@bru.st, 58318@debbugs.gnu.org,
>> >>         bartosz.bubak@gmail.com
>> >> Date: Wed, 12 Oct 2022 08:14:43 +0000
>> >> 
>> >> Eli Zaretskii <eliz@gnu.org> writes:
>> >> 
>> >> >> make bootstrap on my dev machine (-j16) is ~ 3min 10 secs.
>> >> >
>> >> > I meant the time to build a release tarball, which should be much
>> >> > shorter.  Sorry for being unclear.
>> >> 
>> >> Apologies on my side, I'm not very much into release tarball generation.
>> >> If you specify what's the command you are referring to I can time it and
>> >> report.
>> >
>> > It isn't too important, so feel free to disregard.  But if you have a
>> > few minutes, then unpack the Emacs 28.2 release tarball and build it
>> > with -j16.
>> 
>> Ok,
>> 
>> on my machine is  about 40secs the configure phase and ~1min make -j16.
>
> Thanks, that confirms previous findings: compiling all the trampolines
> takes roughly the same time as building the tarball.

Yes on a sixteen thread machine, with less parallelism trampolines
should be considerably faster then building the tarball.

  Andrea