[feature/native-comp] breakage on build

[feature/native-comp] breakage on build

[Phillip Lord]

I am getting a build breakage on native-comp. While I can build
native-comp on windows in source, it seems to fail while using my
packaging scripts. These use out-of-source builds which might be the
problem.

Backtrace is here. I've added a few "message"'s to
`comp-compile-ctxt-to-file' so I could see the full paths. The directory
mentioned does exist but is empty.

I am getting that there is something that isn't happy with windows
paths, but that's a total guess.

Phil



make -C ../lisp autoloads EMACS="../src/bootstrap-emacs.exe"
make[2]: Entering directory '/home/Administrator/emacs-build/build/emacs-28.0.50-snapshot-feature_native-comp-windows/lisp'
 ELC+ELN   /c/msys64/home/Administrator/emacs-build/git/feature/native-comp-windows/lisp/international/titdic-cnv.elc
compiling to c:/msys64/home/Administrator/emacs-build/build/emacs-28.0.50-snapshot-feature_native-comp-windows/native-lisp/28.0.50-x86_64-w64-mingw32-3459bc949453c7833f94b407f8ee7c17/titdic-cnv-765ac3beca7090ba17f799d9600c18e6-a5dbcc84a969a94717512a079e1cc96a.eln
dir: c:/msys64/home/Administrator/emacs-build/build/emacs-28.0.50-snapshot-feature_native-comp-windows/native-lisp/28.0.50-x86_64-w64-mingw32-3459bc949453c7833f94b407f8ee7c17/
Debugger entered--Lisp error: (file-missing "Creating file with prefix" "No such file or directory" "c:/msys64/home/Administrator/emacs-build/build/ema...")
  comp--compile-ctxt-to-file("c:/msys64/home/Administrator/emacs-build/build/ema...")
  (let ((dir (file-name-directory name))) (message "dir: %s" dir) (comp-finalize-relocs) (maphash #'(lambda (_ f) (comp-log-func f 1)) (progn (or (progn (and (memq (type-of comp-ctxt) cl-struct-comp-ctxt-tags) t)) (signal 'wrong-type-argument (list 'comp-ctxt comp-ctxt))) (aref comp-ctxt 11))) (if (file-exists-p dir) nil (condition-case nil (progn (make-directory dir t)) ('file-already-exists nil))) (comp--compile-ctxt-to-file name))
  comp-compile-ctxt-to-file("c:/msys64/home/Administrator/emacs-build/build/ema...")
  (setq compile-result (comp-compile-ctxt-to-file (progn (or (progn (and (memq (type-of comp-ctxt) cl-struct-comp-ctxt-tags) t)) (signal 'wrong-type-argument (list 'comp-ctxt comp-ctxt))) (aref comp-ctxt 7))))
  (unwind-protect (setq compile-result (comp-compile-ctxt-to-file (progn (or (progn (and (memq ... cl-struct-comp-ctxt-tags) t)) (signal 'wrong-type-argument (list 'comp-ctxt comp-ctxt))) (aref comp-ctxt 7)))) (and (comp--release-ctxt) compile-result))
  (let (compile-result) (comp--init-ctxt) (unwind-protect (setq compile-result (comp-compile-ctxt-to-file (progn (or (progn (and ... t)) (signal 'wrong-type-argument (list ... comp-ctxt))) (aref comp-ctxt 7)))) (and (comp--release-ctxt) compile-result)))
  comp-final1()
  (if (or byte-native-for-bootstrap comp-async-compilation) (comp-final1) (let* ((output (progn (or (progn (and ... t)) (signal 'wrong-type-argument (list ... comp-ctxt))) (aref comp-ctxt 7))) (print-escape-newlines t) (print-length nil) (print-level nil) (print-quoted t) (print-gensym t) (print-circle t) (print-escape-multibyte t) (expr (cons 'progn (cons '(require ...) (cons (list ... ... comp-verbose ... comp-libgccjit-reproducer ... comp-ctxt ... ... ... ... ... ...) (cons comp-async-env-modifier-form ...))))) (temp-file (make-temp-file (concat "emacs-int-comp-" (file-name-base output) "-") nil ".el"))) (let ((temp-file temp-file) (temp-buffer (generate-new-buffer " *temp file*" t))) (unwind-protect (prog1 (save-current-buffer (set-buffer temp-buffer) (insert ";; -*-coding: nil; -*-\n") (insert (prin1-to-string expr))) (save-current-buffer (set-buffer temp-buffer) (write-region nil nil temp-file nil 0))) (and (buffer-name temp-buffer) (kill-buffer temp-buffer)))) (let ((temp-buffer (generate-new-buffer " *temp*" t))) (save-current-buffer (set-buffer temp-buffer) (unwind-protect (progn (unwind-protect (if ... output ...) (comp-log-to-buffer ...))) (and (buffer-name temp-buffer) (kill-buffer temp-buffer)))))))
  (if comp-dry-run nil (if (or byte-native-for-bootstrap comp-async-compilation) (comp-final1) (let* ((output (progn (or (progn ...) (signal ... ...)) (aref comp-ctxt 7))) (print-escape-newlines t) (print-length nil) (print-level nil) (print-quoted t) (print-gensym t) (print-circle t) (print-escape-multibyte t) (expr (cons 'progn (cons '... (cons ... ...)))) (temp-file (make-temp-file (concat "emacs-int-comp-" (file-name-base output) "-") nil ".el"))) (let ((temp-file temp-file) (temp-buffer (generate-new-buffer " *temp file*" t))) (unwind-protect (prog1 (save-current-buffer (set-buffer temp-buffer) (insert ";; -*-coding: nil; -*-\n") (insert ...)) (save-current-buffer (set-buffer temp-buffer) (write-region nil nil temp-file nil 0))) (and (buffer-name temp-buffer) (kill-buffer temp-buffer)))) (let ((temp-buffer (generate-new-buffer " *temp*" t))) (save-current-buffer (set-buffer temp-buffer) (unwind-protect (progn (unwind-protect ... ...)) (and (buffer-name temp-buffer) (kill-buffer temp-buffer))))))))
  comp-final(nil)
  funcall(comp-final nil)
  (setq data (funcall pass data))
  (if (memq pass comp-disabled-passes) (progn) (comp-log (format "(%s) Running pass %s:\n" function-or-file pass) 2) (setq data (funcall pass data)) (setq report (cons (cons pass (float-time (time-since t0))) report)) (let* ((--cl-var-- (alist-get pass comp-post-pass-hooks)) (f nil)) (while (consp --cl-var--) (setq f (car --cl-var--)) (funcall f data) (setq --cl-var-- (cdr --cl-var--))) nil))
  (while (progn (setq t0 (current-time)) (consp --cl-var--)) (setq pass (car --cl-var--)) (if (memq pass comp-disabled-passes) (progn) (comp-log (format "(%s) Running pass %s:\n" function-or-file pass) 2) (setq data (funcall pass data)) (setq report (cons (cons pass (float-time (time-since t0))) report)) (let* ((--cl-var-- (alist-get pass comp-post-pass-hooks)) (f nil)) (while (consp --cl-var--) (setq f (car --cl-var--)) (funcall f data) (setq --cl-var-- (cdr --cl-var--))) nil)) (setq --cl-var-- (cdr --cl-var--)) (setq --cl-var-- nil))
  (let* ((report nil) (t0 nil) (--cl-var-- comp-passes) (pass nil) (--cl-var-- t)) (while (progn (setq t0 (current-time)) (consp --cl-var--)) (setq pass (car --cl-var--)) (if (memq pass comp-disabled-passes) (progn) (comp-log (format "(%s) Running pass %s:\n" function-or-file pass) 2) (setq data (funcall pass data)) (setq report (cons (cons pass (float-time (time-since t0))) report)) (let* ((--cl-var-- (alist-get pass comp-post-pass-hooks)) (f nil)) (while (consp --cl-var--) (setq f (car --cl-var--)) (funcall f data) (setq --cl-var-- (cdr --cl-var--))) nil)) (setq --cl-var-- (cdr --cl-var--)) (setq --cl-var-- nil)) (if comp-log-time-report (progn (comp-log (format "Done compiling %s" data) 0) (let* ((--cl-var-- (reverse report)) (time nil) (pass nil)) (while (consp --cl-var--) (setq time (car --cl-var--) pass (car-safe (prog1 time ...))) (comp-log (format "Pass %s took: %fs." pass time) 0) (setq --cl-var-- (cdr --cl-var--))) nil))) nil)
  (condition-case err (let* ((report nil) (t0 nil) (--cl-var-- comp-passes) (pass nil) (--cl-var-- t)) (while (progn (setq t0 (current-time)) (consp --cl-var--)) (setq pass (car --cl-var--)) (if (memq pass comp-disabled-passes) (progn) (comp-log (format "(%s) Running pass %s:\n" function-or-file pass) 2) (setq data (funcall pass data)) (setq report (cons (cons pass (float-time ...)) report)) (let* ((--cl-var-- (alist-get pass comp-post-pass-hooks)) (f nil)) (while (consp --cl-var--) (setq f (car --cl-var--)) (funcall f data) (setq --cl-var-- (cdr --cl-var--))) nil)) (setq --cl-var-- (cdr --cl-var--)) (setq --cl-var-- nil)) (if comp-log-time-report (progn (comp-log (format "Done compiling %s" data) 0) (let* ((--cl-var-- (reverse report)) (time nil) (pass nil)) (while (consp --cl-var--) (setq time (car --cl-var--) pass (car-safe ...)) (comp-log (format "Pass %s took: %fs." pass time) 0) (setq --cl-var-- (cdr --cl-var--))) nil))) nil) (native-compiler-error (let ((err-val (cdr err))) (signal (car err) (if (consp err-val) (cons function-or-file err-val) (list function-or-file err-val))))))
  (let* ((data function-or-file) (comp-native-compiling t) (byte-native-qualities nil) (byte-compile-debug t) (comp-ctxt (make-comp-ctxt :output output :with-late-load with-late-load))) (comp-log "\n\f\n" 1) (condition-case err (let* ((report nil) (t0 nil) (--cl-var-- comp-passes) (pass nil) (--cl-var-- t)) (while (progn (setq t0 (current-time)) (consp --cl-var--)) (setq pass (car --cl-var--)) (if (memq pass comp-disabled-passes) (progn) (comp-log (format "(%s) Running pass %s:\n" function-or-file pass) 2) (setq data (funcall pass data)) (setq report (cons (cons pass ...) report)) (let* ((--cl-var-- ...) (f nil)) (while (consp --cl-var--) (setq f ...) (funcall f data) (setq --cl-var-- ...)) nil)) (setq --cl-var-- (cdr --cl-var--)) (setq --cl-var-- nil)) (if comp-log-time-report (progn (comp-log (format "Done compiling %s" data) 0) (let* ((--cl-var-- ...) (time nil) (pass nil)) (while (consp --cl-var--) (setq time ... pass ...) (comp-log ... 0) (setq --cl-var-- ...)) nil))) nil) (native-compiler-error (let ((err-val (cdr err))) (signal (car err) (if (consp err-val) (cons function-or-file err-val) (list function-or-file err-val)))))) (if (stringp function-or-file) data (native-elisp-load data)))
  comp--native-compile("C:/msys64/home/Administrator/emacs-build/git/featu...")

Re: [feature/native-comp] breakage on build

[akrl--- via Emacs development discussions.]

Phillip Lord <phillip.lord@russet.org.uk> writes:

> I am getting a build breakage on native-comp. While I can build
> native-comp on windows in source, it seems to fail while using my
> packaging scripts. These use out-of-source builds which might be the
> problem.
>
> Backtrace is here. I've added a few "message"'s to
> `comp-compile-ctxt-to-file' so I could see the full paths. The directory
> mentioned does exist but is empty.
>
> I am getting that there is something that isn't happy with windows
> paths, but that's a total guess.
>
> Phil

Hi Phil,

I tried just now an out of tree build on x86_64-pc-linux-gnu and it
worked.  This might confirm your idea of a Windows specific issue?

Would be interesting to debug and see exactly what's the file that we
can't create and why.

  Andrea

Re: [feature/native-comp] breakage on build

[phillip.lord]

On 2021-01-30 13:26, akrl--- via "Emacs development discussions." wrote:
> Phillip Lord <phillip.lord@russet.org.uk> writes:
> 
>> I am getting a build breakage on native-comp. While I can build
>> native-comp on windows in source, it seems to fail while using my
>> packaging scripts. These use out-of-source builds which might be the
>> problem.
>> 
>> Backtrace is here. I've added a few "message"'s to
>> `comp-compile-ctxt-to-file' so I could see the full paths. The 
>> directory
>> mentioned does exist but is empty.
>> 
>> I am getting that there is something that isn't happy with windows
>> paths, but that's a total guess.
>> 
>> Phil
> 
> Hi Phil,
> 
> I tried just now an out of tree build on x86_64-pc-linux-gnu and it
> worked.  This might confirm your idea of a Windows specific issue?
> 
> Would be interesting to debug and see exactly what's the file that we
> can't create and why.


I will give it a go and see what I can find. I'd checked an OOS build on 
linux already, but I don't think I have done a "normal" one, that is not 
using my packaging scripts, on windows. Will try.

Re: [feature/native-comp] breakage on build

[phillip.lord]

On 2021-01-30 14:40, phillip.lord@russet.org.uk wrote:
> On 2021-01-30 13:26, akrl--- via "Emacs development discussions." 
> wrote:
>> Phillip Lord <phillip.lord@russet.org.uk> writes:
>> 
>>> I am getting a build breakage on native-comp. While I can build
>>> native-comp on windows in source, it seems to fail while using my
>>> packaging scripts. These use out-of-source builds which might be the
>>> problem.
>>> 
>>> Backtrace is here. I've added a few "message"'s to
>>> `comp-compile-ctxt-to-file' so I could see the full paths. The 
>>> directory
>>> mentioned does exist but is empty.
>>> 
>>> I am getting that there is something that isn't happy with windows
>>> paths, but that's a total guess.
>>> 
>>> Phil
>> 
>> Hi Phil,
>> 
>> I tried just now an out of tree build on x86_64-pc-linux-gnu and it
>> worked.  This might confirm your idea of a Windows specific issue?
>> 
>> Would be interesting to debug and see exactly what's the file that we
>> can't create and why.
> 
> 
> I will give it a go and see what I can find. I'd checked an OOS build
> on linux already, but I don't think I have done a "normal" one, that
> is not using my packaging scripts, on windows. Will try.


Actually, the error message appears to make from (and only from) 
make-temp-file-internal.

comp.c is nearly the only think that calls that function from c, so I'd 
guess that this is the culprit. I wonder if the windows drive letters 
("c:/msys64") use magic file handling.

Certainly

(make-temp-file-internal "c:/fred" nil "suf" nil)

raises the same error on linux (am afraid all my windows machines are 
inaccessible at the moment).

Phil

Re: [feature/native-comp] breakage on build

[Eli Zaretskii]

> Date: Sat, 30 Jan 2021 15:01:53 +0000
> From: phillip.lord@russet.org.uk
> Cc: emacs-devel@gnu.org
> 
> Actually, the error message appears to make from (and only from) 
> make-temp-file-internal.
> 
> comp.c is nearly the only think that calls that function from c, so I'd 
> guess that this is the culprit. I wonder if the windows drive letters 
> ("c:/msys64") use magic file handling.
> 
> Certainly
> 
> (make-temp-file-internal "c:/fred" nil "suf" nil)
> 
> raises the same error on linux (am afraid all my windows machines are 
> inaccessible at the moment).

The above works on my Windows machine.

On GNU/Linux the name "c:/fred" is relative, so the file will be
created under the Emacs working directory.

What kind of failure do you see when you invoke the above?

Re: [feature/native-comp] breakage on build

[Eli Zaretskii]

I have a guess why it fails, based on this part of the trace:

  compiling to c:/msys64/home/Administrator/emacs-build/build/emacs-28.0.50-snapshot-feature_native-comp-windows/native-lisp/28.0.50-x86_64-w64-mingw32-3459bc949453c7833f94b407f8ee7c17/titdic-cnv-765ac3beca7090ba17f799d9600c18e6-a5dbcc84a969a94717512a079e1cc96a.eln

The name of that file is suspiciously close to the 260-byte limit of
file-name length that Win32 APIs support.  So my guess is that the
temporary file had a few more characters (like the 6 characters which
replace the XXXXXX, perhaps?), and that caused the failure.

Solution: use a directory with a shorter name (the above is really
excessive in its leading part, IMO).

Re: [feature/native-comp] breakage on build

[Eli Zaretskii]

> Date: Sat, 30 Jan 2021 18:38:10 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: akrl@sdf.org, emacs-devel@gnu.org
> 
>   compiling to c:/msys64/home/Administrator/emacs-build/build/emacs-28.0.50-snapshot-feature_native-comp-windows/native-lisp/28.0.50-x86_64-w64-mingw32-3459bc949453c7833f94b407f8ee7c17/titdic-cnv-765ac3beca7090ba17f799d9600c18e6-a5dbcc84a969a94717512a079e1cc96a.eln
> 
> The name of that file is suspiciously close to the 260-byte limit of
> file-name length that Win32 APIs support.  So my guess is that the
> temporary file had a few more characters (like the 6 characters which
> replace the XXXXXX, perhaps?), and that caused the failure.

Btw, Andrea: why do we need no less than 3 32-character hashes in that
file name?  With 26-byte limit on file names that libgccjit can use,
these 96 characters use up almost half of that space, and that will
bite us time and again down the road on MS-Windows.  Can we use just
one hash?

Re: [feature/native-comp] breakage on build

[phillip.lord]

On 2021-01-30 17:06, Eli Zaretskii wrote:
>> Date: Sat, 30 Jan 2021 18:38:10 +0200
>> From: Eli Zaretskii <eliz@gnu.org>
>> Cc: akrl@sdf.org, emacs-devel@gnu.org
>> 
>>   compiling to 
>> c:/msys64/home/Administrator/emacs-build/build/emacs-28.0.50-snapshot-feature_native-comp-windows/native-lisp/28.0.50-x86_64-w64-mingw32-3459bc949453c7833f94b407f8ee7c17/titdic-cnv-765ac3beca7090ba17f799d9600c18e6-a5dbcc84a969a94717512a079e1cc96a.eln
>> 
>> The name of that file is suspiciously close to the 260-byte limit of
>> file-name length that Win32 APIs support.  So my guess is that the
>> temporary file had a few more characters (like the 6 characters which
>> replace the XXXXXX, perhaps?), and that caused the failure.
> 
> Btw, Andrea: why do we need no less than 3 32-character hashes in that
> file name?  With 26-byte limit on file names that libgccjit can use,
> these 96 characters use up almost half of that space, and that will
> bite us time and again down the road on MS-Windows.  Can we use just
> one hash?

I've just been checking the .eln-cache on my linux machine. The file 
names are distributed between 153 and 192 characters. That already seems 
uncomfortable close. The standard location for will be deeper on windows 
for a given user name, I think. So this looks like a potentially 
significant problem.

Phil

Re: [feature/native-comp] breakage on build

[akrl--- via Emacs development discussions.]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Sat, 30 Jan 2021 18:38:10 +0200
>> From: Eli Zaretskii <eliz@gnu.org>
>> Cc: akrl@sdf.org, emacs-devel@gnu.org
>> 
>>   compiling to
>> c:/msys64/home/Administrator/emacs-build/build/emacs-28.0.50-snapshot-feature_native-comp-windows/native-lisp/28.0.50-x86_64-w64-mingw32-3459bc949453c7833f94b407f8ee7c17/titdic-cnv-765ac3beca7090ba17f799d9600c18e6-a5dbcc84a969a94717512a079e1cc96a.eln
>> 
>> The name of that file is suspiciously close to the 260-byte limit of
>> file-name length that Win32 APIs support.  So my guess is that the
>> temporary file had a few more characters (like the 6 characters which
>> replace the XXXXXX, perhaps?), and that caused the failure.
>
> Btw, Andrea: why do we need no less than 3 32-character hashes in that
> file name?  With 26-byte limit on file names that libgccjit can use,
> these 96 characters use up almost half of that space, and that will
> bite us time and again down the road on MS-Windows.  Can we use just
> one hash?
>

Say ATM we have:

native-lisp/28.0.50-x86_64-w64-mingw32-HASH1/titdic-cnv-HASH2-HASH3.eln

- HASH1 disambiguate triplet, Emacs configuration, version etc.  We can
  say this is disambiguating everything that influence the ABI the eln
  expects and exposes.

- HASH2 is just hashing the full patch of the original source file.

- HASH3 is hashing the content of the file.

HASH1 is handy because one can remove selectively the eln files of say a
previous Emacs version or an old configuration.  I often remove all of
these folders but the most recent one.  That said technically HASH1 and
HASH3 could be merged.

HASH2 is handy because there must be within this folder at most one file
with each HASH2.  IOW recompiling we remove all the other .eln sharing
the same HASH2 if present as indeed these files are obsolete.

The lenght of these hashes is coming directly from the algo we use
(md5), I think collision is really not an issue so we can easlily short
these.

  Andrea

Re: [feature/native-comp] breakage on build

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: phillip.lord@russet.org.uk,  emacs-devel@gnu.org
> Date: Sat, 30 Jan 2021 19:44:56 +0000
> 
> native-lisp/28.0.50-x86_64-w64-mingw32-HASH1/titdic-cnv-HASH2-HASH3.eln
> 
> - HASH1 disambiguate triplet, Emacs configuration, version etc.

But that information is already present in the text before:
28.0.50-x86_64-w64-mingw32.  What else is missing that we need a hash?

> The lenght of these hashes is coming directly from the algo we use
> (md5), I think collision is really not an issue so we can easlily short
> these.

Then I think we should indeed short them, because this issue will bite
us again.

Thanks.

Re: [feature/native-comp] breakage on build

[akrl--- via Emacs development discussions.]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <akrl@sdf.org>
>> Cc: phillip.lord@russet.org.uk,  emacs-devel@gnu.org
>> Date: Sat, 30 Jan 2021 19:44:56 +0000
>> 
>> native-lisp/28.0.50-x86_64-w64-mingw32-HASH1/titdic-cnv-HASH2-HASH3.eln
>> 
>> - HASH1 disambiguate triplet, Emacs configuration, version etc.
>
> But that information is already present in the text before:
> 28.0.50-x86_64-w64-mingw32.  What else is missing that we need a hash?

I left the text before as indicative and human readable but
unfortunatelly that's not complete.

The list of all primitive functions contribute to the eln linking
mechanism we use and is accounted in this hash.  Also we can trigger the
generation of a new hash each time we deploy an eln ABI breaking change.

>> The lenght of these hashes is coming directly from the algo we use
>> (md5), I think collision is really not an issue so we can easlily short
>> these.
>
> Then I think we should indeed short them, because this issue will bite
> us again.

That's easy, we should just pick a lenght we like.

Thanks

  Andrea

Re: [feature/native-comp] breakage on build

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: phillip.lord@russet.org.uk,  emacs-devel@gnu.org
> Date: Sat, 30 Jan 2021 20:17:58 +0000
> 
> >> native-lisp/28.0.50-x86_64-w64-mingw32-HASH1/titdic-cnv-HASH2-HASH3.eln
> >> 
> >> - HASH1 disambiguate triplet, Emacs configuration, version etc.
> >
> > But that information is already present in the text before:
> > 28.0.50-x86_64-w64-mingw32.  What else is missing that we need a hash?
> 
> I left the text before as indicative and human readable but
> unfortunatelly that's not complete.
> 
> The list of all primitive functions contribute to the eln linking
> mechanism we use and is accounted in this hash.  Also we can trigger the
> generation of a new hash each time we deploy an eln ABI breaking change.

Sorry, I don't follow.  You said HASH1 disambiguates the triplet, and
that is already present in the name.  Please elaborate on the other
factors that affect HASH1, as what you said (primitives that
contribute? eln ABI breaking changes?) isn't enough for me to
understand what information they record.  And in any case, if the
triplet is in HASH1, why not remove its human-readable representation
and save a few bytes?

Re: [feature/native-comp] breakage on build

[akrl--- via Emacs development discussions.]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <akrl@sdf.org>
>> Cc: phillip.lord@russet.org.uk,  emacs-devel@gnu.org
>> Date: Sat, 30 Jan 2021 20:17:58 +0000
>> 
>> >> native-lisp/28.0.50-x86_64-w64-mingw32-HASH1/titdic-cnv-HASH2-HASH3.eln
>> >> 
>> >> - HASH1 disambiguate triplet, Emacs configuration, version etc.
>> >
>> > But that information is already present in the text before:
>> > 28.0.50-x86_64-w64-mingw32.  What else is missing that we need a hash?
>> 
>> I left the text before as indicative and human readable but
>> unfortunatelly that's not complete.
>> 
>> The list of all primitive functions contribute to the eln linking
>> mechanism we use and is accounted in this hash.  Also we can trigger the
>> generation of a new hash each time we deploy an eln ABI breaking change.
>
> Sorry, I don't follow.  You said HASH1 disambiguates the triplet, and
> that is already present in the name.  Please elaborate on the other
> factors that affect HASH1, as what you said (primitives that
> contribute? eln ABI breaking changes?) isn't enough for me to
> understand what information they record.

Sure,

each .eln while executing must indeed call into Emacs C code.  Call
happens through function pointers, say we must call Fmessage, something
like this will be rendered into the eln:

freloc_link_table->Fmessage (...)

'freloc_link_table' is a pointer statically allocated within the eln
object, it gets set while loading to point to the right data structure
in memory.  This data structure is a C structure holding all function
pointers to all Emacs primitive functions.

For this reason we must be sure that the definition of
'freloc_link_table' use while compiling the eln matches the one offered
by the Emacs loading the eln file.

This is the main reason eln are not portable between different Emacs
configurations.

HASH1 has the duty to disambiguates this and everything else that could
reflect into an incompatibility of the eln (`system-configuration`
`emacs-version`).

As from time to time we can refine mechanisms like the load mechanism or
anything else and consequentially generating another incompatibility the
macro ABI_VERSION also contribute to HASH1.  I bump a new number there
each time I realize I'm changing something that introduce and
incompatibility.

To summarize eln was never designed for compatibility across versions or
configurations and HASH1 is made to guard against that.

> And in any case, if the triplet is in HASH1, why not remove its
> human-readable representation and save a few bytes?

At the time was thought to be convinient to have the triplet also as
human readable.  Indeed no problem from my side on removing it if this
is problematic (wasn't aware of this Windows limitation).

That said I think the main cure for this Windows file length issue is to
diet down the hash length to something more reasonable (16 or 8
characters?).

Regards

  Andrea

Re: [feature/native-comp] breakage on build

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: phillip.lord@russet.org.uk,  emacs-devel@gnu.org
> Date: Mon, 01 Feb 2021 11:01:04 +0000
> 
> each .eln while executing must indeed call into Emacs C code.  Call
> happens through function pointers, say we must call Fmessage, something
> like this will be rendered into the eln:
> 
> freloc_link_table->Fmessage (...)
> 
> 'freloc_link_table' is a pointer statically allocated within the eln
> object, it gets set while loading to point to the right data structure
> in memory.  This data structure is a C structure holding all function
> pointers to all Emacs primitive functions.
> 
> For this reason we must be sure that the definition of
> 'freloc_link_table' use while compiling the eln matches the one offered
> by the Emacs loading the eln file.
> 
> This is the main reason eln are not portable between different Emacs
> configurations.
> 
> HASH1 has the duty to disambiguates this and everything else that could
> reflect into an incompatibility of the eln (`system-configuration`
> `emacs-version`).
> 
> As from time to time we can refine mechanisms like the load mechanism or
> anything else and consequentially generating another incompatibility the
> macro ABI_VERSION also contribute to HASH1.  I bump a new number there
> each time I realize I'm changing something that introduce and
> incompatibility.
> 
> To summarize eln was never designed for compatibility across versions or
> configurations and HASH1 is made to guard against that.

OK, thanks.  I understand the reasons now.  However, I'm not sure we
should encode all this information in the .eln file's name.  For
starters, some of those reasons are also true for *.elc files, right?
For example, suppose that a primitive that the .el file calls changed
its signature in backward-incompatible ways -- the .el files which
call it need to be recompiled, or else they will fail, either silently
or noisily.  Right?  But we don't encode those details in the names of
the *.elc files, do we?  Why should the *.eln files be different?

Next, why not encode this information in some data in the *.eln files,
and have the code which loads the *.eln files check the compatibility
and reject the load if they don't match?  Why encode this information
in the file's name?

> > And in any case, if the triplet is in HASH1, why not remove its
> > human-readable representation and save a few bytes?
> 
> At the time was thought to be convinient to have the triplet also as
> human readable.  Indeed no problem from my side on removing it if this
> is problematic (wasn't aware of this Windows limitation).
> 
> That said I think the main cure for this Windows file length issue is to
> diet down the hash length to something more reasonable (16 or 8
> characters?).

I think we should do both.  The chances that the same cache directory
will be used for different architectures is pretty low anyway.

Re: [feature/native-comp] breakage on build

[akrl--- via Emacs development discussions.]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <akrl@sdf.org>
>> Cc: phillip.lord@russet.org.uk,  emacs-devel@gnu.org
>> Date: Mon, 01 Feb 2021 11:01:04 +0000
>> 
>> each .eln while executing must indeed call into Emacs C code.  Call
>> happens through function pointers, say we must call Fmessage, something
>> like this will be rendered into the eln:
>> 
>> freloc_link_table->Fmessage (...)
>> 
>> 'freloc_link_table' is a pointer statically allocated within the eln
>> object, it gets set while loading to point to the right data structure
>> in memory.  This data structure is a C structure holding all function
>> pointers to all Emacs primitive functions.
>> 
>> For this reason we must be sure that the definition of
>> 'freloc_link_table' use while compiling the eln matches the one offered
>> by the Emacs loading the eln file.
>> 
>> This is the main reason eln are not portable between different Emacs
>> configurations.
>> 
>> HASH1 has the duty to disambiguates this and everything else that could
>> reflect into an incompatibility of the eln (`system-configuration`
>> `emacs-version`).
>> 
>> As from time to time we can refine mechanisms like the load mechanism or
>> anything else and consequentially generating another incompatibility the
>> macro ABI_VERSION also contribute to HASH1.  I bump a new number there
>> each time I realize I'm changing something that introduce and
>> incompatibility.
>> 
>> To summarize eln was never designed for compatibility across versions or
>> configurations and HASH1 is made to guard against that.
>
> OK, thanks.  I understand the reasons now.  However, I'm not sure we
> should encode all this information in the .eln file's name.  For
> starters, some of those reasons are also true for *.elc files, right?
> For example, suppose that a primitive that the .el file calls changed
> its signature in backward-incompatible ways -- the .el files which
> call it need to be recompiled, or else they will fail, either silently
> or noisily.  Right?  But we don't encode those details in the names of
> the *.elc files, do we?  Why should the *.eln files be different?

I think with eln the situation is considerably more sentitive than the
elc one in this respect.  The reason is that we (on purpose) skip
Ffuncall to call directly into C code, as a consequence an
incompatibility will most certainly lead to a crash and not a runtime
error.

> Next, why not encode this information in some data in the *.eln files,
> and have the code which loads the *.eln files check the compatibility
> and reject the load if they don't match?  Why encode this information
> in the file's name?

We can imagine two alternative scenarios:

1- We merge HASH1 and HASH3

  - In this case everything will work as of now but certain operations
    will become more difficult, the typical example is manually removing
    all the eln left by an old Emacs configuration that is not anymore
    in use (me and others do this quite often).

2- We move HASH1 into the eln as metadata.

  - The limitation of 1 persists

  - To check that each eln is loadable one has: to 'dlopen' it, read the
    metadata ('dlsym'), verify and in case reject.  This is certainly
    way more expansive than directly searching for a given filename and
    might impact negatively startup time, especially on certain OSes
    where opening files is an expensive operation.

  - We'd have name clashes for different eln coming from different
    Emacs configurations/versions.

>> > And in any case, if the triplet is in HASH1, why not remove its
>> > human-readable representation and save a few bytes?
>> 
>> At the time was thought to be convinient to have the triplet also as
>> human readable.  Indeed no problem from my side on removing it if this
>> is problematic (wasn't aware of this Windows limitation).
>> 
>> That said I think the main cure for this Windows file length issue is to
>> diet down the hash length to something more reasonable (16 or 8
>> characters?).
>
> I think we should do both.  The chances that the same cache directory
> will be used for different architectures is pretty low anyway.

Okay.

Shall we pick a length?  8 characters is probably more than okay but we
can go for 16 if we prefer to stay on the safe side.

Thanks

  Andrea

Re: [feature/native-comp] breakage on build

[akrl--- via Emacs development discussions.]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <akrl@sdf.org>
>> Cc: phillip.lord@russet.org.uk,  emacs-devel@gnu.org
>> Date: Mon, 01 Feb 2021 16:20:17 +0000
>> 
>> > OK, thanks.  I understand the reasons now.  However, I'm not sure we
>> > should encode all this information in the .eln file's name.  For
>> > starters, some of those reasons are also true for *.elc files, right?
>> > For example, suppose that a primitive that the .el file calls changed
>> > its signature in backward-incompatible ways -- the .el files which
>> > call it need to be recompiled, or else they will fail, either silently
>> > or noisily.  Right?  But we don't encode those details in the names of
>> > the *.elc files, do we?  Why should the *.eln files be different?
>> 
>> I think with eln the situation is considerably more sentitive than the
>> elc one in this respect.  The reason is that we (on purpose) skip
>> Ffuncall to call directly into C code, as a consequence an
>> incompatibility will most certainly lead to a crash and not a runtime
>> error.
>
> Crash aside, I don't see the crucial difference, sorry.

Agree, for me runtime error vs unrecoverable crash is a crucial
difference than easily can drive to different design decisions.

> And if we
> detect incompatibility and prevent the crash, I seed no difference at
> all, as that can happen with *.elc files as well.
>
>> 2- We move HASH1 into the eln as metadata.
>> 
>>   - The limitation of 1 persists
>> 
>>   - To check that each eln is loadable one has: to 'dlopen' it, read the
>>     metadata ('dlsym'), verify and in case reject.  This is certainly
>>     way more expansive than directly searching for a given filename and
>>     might impact negatively startup time, especially on certain OSes
>>     where opening files is an expensive operation.
>
> How much more expensive?  I'd be surprised to hear it's significant,
> since in-memory operations are much faster than file I/O.  Is it
> possible to measure the difference, so that we could make a decision
> based on facts?

Not easily without implementing it.  Also we should then measure on all
supported platforms.  And still we'd have the first mentioned
limitation.

>>   - We'd have name clashes for different eln coming from different
>>     Emacs configurations/versions.
>
> A different version will have a different directory, like it does now
> with *.elc files.  As for different configurations: if you mean the
> same architecture, how probable is that, for normal users?  If the
> issue happens with Emacs developers, wed could design something
> special for that case.

Configuring with different flags can lead to a different set of exposed
primitives and as a consequence to eln incompatibility so yeah, I think
this is quite sensitive.

> I guess I'm saying that using the file names to record this
> information is not necessarily justified by the issues it is supposed
> to solve, and there are alternative solutions with smaller costs.
>
>> Shall we pick a length?  8 characters is probably more than okay but we
>> can go for 16 if we prefer to stay on the safe side.
>
> If 8 is enough, then why go for 16?  How is that safer?

I agree 8 is enough, 16 would be safer as reduces further the clash
probability but with 8 is already very low (1/1785793904896 if I'm not
wrong).

Regards

  Andrea

Re: [feature/native-comp] breakage on build

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:

>> Shall we pick a length?  8 characters is probably more than okay but we
>> can go for 16 if we prefer to stay on the safe side.
>
> If 8 is enough, then why go for 16?  How is that safer?


They don't have to all be the same length. IIUC, one distinguishes
between different versions of Emacs, the other between .el. There are
not that many of the former, but quite a few of the latter.

It's also possible to use both file names (for identification of the
right file) and internal data (to avoid crashes after collisions).

I see that they have removed this limit for Windows 10, at least if you
ask Windows really nicely. Bout time, been a pain for years.

Phil

Re: [feature/native-comp] breakage on build

[Eli Zaretskii]

> From: Phillip Lord <phillip.lord@russet.org.uk>
> Cc: Andrea Corallo <akrl@sdf.org>,  emacs-devel@gnu.org
> Date: Mon, 01 Feb 2021 19:06:48 +0000
> 
> I see that they have removed this limit for Windows 10, at least if you
> ask Windows really nicely. Bout time, been a pain for years.

AFAIK, it's off by default, and even if you turn it on, many/most
applications will be unable to handle such long file names, even on
Windows 10.  So we will have to deal with this issue for quite some
time.

Re: [feature/native-comp] breakage on build

[Stefan Monnier]

> I think with eln the situation is considerably more sentitive than the
> elc one in this respect.  The reason is that we (on purpose) skip
> Ffuncall to call directly into C code, as a consequence an
> incompatibility will most certainly lead to a crash and not
> a runtime error.

Indeed, it's more like the `pdmp` fingerprint check.

> 1- We merge HASH1 and HASH3
>
>   - In this case everything will work as of now but certain operations
>     will become more difficult, the typical example is manually removing
>     all the eln left by an old Emacs configuration that is not anymore
>     in use (me and others do this quite often).

Can we erase those files based on their `mtime` instead
(maybe comparing their `mtime` to that of the `emacs` executable)?

> 2- We move HASH1 into the eln as metadata.
>
>   - The limitation of 1 persists
>
>   - To check that each eln is loadable one has: to 'dlopen' it, read the
>     metadata ('dlsym'), verify and in case reject.  This is certainly
>     way more expansive than directly searching for a given filename and
>     might impact negatively startup time, especially on certain OSes
>     where opening files is an expensive operation.
>
>   - We'd have name clashes for different eln coming from different
>     Emacs configurations/versions.

We can combine that with a shorter hash in the file-name.  The short
hash in the file name will eliminate the vast majority of name clashes.


        Stefan

Re: [feature/native-comp] breakage on build

[akrl--- via Emacs development discussions.]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> 1- We merge HASH1 and HASH3
>>
>>   - In this case everything will work as of now but certain operations
>>     will become more difficult, the typical example is manually removing
>>     all the eln left by an old Emacs configuration that is not anymore
>>     in use (me and others do this quite often).
>
> Can we erase those files based on their `mtime` instead
> (maybe comparing their `mtime` to that of the `emacs` executable)?

Mmmh not really, eln files are created on demand.  I could have 2
Emacsen one used from 1st Jan to 1st Feb and the other one used from
15th Jan to 15th Feb.  The day I want to get rid of the eln from the 1st
Emacs would be hard to distinguish them based only on the mtime.

I suspect removing the hash1 directory would only introduce issues while
I must say to me is unclear which issue we'd solved.

Regards

  Andrea

Re: [feature/native-comp] breakage on build

[akrl--- via Emacs development discussions.]

Eli Zaretskii <eliz@gnu.org> writes:

[...]

>> > And in any case, if the triplet is in HASH1, why not remove its
>> > human-readable representation and save a few bytes?
>> 
>> At the time was thought to be convinient to have the triplet also as
>> human readable.  Indeed no problem from my side on removing it if this
>> is problematic (wasn't aware of this Windows limitation).
>> 
>> That said I think the main cure for this Windows file length issue is to
>> diet down the hash length to something more reasonable (16 or 8
>> characters?).
>
> I think we should do both.  The chances that the same cache directory
> will be used for different architectures is pretty low anyway.

Last question: shall we remove also the emacs version or only the
triplet?

Thanks

  Andrea

Re: [feature/native-comp] breakage on build

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: phillip.lord@russet.org.uk,  emacs-devel@gnu.org
> Date: Tue, 02 Feb 2021 20:26:54 +0000
> 
> Last question: shall we remove also the emacs version or only the
> triplet?

I think both, but if you can describe a situation where the user may
want to know the version to which the file belongs, let's discuss it.

Re: [feature/native-comp] breakage on build

[akrl--- via Emacs development discussions.]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <akrl@sdf.org>
>> Cc: phillip.lord@russet.org.uk,  emacs-devel@gnu.org
>> Date: Tue, 02 Feb 2021 20:26:54 +0000
>> 
>> Last question: shall we remove also the emacs version or only the
>> triplet?
>
> I think both, but if you can describe a situation where the user may
> want to know the version to which the file belongs, let's discuss it.

I can't come up with something other than what I've already mentioned so
I think we can remove both.

  Andrea

Re: [feature/native-comp] breakage on build

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <akrl@sdf.org>
>> Cc: phillip.lord@russet.org.uk,  emacs-devel@gnu.org
>> Date: Tue, 02 Feb 2021 20:26:54 +0000
>> 
>> Last question: shall we remove also the emacs version or only the
>> triplet?
>
> I think both, but if you can describe a situation where the user may
> want to know the version to which the file belongs, let's discuss it.


At the moment, .eln-cache is going to just pile up in size, no, even for
normal users over versions. So, I guess the version number is useful to
know which is oldest.

Still there are other ways to achieve the same thing.

Phil

Re: [feature/native-comp] breakage on build

[akrl--- via Emacs development discussions.]

Phillip Lord <phillip.lord@russet.org.uk> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> From: Andrea Corallo <akrl@sdf.org>
>>> Cc: phillip.lord@russet.org.uk,  emacs-devel@gnu.org
>>> Date: Tue, 02 Feb 2021 20:26:54 +0000
>>> 
>>> Last question: shall we remove also the emacs version or only the
>>> triplet?
>>
>> I think both, but if you can describe a situation where the user may
>> want to know the version to which the file belongs, let's discuss it.
>
>
> At the moment, .eln-cache is going to just pile up in size, no, even for
> normal users over versions. So, I guess the version number is useful to
> know which is oldest.

I pushed two commits to diet down the hashes and remove the triplet from
eln filenames, now looking like the following:

~/.emacs.d/eln-cache/28.0.50-a45dccd2/advice-791b3e89-08434428.eln

Thinking about I agree having the version there is still helpful so I
letf it for now, anyway I've no strong feeling against removing it if
Eli prefers.

Thanks

  Andrea

Re: [feature/native-comp] breakage on build

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: Eli Zaretskii <eliz@gnu.org>,  emacs-devel@gnu.org
> Date: Wed, 03 Feb 2021 20:27:00 +0000
> 
> I pushed two commits to diet down the hashes and remove the triplet from
> eln filenames, now looking like the following:
> 
> ~/.emacs.d/eln-cache/28.0.50-a45dccd2/advice-791b3e89-08434428.eln
> 
> Thinking about I agree having the version there is still helpful so I
> letf it for now, anyway I've no strong feeling against removing it if
> Eli prefers.

Thanks.  Let's leave it at that, and see if it causes problems in some
situation.

Re: [feature/native-comp] breakage on build

[Phillip Lord]

akrl--- via "Emacs development discussions." <emacs-devel@gnu.org>
writes:

> Phillip Lord <phillip.lord@russet.org.uk> writes:
>> At the moment, .eln-cache is going to just pile up in size, no, even for
>> normal users over versions. So, I guess the version number is useful to
>> know which is oldest.
>
> I pushed two commits to diet down the hashes and remove the triplet from
> eln filenames, now looking like the following:
>
> ~/.emacs.d/eln-cache/28.0.50-a45dccd2/advice-791b3e89-08434428.eln
>
> Thinking about I agree having the version there is still helpful so I
> letf it for now, anyway I've no strong feeling against removing it if
> Eli prefers.


I've now got native-comp building successfully to a package.

At the moment, the compilation process only runs successfully when run
from an mingw64 shell, so I haven't got the dependencies working
correctly yet.

The error I am getting is this:

Compiling c:/msys64/home/Administrator/emacs-upload/emacs-28.0.50-snapshot-feature_native-comp-windows-2021-02-04/share/emacs/28.0.50/lisp/emacs-lisp/cl-lib.el...
compiling to c:/Users/Administrator/AppData/Roaming/.emacs.d/eln-cache/28.0.50-14889799/cl-lib-8b938900-d1df28b7.eln
x86_64-w64-mingw32-gcc-10.2.0: fatal error: cannot execute 'as': CreateProcess: No such file or directory
compilation terminated.


I've tried putting as.exe manually into the package but that isn't
enough. as.exe doesn't seem to have any dll dependencies itself.

Andrea what other tools does native-comp require to have around? Is
there a good way to debug the compilation process, so I can work out
what the file missing is?

Phil

Re: [feature/native-comp] breakage on build

[Eli Zaretskii]

> From: Phillip Lord <phillip.lord@russet.org.uk>
> Cc: Andrea Corallo <akrl@sdf.org>,  Eli Zaretskii <eliz@gnu.org>
> Date: Thu, 04 Feb 2021 14:39:50 +0000
> 
> At the moment, the compilation process only runs successfully when run
> from an mingw64 shell

What does that mean, exactly?  Isn't libgccjit being invoked by Emacs
itself? then how does the shell, any shell, enter the picture?

> The error I am getting is this:
> 
> Compiling c:/msys64/home/Administrator/emacs-upload/emacs-28.0.50-snapshot-feature_native-comp-windows-2021-02-04/share/emacs/28.0.50/lisp/emacs-lisp/cl-lib.el...
> compiling to c:/Users/Administrator/AppData/Roaming/.emacs.d/eln-cache/28.0.50-14889799/cl-lib-8b938900-d1df28b7.eln
> x86_64-w64-mingw32-gcc-10.2.0: fatal error: cannot execute 'as': CreateProcess: No such file or directory
> compilation terminated.

Why do you see the "x86_64-w64-mingw32-gcc-10.2.0" thing? are you
cross-building Emacs or something?

> I've tried putting as.exe manually into the package but that isn't
> enough. as.exe doesn't seem to have any dll dependencies itself.

Is as.exe on PATH?  If not, where is it?

Re: [feature/native-comp] breakage on build

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Phillip Lord <phillip.lord@russet.org.uk>
>> Cc: Andrea Corallo <akrl@sdf.org>,  Eli Zaretskii <eliz@gnu.org>
>> Date: Thu, 04 Feb 2021 14:39:50 +0000
>> 
>> At the moment, the compilation process only runs successfully when run
>> from an mingw64 shell
>
> What does that mean, exactly?  Isn't libgccjit being invoked by Emacs
> itself? then how does the shell, any shell, enter the picture?

If you invoke runemacs.exe it picks up its environment from the mingw64
shell. That means anything installed on the local machine to the
/mingw64, any dlls, anything will be available to Emacs, even if they
are not present in the bundle.

If you invoke runemacs.exe by clicking on it in explorer, then it only
looks for DLLs in the same directory (and what ever windows normally
does I guess).

So, I have the right dependencies installed on my build machine, but
haven't packaged them up correctly.


>> The error I am getting is this:
>> 
>> Compiling
>> c:/msys64/home/Administrator/emacs-upload/emacs-28.0.50-snapshot-feature_native-comp-windows-2021-02-04/share/emacs/28.0.50/lisp/emacs-lisp/cl-lib.el...
>> compiling to
>> c:/Users/Administrator/AppData/Roaming/.emacs.d/eln-cache/28.0.50-14889799/cl-lib-8b938900-d1df28b7.eln
>> x86_64-w64-mingw32-gcc-10.2.0: fatal error: cannot execute 'as':
>> CreateProcess: No such file or directory
>> compilation terminated.
>
> Why do you see the "x86_64-w64-mingw32-gcc-10.2.0" thing? are you
> cross-building Emacs or something?

Hmmm. I don't know why I see that. No, I am not cross building.


>> I've tried putting as.exe manually into the package but that isn't
>> enough. as.exe doesn't seem to have any dll dependencies itself.
>
> Is as.exe on PATH?  If not, where is it?

Not in my package at the moment. I have tried adding it next to the
runemacs.exe, and expected that to work, but it didn't.

When I become world dictator, I shall people from naming their software
after common words!

Phil

Re: [feature/native-comp] breakage on build

[Eli Zaretskii]

> From: Phillip Lord <phillip.lord@russet.org.uk>
> Cc: emacs-devel@gnu.org,  akrl@sdf.org
> Date: Thu, 04 Feb 2021 15:52:54 +0000
> 
> If you invoke runemacs.exe it picks up its environment from the mingw64
> shell. That means anything installed on the local machine to the
> /mingw64, any dlls, anything will be available to Emacs, even if they
> are not present in the bundle.
> 
> If you invoke runemacs.exe by clicking on it in explorer, then it only
> looks for DLLs in the same directory (and what ever windows normally
> does I guess).
> 
> So, I have the right dependencies installed on my build machine, but
> haven't packaged them up correctly.

The DLLs should be on PATH.  It is not enough to have them in the same
directory as the executable.

> >> I've tried putting as.exe manually into the package but that isn't
> >> enough. as.exe doesn't seem to have any dll dependencies itself.
> >
> > Is as.exe on PATH?  If not, where is it?
> 
> Not in my package at the moment.

I suggest to put as.exe on PATH.

Re: [feature/native-comp] breakage on build

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

>> So, I have the right dependencies installed on my build machine, but
>> haven't packaged them up correctly.
>
> The DLLs should be on PATH.  It is not enough to have them in the same
> directory as the executable.

This is not correct. In fact, the application's directory is the first
place where the dll is searched, unless the application explicitly
changes the search strategy.

>> >> I've tried putting as.exe manually into the package but that isn't
>> >> enough. as.exe doesn't seem to have any dll dependencies itself.
>> >
>> > Is as.exe on PATH?  If not, where is it?
>> 
>> Not in my package at the moment.
>
> I suggest to put as.exe on PATH.

This would be problematic. In general, applications that put things on
PATH (or modify PATH in general) are frowned upon by many power users,
for good reasons.

Re: [feature/native-comp] breakage on build

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Thu, 04 Feb 2021 21:24:38 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> So, I have the right dependencies installed on my build machine, but
> >> haven't packaged them up correctly.
> >
> > The DLLs should be on PATH.  It is not enough to have them in the same
> > directory as the executable.
> 
> This is not correct. In fact, the application's directory is the first
> place where the dll is searched, unless the application explicitly
> changes the search strategy.

The original message to which I was replying wasn't talking about the
DLLs in the same directory as Emacs, it was talking about MinGW64 DLLs
that are outside of that directory.

> > I suggest to put as.exe on PATH.
> 
> This would be problematic. In general, applications that put things on
> PATH (or modify PATH in general) are frowned upon by many power users,
> for good reasons.

Do you have any alternative suggestion that won't be frowned upon?
Then please describe it.

In general, as.exe is part of Binutils, and as such, is installed in
$exec_prefix/bin.  It is not an auxiliary program installed in
$exec_prefix/libexec.  So IMO putting it on PATH is exactly TRT.

And if you think Emacs shouldn't put things on PATH, then what should
we do with what we already put there (runemacs.exe, emacsclient.exe,
addpm.exe, etags.exe, and a few others)?

Re: [feature/native-comp] breakage on build

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

>> > I suggest to put as.exe on PATH.
>> 
>> This would be problematic. In general, applications that put things on
>> PATH (or modify PATH in general) are frowned upon by many power users,
>> for good reasons.
>
> Do you have any alternative suggestion that won't be frowned upon?
> Then please describe it.

Put all binaries on the same directory as emacs.exe/runemacs.exe, except
probably those which are executed by a driver and that driver expects
them to be on certain path relative to the driver's path (AFAIK gcc.exe
does that, dunno about libgccjit.dll (or whatever its name is)).

> In general, as.exe is part of Binutils, and as such, is installed in
> $exec_prefix/bin.  It is not an auxiliary program installed in
> $exec_prefix/libexec.  So IMO putting it on PATH is exactly TRT.
>
> And if you think Emacs shouldn't put things on PATH, then what should
> we do with what we already put there (runemacs.exe, emacsclient.exe,
> addpm.exe, etags.exe, and a few others)?

My Emacs installs on Windows never required to touch PATH. Now that I
use Emacs from the MSYS2/MinGW-w64 environment (which means that
runemacs.exe is executed from the mingw64/bin directory) PATH and
exec-path are modified by my emacs.el but, outside of Emacs, PATH is
unaltered (it contains nothing related to MSYS2/MinGW-w64).

It is true that currently the user must modify exec-path to run
executables that exist on the same directory as emacs.exe (when executed
from a Windows shortcut, not from a MSYS2/Mingw-w64 shell), but that is
a burden that we can solve.

Re: [feature/native-comp] breakage on build

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Thu, 04 Feb 2021 23:08:15 +0100
> 
> >> This would be problematic. In general, applications that put things on
> >> PATH (or modify PATH in general) are frowned upon by many power users,
> >> for good reasons.
> >
> > Do you have any alternative suggestion that won't be frowned upon?
> > Then please describe it.
> 
> Put all binaries on the same directory as emacs.exe/runemacs.exe

That won't work, not in general: searching executables doesn't
consider the directory from which the program was invoked, it only
considers the current working directory.

> except
> probably those which are executed by a driver and that driver expects
> them to be on certain path relative to the driver's path (AFAIK gcc.exe
> does that, dunno about libgccjit.dll (or whatever its name is)).

All of GCC does that.  Which means people who have a full development
environment installed will now need to have two copies of it, at least
potentially, unless they configure their directories in a very
specific manner.  I don't see how this is better in general than
telling people to put stuff on PATH.

> > And if you think Emacs shouldn't put things on PATH, then what should
> > we do with what we already put there (runemacs.exe, emacsclient.exe,
> > addpm.exe, etags.exe, and a few others)?
> 
> My Emacs installs on Windows never required to touch PATH. Now that I
> use Emacs from the MSYS2/MinGW-w64 environment (which means that
> runemacs.exe is executed from the mingw64/bin directory) PATH and
> exec-path are modified by my emacs.el but, outside of Emacs, PATH is
> unaltered (it contains nothing related to MSYS2/MinGW-w64).
> 
> It is true that currently the user must modify exec-path to run
> executables that exist on the same directory as emacs.exe (when executed
> from a Windows shortcut, not from a MSYS2/Mingw-w64 shell), but that is
> a burden that we can solve.

This arrangement evidently works for you, but it doesn't mean it is
the best one.  For example, AFAIU it generally doesn't allow invoking
the commands outside of Emacs.  I don't think it's a good idea to have
all those useful programs ghettoized to work only from Emacs, because
Emacs is just one tool in the user's tool-chest.

Re: [feature/native-comp] breakage on build

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Óscar Fuentes <ofv@wanadoo.es>
>> Date: Thu, 04 Feb 2021 23:08:15 +0100
>> 
>> >> This would be problematic. In general, applications that put things on
>> >> PATH (or modify PATH in general) are frowned upon by many power users,
>> >> for good reasons.
>> >
>> > Do you have any alternative suggestion that won't be frowned upon?
>> > Then please describe it.
>> 
>> Put all binaries on the same directory as emacs.exe/runemacs.exe
>
> That won't work, not in general: searching executables doesn't
> consider the directory from which the program was invoked, it only
> considers the current working directory.

Yes, that's because I suggest adding emacs.exe's directory to exec-path
(on Windows).

>> except
>> probably those which are executed by a driver and that driver expects
>> them to be on certain path relative to the driver's path (AFAIK gcc.exe
>> does that, dunno about libgccjit.dll (or whatever its name is)).
>
> All of GCC does that.  Which means people who have a full development
> environment installed will now need to have two copies of it, at least
> potentially, unless they configure their directories in a very
> specific manner.  I don't see how this is better in general than
> telling people to put stuff on PATH.

I'm not sure what's your objection here, but, as you very well known,
messing with PATH has a very real potential of subtly breaking things
when different applications depends on it, so this approach doesn't look
very appealing to me (speaking rhetorically, many years ago I learned
the hard way that keeping the contents of PATH to a bare minimum saves a
lot of teeth gnashing and hair pulling and even head banging.)

>> > And if you think Emacs shouldn't put things on PATH, then what should
>> > we do with what we already put there (runemacs.exe, emacsclient.exe,
>> > addpm.exe, etags.exe, and a few others)?
>> 
>> My Emacs installs on Windows never required to touch PATH. Now that I
>> use Emacs from the MSYS2/MinGW-w64 environment (which means that
>> runemacs.exe is executed from the mingw64/bin directory) PATH and
>> exec-path are modified by my emacs.el but, outside of Emacs, PATH is
>> unaltered (it contains nothing related to MSYS2/MinGW-w64).
>> 
>> It is true that currently the user must modify exec-path to run
>> executables that exist on the same directory as emacs.exe (when executed
>> from a Windows shortcut, not from a MSYS2/Mingw-w64 shell), but that is
>> a burden that we can solve.
>
> This arrangement evidently works for you, but it doesn't mean it is
> the best one.  For example, AFAIU it generally doesn't allow invoking
> the commands outside of Emacs.  I don't think it's a good idea to have
> all those useful programs ghettoized to work only from Emacs, because
> Emacs is just one tool in the user's tool-chest.

IIUC one of the problems Phillip has is to include certain tools on the
Emacs distribution (either on the binary package he uploads or on
auxiliary ones that can be installed on request, i.e. MSYS2/MinGW-w64)
on such a way that Things Work (TM). Setting exec-path would solve this.

Furthermore, if the MSYS2/MinGW-w64 way of providing tools ends winning,
Phillip could just simply build an MSYS2 package which current MSYS2
users would download and install while non-MSYS2 users would download an
"installer" that installs MSYS2/MinGW-w64 (from their servers) and then
the Emacs MSYS2 package on top. This has the advantage that all
dependencies can be listed on the package and `pacman' (MSYS2 package
manager) will install them automatically. With this and the mentioned
modification to exec-path, everything works, no PATH meddling required.

Sure enough, there are people that prefer the traditional tarball with
just Emacs binaries, but IMHO it is fair to say that those users must
deal with the dependencies themselves.

Re: [feature/native-comp] breakage on build

[Phillip Lord]

Óscar Fuentes <ofv@wanadoo.es> writes:
>> This arrangement evidently works for you, but it doesn't mean it is
>> the best one.  For example, AFAIU it generally doesn't allow invoking
>> the commands outside of Emacs.  I don't think it's a good idea to have
>> all those useful programs ghettoized to work only from Emacs, because
>> Emacs is just one tool in the user's tool-chest.
>
> IIUC one of the problems Phillip has is to include certain tools on the
> Emacs distribution (either on the binary package he uploads or on
> auxiliary ones that can be installed on request, i.e. MSYS2/MinGW-w64)
> on such a way that Things Work (TM). Setting exec-path would solve this.
>
> Furthermore, if the MSYS2/MinGW-w64 way of providing tools ends winning,
> Phillip could just simply build an MSYS2 package which current MSYS2
> users would download and install while non-MSYS2 users would download an
> "installer" that installs MSYS2/MinGW-w64 (from their servers) and then
> the Emacs MSYS2 package on top. This has the advantage that all
> dependencies can be listed on the package and `pacman' (MSYS2 package
> manager) will install them automatically. With this and the mentioned
> modification to exec-path, everything works, no PATH meddling required.
>
> Sure enough, there are people that prefer the traditional tarball with
> just Emacs binaries, but IMHO it is fair to say that those users must
> deal with the dependencies themselves.


I have made life more complicated, I am afraid, by doing two things at
once. There are three issues here which are related but different.

First is Emacs with associated DLLs. I have just moved this to using
ntldd to work out the dependent DLLs. That seems to be working fine, and
seems better to me that what we had with Emacs-26 which is half of an
msys2 install stuff inside a zip file called "Emacs".

Second, is native-comp. It appears that, as well as the DLLs, libgccjit
also needs some tools locally (as, ld at least). I don't know what those
all are yet, and that is going to need some working out. When I have we
need a way of making them available so that Emacs can run.

The third is how I can make a more tooled up Emacs with associated
tooling. My package of Emacs once installed cannot check spelling which
seems pretty poor for a text editor. Nor interact with a git repo which
is pretty poor for a code editor.

It's attractive in the first instance to conflate these three. We could
install all the dependencies for Emacs and all the tools we want using
pacman. But, I think this is wrong, because Emacs itself
(i.e. emacs.exe) is a fixed point in time; it needs to have a fixed set
of dependencies and they should not update for a given emacs.exe. This
fits poorly with the msys2 rolling release. So, problem one requires us
to extract dependencies from a given release of msys2 then freeze them.
I expect problem two will be the same, but I haven't fixed that yet.

Problem three is different. I think here the solution is to install
msys2 inside the Emacs installation, then fiddle with things so that
Emacs uses it. If this works correctly, then an Emacs installation will
also include and use aspell or hunspell and git. And if we want more
packages, we could run pacman and install anything we chose. It would
even be possible to update things like libjpeg (as a trivial example)
inside this msys without having any impact on Emacs because it would use
the fixed libjpeg that it was installed with. There are undoubtedly lots
of details to worry about here, but I think I know at least how to do
the install process now.

At the moment, number one is working, number two I think needs to be
solved as lots of people will want native comp on windows, and number
three is (and may remain) aspirational.

Phil

Re: [feature/native-comp] breakage on build

[Óscar Fuentes]

Phillip Lord <phillip.lord@russet.org.uk> writes:

> It's attractive in the first instance to conflate these three. We
> could install all the dependencies for Emacs and all the tools we want
> using pacman. But, I think this is wrong, because Emacs itself (i.e.
> emacs.exe) is a fixed point in time; it needs to have a fixed set of
> dependencies and they should not update for a given emacs.exe. This
> fits poorly with the msys2 rolling release. So, problem one requires
> us to extract dependencies from a given release of msys2 then freeze
> them.

Required rebuilds are relatively rare. For instance, AFAIR the only time
that the MSYS2 emacs package had trouble with updated dependencies was
when ImageMagick went to version 7, which was incompatible with Emacs.
And that was a problem with an optional dependency that affected all
platforms, it was not at all like emacs -Q crashed at startup.

Second, it is not mandatory to do `pacman -Suy' every morning. I mean,
if a user updates his packages, it is at his own risk.

Third, by delegating the updates to the MSYS2 folks you don't need to
worry about security and bug fixes on all those dependencies.

Agreed, MSYS2 is a fat beast to depend on, but it provides a truckload
of functionality and saves you from dealing with a tangled hell of
dependencies. It is reasonable to expect that those dependencies will
grow and become more complicated on the future (native-comp is a glaring
example.)

> At the moment, number one is working, number two I think needs to be
> solved as lots of people will want native comp on windows, and number
> three is (and may remain) aspirational.

The goal you are pursuing is difficult enough and, IMHO, you are
complicating it further by adding (somewhat fuzzy) requirements that lie
beyond the point that seems reasonable to me, speaking as an Emacs user.

Re: [feature/native-comp] breakage on build

[Phillip Lord]

Óscar Fuentes <ofv@wanadoo.es> writes:

> Phillip Lord <phillip.lord@russet.org.uk> writes:
>
>> It's attractive in the first instance to conflate these three. We
>> could install all the dependencies for Emacs and all the tools we want
>> using pacman. But, I think this is wrong, because Emacs itself (i.e.
>> emacs.exe) is a fixed point in time; it needs to have a fixed set of
>> dependencies and they should not update for a given emacs.exe. This
>> fits poorly with the msys2 rolling release. So, problem one requires
>> us to extract dependencies from a given release of msys2 then freeze
>> them.
>
> Agreed, MSYS2 is a fat beast to depend on, but it provides a truckload
> of functionality and saves you from dealing with a tangled hell of
> dependencies. It is reasonable to expect that those dependencies will
> grow and become more complicated on the future (native-comp is a glaring
> example.)


Well I agree. But "fat beast" is an understatement. If I install msys2,
Emacs and all the dependencies, it comes in at nearly 2Gb on disk. The
install takes around half an hour on my machine (and only that fast
because I turn of the Windows antimalware service). It also includes a
pile of stuff that the user might not want (i.e. python or ruby) because
they have their own installation already. So I don't think we can avoid
the necessity for an Emacs with dependencies. native-comp is, indeed, a
glaring example, but it's also the only one. Adding the other
dependencies that have arisen (jansson, harfbuzz and gmp) has been
straightforward enough.


But, you are correct, a value add with all the tools installation of
Emacs might as well just be an msys2 install running. I believe you
already have already done that work! Or a powershell script that pulls
down msys2 installs it and dependencies and then pulls down
Emacs-no-deps over the top might be the way to go.



>> At the moment, number one is working, number two I think needs to be
>> solved as lots of people will want native comp on windows, and number
>> three is (and may remain) aspirational.
>
> The goal you are pursuing is difficult enough and, IMHO, you are
> complicating it further by adding (somewhat fuzzy) requirements that lie
> beyond the point that seems reasonable to me, speaking as an Emacs user.


Maybe. What I see is people using Emacs and getting their spelling wrong
because there is no spell checker.

Phil

Re: [feature/native-comp] breakage on build

[Óscar Fuentes]

Phillip Lord <phillip.lord@russet.org.uk> writes:

> Óscar Fuentes <ofv@wanadoo.es> writes:
>
>> Phillip Lord <phillip.lord@russet.org.uk> writes:
>>
>>> It's attractive in the first instance to conflate these three. We
>>> could install all the dependencies for Emacs and all the tools we want
>>> using pacman. But, I think this is wrong, because Emacs itself (i.e.
>>> emacs.exe) is a fixed point in time; it needs to have a fixed set of
>>> dependencies and they should not update for a given emacs.exe. This
>>> fits poorly with the msys2 rolling release. So, problem one requires
>>> us to extract dependencies from a given release of msys2 then freeze
>>> them.
>>
>> Agreed, MSYS2 is a fat beast to depend on, but it provides a truckload
>> of functionality and saves you from dealing with a tangled hell of
>> dependencies. It is reasonable to expect that those dependencies will
>> grow and become more complicated on the future (native-comp is a glaring
>> example.)
>
>
> Well I agree. But "fat beast" is an understatement. If I install msys2,
> Emacs and all the dependencies, it comes in at nearly 2Gb on disk. The
> install takes around half an hour on my machine (and only that fast
> because I turn of the Windows antimalware service).

The dominating factor on my experience is the download time. Telling
pacman to use the right mirror solves this. (See
/etc/pacman.d/mirrorlist.* )

> It also includes a
> pile of stuff that the user might not want (i.e. python or ruby) because
> they have their own installation already.

There are some serious python buffs among the MSYS2 maintainers and it
is listed as a dependency of anything that could remotely use it, but
ruby is not in my installs (searched for ruby.*) and they certainly
contain much more than Emacs needs, although I don't install all
optional dependencies.

Maybe your dependency list would benefit from some review. Just show it.

> So I don't think we can avoid
> the necessity for an Emacs with dependencies. native-comp is, indeed, a
> glaring example, but it's also the only one. Adding the other
> dependencies that have arisen (jansson, harfbuzz and gmp) has been
> straightforward enough.
>
>
> But, you are correct, a value add with all the tools installation of
> Emacs might as well just be an msys2 install running. I believe you
> already have already done that work! Or a powershell script that pulls
> down msys2 installs it and dependencies and then pulls down
> Emacs-no-deps over the top might be the way to go.

I have a trivial bash script that installs several packages. I use it
after a fresh MSYS2 install and then I'm ready to go. Downloading MSYS2,
installing it, running the script... it all takes about 30 seconds of
attended work. It's that simple.

>>> At the moment, number one is working, number two I think needs to be
>>> solved as lots of people will want native comp on windows, and number
>>> three is (and may remain) aspirational.
>>
>> The goal you are pursuing is difficult enough and, IMHO, you are
>> complicating it further by adding (somewhat fuzzy) requirements that lie
>> beyond the point that seems reasonable to me, speaking as an Emacs user.
>
>
> Maybe. What I see is people using Emacs and getting their spelling wrong
> because there is no spell checker.

This Emacs has an spell checker and I get spelling wrong, as well as
grammar and everything else :-)

The route of creating an MSYS2 binary package (well, two, if you wish to
support i686) is as easy as it gets: create the package (I can help you
with it if the current PKGBUILD for mingw-w64-emacs-git in
https://github.com/msys2/MINGW-packages-dev/tree/master/mingw-w64-emacs-git
needs work). Then instruct the users to install MSYS2, download your
binary package and install it with pacman (copy and paste a command on
the MSYS2 shell). The most difficult step is to create a desktop icon
for runemacs.exe, which can be manually accomplished on the usual way
(and we used to have a tool for that, IIRC). That's about two minutes of
attended work.

At that point, you solved the dependencies problem, the source code
availability problem (for the dependencies) and delegated lots of
maintenance work.

But we need /mingw64/bin (its native equivalent, actually) on exec-path
for the .exe dependencies to work. We can tell the users to modify their
.emacs.el, but it would be much nicer if Emacs did that itself.

Oh, and we still miss some dependencies, like `find'. But that can be
added too, once we (the MSYS2 contributors) check that it will not break
other things.

Oh-2, currently MSYS2's Emacs does not install some things (e/ctags,
IIRC) because there are better alternatives on the repository.

Re: [feature/native-comp] breakage on build

[Phillip Lord]

Óscar Fuentes <ofv@wanadoo.es> writes:

> Phillip Lord <phillip.lord@russet.org.uk> writes:
>
>> Óscar Fuentes <ofv@wanadoo.es> writes:
>>
>> Well I agree. But "fat beast" is an understatement. If I install msys2,
>> Emacs and all the dependencies, it comes in at nearly 2Gb on disk. The
>> install takes around half an hour on my machine (and only that fast
>> because I turn of the Windows antimalware service).
>
> The dominating factor on my experience is the download time. Telling
> pacman to use the right mirror solves this. (See
> /etc/pacman.d/mirrorlist.* )

I am running it in an under powered VM, which probably explains the
difference.


>> It also includes a
>> pile of stuff that the user might not want (i.e. python or ruby) because
>> they have their own installation already.
>
> There are some serious python buffs among the MSYS2 maintainers and it
> is listed as a dependency of anything that could remotely use it, but
> ruby is not in my installs (searched for ruby.*) and they certainly
> contain much more than Emacs needs, although I don't install all
> optional dependencies.
>
> Maybe your dependency list would benefit from some review. Just show it.

I just said "ruby" as it appeared in the corner of my eye.

The full dependency list is this:

mingw-w64-x86_64-giflib
mingw-w64-x86_64-gnutls
mingw-w64-x86_64-harfbuzz
mingw-w64-x86_64-jansson
mingw-w64-x86_64-lcms2
mingw-w64-x86_64-libjpeg-turbo
mingw-w64-x86_64-libpng
mingw-w64-x86_64-librsvg
mingw-w64-x86_64-libtiff
mingw-w64-x86_64-libxml2
mingw-w64-x86_64-xpm-nox

I notice that it's missing gmp which I think it should have although
it's gets it transitively. On native-comp this also needs:

mingw-w64-x86_64-libgccjit


>> But, you are correct, a value add with all the tools installation of
>> Emacs might as well just be an msys2 install running. I believe you
>> already have already done that work! Or a powershell script that pulls
>> down msys2 installs it and dependencies and then pulls down
>> Emacs-no-deps over the top might be the way to go.
>
> I have a trivial bash script that installs several packages. I use it
> after a fresh MSYS2 install and then I'm ready to go. Downloading MSYS2,
> installing it, running the script... it all takes about 30 seconds of
> attended work. It's that simple.

Yes. I think that is a good way to go; I was looking at something
similar, except it would install msys2 as well over the top of an Emacs
install. And then update the msys2 install to be useful.


>>>> At the moment, number one is working, number two I think needs to be
>>>> solved as lots of people will want native comp on windows, and number
>>>> three is (and may remain) aspirational.
>>>
>>> The goal you are pursuing is difficult enough and, IMHO, you are
>>> complicating it further by adding (somewhat fuzzy) requirements that lie
>>> beyond the point that seems reasonable to me, speaking as an Emacs user.
>>
>>
>> Maybe. What I see is people using Emacs and getting their spelling wrong
>> because there is no spell checker.
>
> This Emacs has an spell checker and I get spelling wrong, as well as
> grammar and everything else :-)
>
> The route of creating an MSYS2 binary package (well, two, if you wish to
> support i686) is as easy as it gets: create the package (I can help you
> with it if the current PKGBUILD for mingw-w64-emacs-git in
> https://github.com/msys2/MINGW-packages-dev/tree/master/mingw-w64-emacs-git
> needs work). Then instruct the users to install MSYS2, download your
> binary package and install it with pacman (copy and paste a command on
> the MSYS2 shell). The most difficult step is to create a desktop icon
> for runemacs.exe, which can be manually accomplished on the usual way
> (and we used to have a tool for that, IIRC). That's about two minutes of
> attended work.


I was thinking of doing it the other way up. We install Emacs as now
(either unzipping or using the installer). Then, Emacs runs, detects the
first run and says "do you want to install msys2 also". If it does, then
away we go, otherwise, we leave as is -- an editor rather than a pilot
of multiple tools.

My original though was to bundle this into the install, but having done
a little bit of picking, I think a better solution would be to have the
first run of Emacs install a package from ELPA, which would actually do
the msys install.


> At that point, you solved the dependencies problem, the source code
> availability problem (for the dependencies) and delegated lots of
> maintenance work.
>
> But we need /mingw64/bin (its native equivalent, actually) on exec-path
> for the .exe dependencies to work. We can tell the users to modify their
> .emacs.el, but it would be much nicer if Emacs did that itself.

I would agree. This would also mean we could point Emacs as an existing
msys2 installation. Or install Emacs in one place and msys2 in another.


> Oh, and we still miss some dependencies, like `find'. But that can be
> added too, once we (the MSYS2 contributors) check that it will not break
> other things.
>
> Oh-2, currently MSYS2's Emacs does not install some things (e/ctags,
> IIRC) because there are better alternatives on the repository.

And some critical things, like git which is not available as a mingw64
package; I have knocked together an mingw package for it; I have no idea
whether the msys2 maintainers would be interested in it.

Are you interested in helping with something along these lines? I have
only so much time to give to Emacs and it will take me sometime to get
any of this done. If so, we can write up a slightly more detailed plan.

Phil

Re: [feature/native-comp] breakage on build

[Óscar Fuentes]

Phillip Lord <phillip.lord@russet.org.uk> writes:

>> Maybe your dependency list would benefit from some review. Just show it.
>
> I just said "ruby" as it appeared in the corner of my eye.
>
> The full dependency list is this:
>
> mingw-w64-x86_64-giflib
> mingw-w64-x86_64-gnutls
> mingw-w64-x86_64-harfbuzz
> mingw-w64-x86_64-jansson
> mingw-w64-x86_64-lcms2
> mingw-w64-x86_64-libjpeg-turbo
> mingw-w64-x86_64-libpng
> mingw-w64-x86_64-librsvg
> mingw-w64-x86_64-libtiff
> mingw-w64-x86_64-libxml2
> mingw-w64-x86_64-xpm-nox
>
> I notice that it's missing gmp which I think it should have although
> it's gets it transitively. On native-comp this also needs:

gmp is a dependency of gcc-libs, and gcc-libs is a dependency of almost
every other package that contains binary executable code.

> mingw-w64-x86_64-libgccjit

That depends on gcc, and MSYS2 gcc includes C++ plus quite a few package
dependencies. No wonder that the final result uses so much disk space.

>> The route of creating an MSYS2 binary package (well, two, if you wish to
>> support i686) is as easy as it gets: create the package (I can help you
>> with it if the current PKGBUILD for mingw-w64-emacs-git in
>> https://github.com/msys2/MINGW-packages-dev/tree/master/mingw-w64-emacs-git
>> needs work). Then instruct the users to install MSYS2, download your
>> binary package and install it with pacman (copy and paste a command on
>> the MSYS2 shell). The most difficult step is to create a desktop icon
>> for runemacs.exe, which can be manually accomplished on the usual way
>> (and we used to have a tool for that, IIRC). That's about two minutes of
>> attended work.
>
>
> I was thinking of doing it the other way up. We install Emacs as now
> (either unzipping or using the installer). Then, Emacs runs, detects the
> first run and says "do you want to install msys2 also". If it does, then
> away we go, otherwise, we leave as is -- an editor rather than a pilot
> of multiple tools.

One caveat with this approach is that you can end (you *will* end) with
two copies of the same dependencies on two places, for the dependencies
that you put on Emacs' zip distribution.

> My original though was to bundle this into the install, but having done
> a little bit of picking, I think a better solution would be to have the
> first run of Emacs install a package from ELPA, which would actually do
> the msys install.

This is easier said than done :-)

>> Oh, and we still miss some dependencies, like `find'. But that can be
>> added too, once we (the MSYS2 contributors) check that it will not break
>> other things.
>>
>> Oh-2, currently MSYS2's Emacs does not install some things (e/ctags,
>> IIRC) because there are better alternatives on the repository.
>
> And some critical things, like git which is not available as a mingw64
> package; I have knocked together an mingw package for it; I have no idea
> whether the msys2 maintainers would be interested in it.

Do you have a PKGBUILD? Send me a copy and I'll look at it.

> Are you interested in helping with something along these lines? I have
> only so much time to give to Emacs and it will take me sometime to get
> any of this done. If so, we can write up a slightly more detailed plan.

I can help with the MSYS2/MinGW-w64 part asking questions, dealing with
the MSYS2 maintainers and other short-term tasks. It would be
disingenuous from me to offer my assistance for implementing those
elaborated plans you have, but I will take a look at wathever you send
to me or publish here.

Re: [feature/native-comp] breakage on build

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Sun, 07 Feb 2021 20:38:00 +0100
> 
> gmp is a dependency of gcc-libs, and gcc-libs is a dependency of almost
> every other package that contains binary executable code.

GMP is now (since v27.1) a direct dependency of Emacs, since we
support bignums.

Re: [feature/native-comp] breakage on build

[Stefan Monnier]

>> gmp is a dependency of gcc-libs, and gcc-libs is a dependency of almost
>> every other package that contains binary executable code.
>
> GMP is now (since v27.1) a direct dependency of Emacs, since we
> support bignums.

But it's optional.  So from what Phil says it seems that his Emacs is
built without using libgmp (falling back on the mini-gmp that's bundled
with Emacs instead), which is probably suboptimal in the case where do
have a hard dependency on something that already require gmp (like
libgnutls).


        Stefan

Re: [feature/native-comp] breakage on build

[Óscar Fuentes]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> gmp is a dependency of gcc-libs, and gcc-libs is a dependency of almost
>>> every other package that contains binary executable code.
>>
>> GMP is now (since v27.1) a direct dependency of Emacs, since we
>> support bignums.
>
> But it's optional.  So from what Phil says it seems that his Emacs is
> built without using libgmp (falling back on the mini-gmp that's bundled
> with Emacs instead), which is probably suboptimal in the case where do
> have a hard dependency on something that already require gmp (like
> libgnutls).

If Emacs' configure script detects and uses gmp by default, Phillip's
build is being built with libgmp support, because it is certainly
present on his MSYS2 install.

(Also to note, MSYS2 does not make separate packages libfoo and
libfoo-dev.)

Re: [feature/native-comp] breakage on build

[Phillip Lord]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> gmp is a dependency of gcc-libs, and gcc-libs is a dependency of almost
>>> every other package that contains binary executable code.
>>
>> GMP is now (since v27.1) a direct dependency of Emacs, since we
>> support bignums.
>
> But it's optional.  So from what Phil says it seems that his Emacs is
> built without using libgmp (falling back on the mini-gmp that's bundled
> with Emacs instead), which is probably suboptimal in the case where do
> have a hard dependency on something that already require gmp (like
> libgnutls).

No, it's built with it. I am picking the dependency up
transitively. It's probably going to work in perpetuity, but it's
clearly bad and it should be named as a dependency. I will fix.

Phil

Re: [feature/native-comp] breakage on build

[Phillip Lord]

Óscar Fuentes <ofv@wanadoo.es> writes:

> Phillip Lord <phillip.lord@russet.org.uk> writes:
>
>>> Maybe your dependency list would benefit from some review. Just show it.
>>
>> I notice that it's missing gmp which I think it should have although
>> it's gets it transitively. On native-comp this also needs:
>
> gmp is a dependency of gcc-libs, and gcc-libs is a dependency of almost
> every other package that contains binary executable code.
>
>> mingw-w64-x86_64-libgccjit
>
> That depends on gcc, and MSYS2 gcc includes C++ plus quite a few package
> dependencies. No wonder that the final result uses so much disk space.

Yes, just so. In fact it was a problem with my earlier dependency
schemes. It's fairly hard to avoid pulling in GCC. I have just been
ignoring it.



>> I was thinking of doing it the other way up. We install Emacs as now
>> (either unzipping or using the installer). Then, Emacs runs, detects the
>> first run and says "do you want to install msys2 also". If it does, then
>> away we go, otherwise, we leave as is -- an editor rather than a pilot
>> of multiple tools.
>
> One caveat with this approach is that you can end (you *will* end) with
> two copies of the same dependencies on two places, for the dependencies
> that you put on Emacs' zip distribution.


Well, that depends. If I repackage emacs to look like

emacs-27/msys2/mingw64/bin/runemacs.exe (and all the rest)

then unpacking msys2 over the top of this using their self-extracting
file would mean that Emacs ends up exactly in place. But, yes,
alternatively, there would be two copies.
   

>> My original though was to bundle this into the install, but having done
>> a little bit of picking, I think a better solution would be to have the
>> first run of Emacs install a package from ELPA, which would actually do
>> the msys install.
>
> This is easier said than done :-)

Why? I think the code for an unattended install is actually in the
.github actions for msys2.

An alternative would be to just stick a bash script onto the FTP site,
and tell people "install msys2 then run this". This is not such a bad
option.


>> And some critical things, like git which is not available as a mingw64
>> package; I have knocked together an mingw package for it; I have no idea
>> whether the msys2 maintainers would be interested in it.
>
> Do you have a PKGBUILD? Send me a copy and I'll look at it.

https://github.com/phillord/MINGW-packages/tree/feature/mingw64-git/mingw-w64-git



>> Are you interested in helping with something along these lines? I have
>> only so much time to give to Emacs and it will take me sometime to get
>> any of this done. If so, we can write up a slightly more detailed plan.
>
> I can help with the MSYS2/MinGW-w64 part asking questions, dealing with
> the MSYS2 maintainers and other short-term tasks. It would be
> disingenuous from me to offer my assistance for implementing those
> elaborated plans you have, but I will take a look at wathever you send
> to me or publish here.


Thank you.

Phil

Re: [feature/native-comp] breakage on build

[Óscar Fuentes]

Phillip Lord <phillip.lord@russet.org.uk> writes:

>>> My original though was to bundle this into the install, but having done
>>> a little bit of picking, I think a better solution would be to have the
>>> first run of Emacs install a package from ELPA, which would actually do
>>> the msys install.
>>
>> This is easier said than done :-)
>
> Why? I think the code for an unattended install is actually in the
> .github actions for msys2.

The hard part is not to install MSYS2, is to manage the install action
from Emacs. Ask the user, remember his decision, provide a mechanism in
case he changes his mind, detect and use existing installs, decide where
to install... and more.

> An alternative would be to just stick a bash script onto the FTP site,
> and tell people "install msys2 then run this". This is not such a bad
> option.

That's along the lines of what I described on a previous message,
although I went to the extreme of suggesting to provide Emacs as a
pacman package.

Do you know that a pacman package contains a functional install? That
means that once you have the MSYS2 binary package, by simply
uncompressing it, adding the required dlls to mingw64/bin and zipping
the contents of the mingw64 directory, you have your .zip distribution.

>>> And some critical things, like git which is not available as a mingw64
>>> package; I have knocked together an mingw package for it; I have no idea
>>> whether the msys2 maintainers would be interested in it.
>>
>> Do you have a PKGBUILD? Send me a copy and I'll look at it.
>
> https://github.com/phillord/MINGW-packages/tree/feature/mingw64-git/mingw-w64-git

That lacks SSL/SSH and Perl dependencies for the commands that still
depend on it.

After looking at

https://github.com/msys2/MSYS2-packages/blob/master/git/PKGBUILD

I was expecting a more complex procedure.

If something as simple as this works, great. Otherwise, another
possibility is to install MSYS2 git (the POSIX package) and tell the
users to set vc-git-program / magit-git-executable, suppossing that it
works. It is slower than the native port, though.

Re: [feature/native-comp] breakage on build

[Phillip Lord]

Óscar Fuentes <ofv@wanadoo.es> writes:

> Phillip Lord <phillip.lord@russet.org.uk> writes:
>
>>>> My original though was to bundle this into the install, but having done
>>>> a little bit of picking, I think a better solution would be to have the
>>>> first run of Emacs install a package from ELPA, which would actually do
>>>> the msys install.
>>>
>>> This is easier said than done :-)
>>
>> Why? I think the code for an unattended install is actually in the
>> .github actions for msys2.
>
> The hard part is not to install MSYS2, is to manage the install action
> from Emacs. Ask the user, remember his decision, provide a mechanism in
> case he changes his mind, detect and use existing installs, decide where
> to install... and more.


This is a little tricky, mostly because msys2 does some stuff when it
installs -- there is the "update and force close the terminal".

I was thinking of something along the lines of:

 - Do you want to use msys2 to provide additional tools
   - Do you want Emacs to install for you, or do you have an
      existing...
      - if install: where, or internal?
      - if existing: where is it
   - Do you want to install some standard packages that Emacs uses



>> An alternative would be to just stick a bash script onto the FTP site,
>> and tell people "install msys2 then run this". This is not such a bad
>> option.
>
> That's along the lines of what I described on a previous message,
> although I went to the extreme of suggesting to provide Emacs as a
> pacman package.

Yes. In this case, we'd have two ways of getting an Emacs. One which is
the current version -- a compiled Emacs with a few extra DLLS, and one
inside msys2 as a normal pacman package. It would mean that the two
packages would be built separately.


> Do you know that a pacman package contains a functional install? That
> means that once you have the MSYS2 binary package, by simply
> uncompressing it, adding the required dlls to mingw64/bin and zipping
> the contents of the mingw64 directory, you have your .zip distribution.

I hadn't thought of going that route, but it's an interesting idea.


>>>> And some critical things, like git which is not available as a mingw64
>>>> package; I have knocked together an mingw package for it; I have no idea
>>>> whether the msys2 maintainers would be interested in it.
>>>
>>> Do you have a PKGBUILD? Send me a copy and I'll look at it.
>>
>> https://github.com/phillord/MINGW-packages/tree/feature/mingw64-git/mingw-w64-git
>
> That lacks SSL/SSH and Perl dependencies for the commands that still
> depend on it.
>
> After looking at
>
> https://github.com/msys2/MSYS2-packages/blob/master/git/PKGBUILD
>
> I was expecting a more complex procedure.

Me too. The msys version is patched, I notice.

I also don't know whether there is a reason that msys2 doesn't already
have a mingw64 package for git. Maybe just because it doesn't, or
because people use gitforwindows, or because the simple approach taken
in my branch is hopelessly broken.


> If something as simple as this works, great. Otherwise, another
> possibility is to install MSYS2 git (the POSIX package) and tell the
> users to set vc-git-program / magit-git-executable, suppossing that it
> works. It is slower than the native port, though.

Yes. I think Eli is worried about the "supposing it works" bit.

Phil

Re: [feature/native-comp] breakage on build

[Eli Zaretskii]

> From: Phillip Lord <phillip.lord@russet.org.uk>
> Date: Fri, 05 Feb 2021 21:29:53 +0000
> Cc: emacs-devel@gnu.org
> 
> Second, is native-comp. It appears that, as well as the DLLs, libgccjit
> also needs some tools locally (as, ld at least). I don't know what those
> all are yet, and that is going to need some working out.

I think libgccjit assumes that the user has a fully-functional
installation of GCC, Binutils, and runtime libraries in the "usual
places".  The library names you saw in the error message you posted
earlier (the names which begin with "-l", like -lmingw32 and -lgcc_s,
and object files like crtbegin.o), are the set of the runtime
libraries and object files needed for native-compilation.

As for their places, GCC on Windows generally looks for its files
relatively to the parent directory of the directory where its
executable gcc.exe lives.  Which means you will have to have
directories like lib/, libexec/, and share/ under the same parent
directory where you have bin/libgccjit-0.dll, and you will need to
populate those by the programs and libraries GCC uses when compiling.
For example, I assume you will have to have
libexec/gcc/mingw32/10.0.0/cc1.exe (assuming your GCC version is
10.0.0).  Is this the kind of tree you have there?

In any case, an important command to see if you have a required file
in place is "gcc -print-file-name=LIBRARY".  If this shows a full
absolute file name, you are set with regards to that LIBRARY; if it
just echoes LIBRARY as you typed it, it means GCC cannot find it.

Re: [feature/native-comp] breakage on build

[Andrea Corallo via Emacs development discussions.]

Phillip Lord <phillip.lord@russet.org.uk> writes:

> akrl--- via "Emacs development discussions." <emacs-devel@gnu.org>
> writes:
>
>> Phillip Lord <phillip.lord@russet.org.uk> writes:
>>> At the moment, .eln-cache is going to just pile up in size, no, even for
>>> normal users over versions. So, I guess the version number is useful to
>>> know which is oldest.
>>
>> I pushed two commits to diet down the hashes and remove the triplet from
>> eln filenames, now looking like the following:
>>
>> ~/.emacs.d/eln-cache/28.0.50-a45dccd2/advice-791b3e89-08434428.eln
>>
>> Thinking about I agree having the version there is still helpful so I
>> letf it for now, anyway I've no strong feeling against removing it if
>> Eli prefers.
>
>
> I've now got native-comp building successfully to a package.

nice!

> At the moment, the compilation process only runs successfully when run
> from an mingw64 shell, so I haven't got the dependencies working
> correctly yet.
>
> The error I am getting is this:
>
> Compiling c:/msys64/home/Administrator/emacs-upload/emacs-28.0.50-snapshot-feature_native-comp-windows-2021-02-04/share/emacs/28.0.50/lisp/emacs-lisp/cl-lib.el...
> compiling to c:/Users/Administrator/AppData/Roaming/.emacs.d/eln-cache/28.0.50-14889799/cl-lib-8b938900-d1df28b7.eln
> x86_64-w64-mingw32-gcc-10.2.0: fatal error: cannot execute 'as': CreateProcess: No such file or directory
> compilation terminated.

Mmmh ATM I don't know why the internal GCC driver is not finding as,
might be for one of these env vars [1]?

>
> I've tried putting as.exe manually into the package but that isn't
> enough. as.exe doesn't seem to have any dll dependencies itself.
>
> Andrea what other tools does native-comp require to have around? Is
> there a good way to debug the compilation process, so I can work out
> what the file missing is?

I'd say linker and assembler.

I think if we don't solve with the mentioned env vars or we are unappy
with the solution these are really questions for jit@gcc.gnu.org.

Thanks

  Andrea

[1] <https://gcc.gnu.org/onlinedocs/jit/internals/index.html#environment-variables>

Re: [feature/native-comp] breakage on build

[Phillip Lord]

Andrea Corallo <akrl@sdf.org> writes:

> Phillip Lord <phillip.lord@russet.org.uk> writes:
>
>
>>
>> Compiling c:/msys64/home/Administrator/emacs-upload/emacs-28.0.50-snapshot-feature_native-comp-windows-2021-02-04/share/emacs/28.0.50/lisp/emacs-lisp/cl-lib.el...
>> compiling to c:/Users/Administrator/AppData/Roaming/.emacs.d/eln-cache/28.0.50-14889799/cl-lib-8b938900-d1df28b7.eln
>> x86_64-w64-mingw32-gcc-10.2.0: fatal error: cannot execute 'as': CreateProcess: No such file or directory
>> compilation terminated.
>
> Mmmh ATM I don't know why the internal GCC driver is not finding as,
> might be for one of these env vars [1]?
>
>>
>> I've tried putting as.exe manually into the package but that isn't
>> enough. as.exe doesn't seem to have any dll dependencies itself.
>>
>> Andrea what other tools does native-comp require to have around? Is
>> there a good way to debug the compilation process, so I can work out
>> what the file missing is?
>
> I'd say linker and assembler.
>
> I think if we don't solve with the mentioned env vars or we are unappy
> with the solution these are really questions for jit@gcc.gnu.org.


I've tried adding ld.exe and as.exe, and altering the path (at runtime
which Eli says is bad).

I am now getting this error message; it's different so that's obviously
part of the problem.

   Warning (comp): Debugger entered--Lisp error: (native-ice "c:/msys64/home/Administrator/emacs-upload/emacs-28..." "failed to compile" "c:/Users/Administrator/AppData/Roaming/.emacs.d/el..." "error invoking gcc driver") Disable showing Disable logging


I'm try following up on jit@gcc as you suggest. They may well know what
I need for a minimal functional install. Once I have that, I can work
out how to achieve it in the package. At this rate, though, I fear that
I am moving back to shipping half of msys.


Phil

Re: [feature/native-comp] breakage on build

[Andrea Corallo via Emacs development discussions.]

Phillip Lord <phillip.lord@russet.org.uk> writes:

> Andrea Corallo <akrl@sdf.org> writes:
>
>> Phillip Lord <phillip.lord@russet.org.uk> writes:
>>
>>
>>>
>>> Compiling c:/msys64/home/Administrator/emacs-upload/emacs-28.0.50-snapshot-feature_native-comp-windows-2021-02-04/share/emacs/28.0.50/lisp/emacs-lisp/cl-lib.el...
>>> compiling to c:/Users/Administrator/AppData/Roaming/.emacs.d/eln-cache/28.0.50-14889799/cl-lib-8b938900-d1df28b7.eln
>>> x86_64-w64-mingw32-gcc-10.2.0: fatal error: cannot execute 'as': CreateProcess: No such file or directory
>>> compilation terminated.
>>
>> Mmmh ATM I don't know why the internal GCC driver is not finding as,
>> might be for one of these env vars [1]?
>>
>>>
>>> I've tried putting as.exe manually into the package but that isn't
>>> enough. as.exe doesn't seem to have any dll dependencies itself.
>>>
>>> Andrea what other tools does native-comp require to have around? Is
>>> there a good way to debug the compilation process, so I can work out
>>> what the file missing is?
>>
>> I'd say linker and assembler.
>>
>> I think if we don't solve with the mentioned env vars or we are unappy
>> with the solution these are really questions for jit@gcc.gnu.org.
>
>
> I've tried adding ld.exe and as.exe, and altering the path (at runtime
> which Eli says is bad).
>
> I am now getting this error message; it's different so that's obviously
> part of the problem.
>
>    Warning (comp): Debugger entered--Lisp error: (native-ice
> "c:/msys64/home/Administrator/emacs-upload/emacs-28..." "failed to
> compile" "c:/Users/Administrator/AppData/Roaming/.emacs.d/el..."
> "error invoking gcc driver") Disable showing Disable logging
>
>
> I'm try following up on jit@gcc as you suggest. They may well know what
> I need for a minimal functional install. Once I have that, I can work
> out how to achieve it in the package. At this rate, though, I fear that
> I am moving back to shipping half of msys.

Hi Phillip,

you should see the full error in the *Async-native-compile-log* buffer.

That said I'd suggest if possible to try to package the simplest
libgccjit "hello world" [1] to investigate what's needed for that to
run, collecting errors and debugging will be quicker, same for reporting
to the libgccjit mailing list in case.

Thanks for your effort!

  Andrea

[1] <https://gcc.gnu.org/onlinedocs/jit/intro/tutorial01.html>

Re: [feature/native-comp] breakage on build

[Phillip Lord]

Andrea Corallo <akrl@sdf.org> writes:

> Phillip Lord <phillip.lord@russet.org.uk> writes:
>
>> Andrea Corallo <akrl@sdf.org> writes:
>>
>>> Phillip Lord <phillip.lord@russet.org.uk> writes:
>>>
>> I've tried adding ld.exe and as.exe, and altering the path (at runtime
>> which Eli says is bad).
>>
>> I am now getting this error message; it's different so that's obviously
>> part of the problem.
>>
>>    Warning (comp): Debugger entered--Lisp error: (native-ice
>> "c:/msys64/home/Administrator/emacs-upload/emacs-28..." "failed to
>> compile" "c:/Users/Administrator/AppData/Roaming/.emacs.d/el..."
>> "error invoking gcc driver") Disable showing Disable logging
>>
>>
>> I'm try following up on jit@gcc as you suggest. They may well know what
>> I need for a minimal functional install. Once I have that, I can work
>> out how to achieve it in the package. At this rate, though, I fear that
>> I am moving back to shipping half of msys.
>
> Hi Phillip,
>
> you should see the full error in the *Async-native-compile-log*
> buffer.


Oh, yes, looking in the wrong buffer, silly me.

It's ld/as not finding things that is causing the problem.

Phil

Re: [feature/native-comp] breakage on build

[Eli Zaretskii]

> From: Phillip Lord <phillip.lord@russet.org.uk>
> Cc: akrl--- via "Emacs development discussions." <emacs-devel@gnu.org>,  Eli
>  Zaretskii <eliz@gnu.org>
> Date: Fri, 05 Feb 2021 10:05:21 +0000
> 
> It's ld/as not finding things that is causing the problem.

What stuff do they fail to find?  Can you show the error messages?

Re: [feature/native-comp] breakage on build

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Phillip Lord <phillip.lord@russet.org.uk>
>> Cc: akrl--- via "Emacs development discussions." <emacs-devel@gnu.org>,  Eli
>>  Zaretskii <eliz@gnu.org>
>> Date: Fri, 05 Feb 2021 10:05:21 +0000
>> 
>> It's ld/as not finding things that is causing the problem.
>
> What stuff do they fail to find?  Can you show the error messages?

Pretty much everything as far as I can tell.

Having added as.exe, ld.exe to the Emacs bin directory, and having
updated PATH and exec-path to include this directory (from inside Emacs
-- whether this is the long terms solution at the moment, I do not
care yet), I get this error.



   Compiling c:/msys64/home/Administrator/emacs-upload/emacs-28.0.50-snapshot-feature_native-comp-windows-2021-02-04/share/emacs/28.0.50/lisp/emacs-lisp/cl-lib.el...
compiling to c:/Users/Administrator/AppData/Roaming/.emacs.d/eln-cache/28.0.50-14889799/cl-lib-8b938900-d1df28b7.eln
x86_64-w64-mingw32-gcc-10.2.0: fatal error: cannot execute 'as': CreateProcess: No such file or directory
compilation terminated.
ld: cannot find dllcrt2.o: No such file or directory
ld: cannot find crtbegin.o: No such file or directory
ld: cannot find -lmingw32
ld: cannot find -lgcc_s
ld: cannot find -lgcc
ld: cannot find -lmoldname
ld: cannot find -lmingwex
ld: cannot find -lmsvcrt
ld: cannot find -lkernel32
ld: cannot find -lpthread
ld: cannot find -ladvapi32
ld: cannot find -lshell32
ld: cannot find -luser32
ld: cannot find -lkernel32
ld: cannot find -lmingw32
ld: cannot find -lgcc_s
ld: cannot find -lgcc
ld: cannot find -lmoldname
ld: cannot find -lmingwex
ld: cannot find -lmsvcrt
ld: cannot find -lkernel32
ld: cannot find crtend.o: No such file or directory
Compiling c:/msys64/home/Administrator/emacs-upload/emacs-28.0.50-snapshot-feature_native-comp-windows-2021-02-04/share/emacs/28.0.50/lisp/emacs-lisp/cconv.el...
compiling to c:/Users/Administrator/AppData/Roaming/.emacs.d/eln-cache/28.0.50-14889799/cconv-3b1f1f98-c52b23db.eln
libgccjit.so: error: error invoking gcc driver
Debugger entered--Lisp error: (native-ice "c:/msys64/home/Administrator/emacs-upload/emacs-28..." "failed to compile" "c:/Users/Administrator/AppData/Roaming/.emacs.d/el..." "error invoking gcc driver")
  signal(native-ice ("c:/msys64/home/Administrator/emacs-upload/emacs-28..." "failed to compile" "c:/Users/Administrator/AppData/Roaming/.emacs.d/el..." "error invoking gcc driver"))
  comp--native-compile("c:/msys64/home/Administrator/emacs-upload/emacs-28..." t)
  load-with-code-conversion("c:/Users/ADMINI~1/AppData/Local/Temp/2/emacs-async..." "c:/Users/ADMINI~1/AppData/Local/Temp/2/emacs-async..." nil t)
  command-line-1(("-l" "c:/Users/ADMINI~1/AppData/Local/Temp/2/emacs-async..."))
  command-line()
  normal-top-level()



I have checked the library paths as well as I know how (gcc
--print-search-dirs). I have reformatted the output with some new lines
to make it readable.


** Search paths (broken)
   gcc --print-search-dirs
   install: c:/Users/Administrator/Desktop/emacs-28.0.50-snapshot-feature_native-comp-windows-2021-02-04/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/
   programs: =c:/Users/Administrator/Desktop/emacs-28.0.50-snapshot-feature_native-comp-windows-2021-02-04/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/;c:/Users/Administrator/Desktop/emacs-28.0.50-snapshot-feature_native-comp-windows-2021-02-04/bin/../lib/gcc/;c:/Users/Administrator/Desktop/emacs-28.0.50-snapshot-feature_native-comp-windows-2021-02-04/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/x86_64-w64-mingw32/10.2.0/;c:/Users/Administrator/Desktop/emacs-28.0.50-snapshot-feature_native-comp-windows-2021-02-04/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/

   libraries:
   =c:/Users/Administrator/Desktop/emacs-28.0.50-snapshot-feature_native-comp-windows-2021-02-04/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/;
   c:/Users/Administrator/Desktop/emacs-28.0.50-snapshot-feature_native-comp-windows-2021-02-04/bin/../lib/gcc/;
   c:/Users/Administrator/Desktop/emacs-28.0.50-snapshot-feature_native-comp-windows-2021-02-04/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/lib/x86_64-w64-mingw32/10.2.0/;
   c:/Users/Administrator/Desktop/emacs-28.0.50-snapshot-feature_native-comp-windows-2021-02-04/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/lib/../lib/;
   c:/Users/Administrator/Desktop/emacs-28.0.50-snapshot-feature_native-comp-windows-2021-02-04/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../x86_64-w64-mingw32/10.2.0/;
   c:/Users/Administrator/Desktop/emacs-28.0.50-snapshot-feature_native-comp-windows-2021-02-04/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../lib/;
   D:/a/_temp/msys/msys64/mingw64/x86_64-w64-mingw32/lib/x86_64-w64-mingw32/10.2.0/;
   D:/a/_temp/msys/msys64/mingw64/x86_64-w64-mingw32/lib/../lib/;
   c:/Users/Administrator/Desktop/emacs-28.0.50-snapshot-feature_native-comp-windows-2021-02-04/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/lib/;
   c:/Users/Administrator/Desktop/emacs-28.0.50-snapshot-feature_native-comp-windows-2021-02-04/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../;
   D:/a/_temp/msys/msys64/mingw64/x86_64-w64-mingw32/lib/




The equivalent version for an Emacs run from mingw64 shell (which works
and JIT compiles), I get this.



** Search paths (working)

   gcc --print-search-dirs
install: C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/
programs: =C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/;C:/msys64/mingw64/bin/../lib/gcc/;C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/x86_64-w64-mingw32/10.2.0/;C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/

   libraries:
   =C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/;
   C:/msys64/mingw64/bin/../lib/gcc/;
   C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/lib/x86_64-w64-mingw32/10.2.0/;
   C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/lib/../lib/;
   C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../x86_64-w64-mingw32/10.2.0/;
   C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../lib/;
   D:/a/_temp/msys/msys64/mingw64/x86_64-w64-mingw32/lib/x86_64-w64-mingw32/10.2.0/;D:/a/_temp/msys/msys64/mingw64/x86_64-w64-mingw32/lib/../lib/;
   C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/lib/;
   C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../;D:/a/_temp/msys/msys64/mingw64/x86_64-w64-mingw32/lib/





On a whim, I tried copying the contents on mingw64/lib/gcc to inside the
Emacs install. That doesn't seem to work.

Phil

Re: [feature/native-comp] breakage on build

[Eli Zaretskii]

> From: Phillip Lord <phillip.lord@russet.org.uk>
> Cc: akrl@sdf.org,  emacs-devel@gnu.org
> Date: Fri, 05 Feb 2021 20:37:31 +0000
> 
> I have checked the library paths as well as I know how (gcc
> --print-search-dirs). I have reformatted the output with some new lines
> to make it readable.

I'm not sure I understand: do these directories really exist, and do
they hold the libraries gcc is looking for?  I mean these directories:

> ** Search paths (broken)
>    gcc --print-search-dirs
>    install: c:/Users/Administrator/Desktop/emacs-28.0.50-snapshot-feature_native-comp-windows-2021-02-04/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/
>    programs: =c:/Users/Administrator/Desktop/emacs-28.0.50-snapshot-feature_native-comp-windows-2021-02-04/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/;c:/Users/Administrator/Desktop/emacs-28.0.50-snapshot-feature_native-comp-windows-2021-02-04/bin/../lib/gcc/;c:/Users/Administrator/Desktop/emacs-28.0.50-snapshot-feature_native-comp-windows-2021-02-04/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/x86_64-w64-mingw32/10.2.0/;c:/Users/Administrator/Desktop/emacs-28.0.50-snapshot-feature_native-comp-windows-2021-02-04/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/
> 
>    libraries:
>    =c:/Users/Administrator/Desktop/emacs-28.0.50-snapshot-feature_native-comp-windows-2021-02-04/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/;
>    c:/Users/Administrator/Desktop/emacs-28.0.50-snapshot-feature_native-comp-windows-2021-02-04/bin/../lib/gcc/;
>    c:/Users/Administrator/Desktop/emacs-28.0.50-snapshot-feature_native-comp-windows-2021-02-04/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/lib/x86_64-w64-mingw32/10.2.0/;
>    c:/Users/Administrator/Desktop/emacs-28.0.50-snapshot-feature_native-comp-windows-2021-02-04/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/lib/../lib/;
>    c:/Users/Administrator/Desktop/emacs-28.0.50-snapshot-feature_native-comp-windows-2021-02-04/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../x86_64-w64-mingw32/10.2.0/;
>    c:/Users/Administrator/Desktop/emacs-28.0.50-snapshot-feature_native-comp-windows-2021-02-04/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../lib/;
>    D:/a/_temp/msys/msys64/mingw64/x86_64-w64-mingw32/lib/x86_64-w64-mingw32/10.2.0/;
>    D:/a/_temp/msys/msys64/mingw64/x86_64-w64-mingw32/lib/../lib/;
>    c:/Users/Administrator/Desktop/emacs-28.0.50-snapshot-feature_native-comp-windows-2021-02-04/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/lib/;
>    c:/Users/Administrator/Desktop/emacs-28.0.50-snapshot-feature_native-comp-windows-2021-02-04/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../;
>    D:/a/_temp/msys/msys64/mingw64/x86_64-w64-mingw32/lib/

If they do exist, does it mean you duplicated the entire MinGW GCC
installation under the same directory that is the parent of the
Emacs's bin/ directory?

What does the following command produce?

  gcc -print-file-name=crtbegin.o

> The equivalent version for an Emacs run from mingw64 shell (which works
> and JIT compiles), I get this.
> 
> 
> 
> ** Search paths (working)
> 
>    gcc --print-search-dirs
> install: C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/
> programs: =C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/;C:/msys64/mingw64/bin/../lib/gcc/;C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw324/10.2.0/../../../../x86_64-w64-mingw32/bin/x86_64-w64-mingw32/10.2.0/;C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/
> 
>    libraries:
>    =C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/;
>    C:/msys64/mingw64/bin/../lib/gcc/;
>    C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/lib/x86_64-w64-mingw32/10.2.0/;
>    C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/lib/../lib/;
>    C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../x86_64-w64-mingw32/10.2.0/;
>    C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../lib/;
>    D:/a/_temp/msys/msys64/mingw64/x86_64-w64-mingw32/lib/x86_64-w64-mingw32/10.2.0/;D:/a/_temp/msys/msys64/mingw64/x86_64-w64-mingw32/lib/../lib/;
>    C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/lib/;
>    C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../;D:/a/_temp/msys/msys64/mingw64/x86_64-w64-mingw32/lib/

Does the mingw64 shell set some GCC-related environment variables,
like COMPILER_PATH or LIBRARY_PATH?

Alternatively, could it be that under the mingw64 shell Emacs invokes
a different copy of gcc/libgccjit, the one that is in its "normal"
installation tree and not in the Emacs tree?

Re: [feature/native-comp] breakage on build

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Phillip Lord <phillip.lord@russet.org.uk>
>> Cc: akrl@sdf.org,  emacs-devel@gnu.org
>> Date: Fri, 05 Feb 2021 20:37:31 +0000
>> 
>> I have checked the library paths as well as I know how (gcc
>> --print-search-dirs). I have reformatted the output with some new lines
>> to make it readable.
>
> I'm not sure I understand: do these directories really exist, and do
> they hold the libraries gcc is looking for?  I mean these directories:

No. The lib/gcc didn't exist at first. If they did hold the libraries
GCC was looking for, I presume it would work. I have no idea where all
that strangeness with relative paths which look to me like they resolve
to the same places repeatedly comes from.


>
>> ** Search paths (broken)
>>    gcc --print-search-dirs
>>    install:

>
> If they do exist, does it mean you duplicated the entire MinGW GCC
> installation under the same directory that is the parent of the
> Emacs's bin/ directory?

I have tried copying /msys64/mingw64/lib/gcc into the Emacs installation
at /lib, yes. That adds 140 Mb to the install, but achieves nothing at
the moment.


>
> What does the following command produce?
>
>   gcc -print-file-name=crtbegin.o


gcc -print-file-name=crtbegin.o
c:/Users/Administrator/Desktop/emacs-28.0.50-snapshot-feature_native-comp-windows-2021-02-04/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/crtbegin.o

This file does exist.


>
>> The equivalent version for an Emacs run from mingw64 shell (which works
>> and JIT compiles), I get this.
>> 
>> 
>> 
>> ** Search paths (working)
>> 
>>    gcc --print-search-dirs
>> install: C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/
>> programs:
>> =C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/;C:/msys64/mingw64/bin/../lib/gcc/;C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw324/10.2.0/../../../../x86_64-w64-mingw32/bin/x86_64-w64-mingw32/10.2.0/;C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/
>> 
>>    libraries:
>>    =C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/;
>>    C:/msys64/mingw64/bin/../lib/gcc/;
>>    C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/lib/x86_64-w64-mingw32/10.2.0/;
>>    C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/lib/../lib/;
>>    C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../x86_64-w64-mingw32/10.2.0/;
>>    C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../lib/;
>>    D:/a/_temp/msys/msys64/mingw64/x86_64-w64-mingw32/lib/x86_64-w64-mingw32/10.2.0/;D:/a/_temp/msys/msys64/mingw64/x86_64-w64-mingw32/lib/../lib/;
>>    C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/lib/;
>>    C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../;D:/a/_temp/msys/msys64/mingw64/x86_64-w64-mingw32/lib/
>
> Does the mingw64 shell set some GCC-related environment variables,
> like COMPILER_PATH or LIBRARY_PATH?


No.


> Alternatively, could it be that under the mingw64 shell Emacs invokes
> a different copy of gcc/libgccjit, the one that is in its "normal"
> installation tree and not in the Emacs tree?


Yes. I do not know how much or how little of the mingw64 install that
Emacs is using; it could be a missing DLL, config file anything at
all. But, the -no-deps version of Emacs happily does native-comp when
launched from inside a mingw64 shell.

I believe that the errors we are getting from Emacs launched from file
explorer indicate that it is finding the packaged libgccjit; I presumed
that it is this that is called as.exe and ld.exe. But Andrea can
probably answer that authoratitive.

Phil

Re: [feature/native-comp] breakage on build

[Eli Zaretskii]

> From: Phillip Lord <phillip.lord@russet.org.uk>
> Cc: akrl@sdf.org,  emacs-devel@gnu.org
> Date: Fri, 05 Feb 2021 21:50:24 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: Phillip Lord <phillip.lord@russet.org.uk>
> >> Cc: akrl@sdf.org,  emacs-devel@gnu.org
> >> Date: Fri, 05 Feb 2021 20:37:31 +0000
> >> 
> >> I have checked the library paths as well as I know how (gcc
> >> --print-search-dirs). I have reformatted the output with some new lines
> >> to make it readable.
> >
> > I'm not sure I understand: do these directories really exist, and do
> > they hold the libraries gcc is looking for?  I mean these directories:
> 
> No. The lib/gcc didn't exist at first. If they did hold the libraries
> GCC was looking for, I presume it would work. I have no idea where all
> that strangeness with relative paths which look to me like they resolve
> to the same places repeatedly comes from.

I believe this is just the standard GCC search path on MS-Windows.

> > If they do exist, does it mean you duplicated the entire MinGW GCC
> > installation under the same directory that is the parent of the
> > Emacs's bin/ directory?
> 
> I have tried copying /msys64/mingw64/lib/gcc into the Emacs installation
> at /lib, yes.

/msys64/mingw64/lib/gcc is not enough, you also need some of the
/msys64/mingw64/libexec/gcc (assuming MinGW64 has it -- I don't use
MinGW64, so I cannot check).

> > What does the following command produce?
> >
> >   gcc -print-file-name=crtbegin.o
> 
> gcc -print-file-name=crtbegin.o
> c:/Users/Administrator/Desktop/emacs-28.0.50-snapshot-feature_native-comp-windows-2021-02-04/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/crtbegin.o
> 
> This file does exist.

Then I admit I don't understand this part of the error messages:

> compiling to c:/Users/Administrator/AppData/Roaming/.emacs.d/eln-cache/28.0.50-14889799/cl-lib-8b938900-d1df28b7.eln
> x86_64-w64-mingw32-gcc-10.2.0: fatal error: cannot execute 'as': CreateProcess: No such file or directory
> compilation terminated.
> ld: cannot find dllcrt2.o: No such file or directory
> ld: cannot find crtbegin.o: No such file or directory
> ld: cannot find -lmingw32
> ld: cannot find -lgcc_s
> ld: cannot find -lgcc
> ld: cannot find -lmoldname
> ld: cannot find -lmingwex
> ld: cannot find -lmsvcrt
> ld: cannot find -lkernel32
> ld: cannot find -lpthread
> ld: cannot find -ladvapi32
> ld: cannot find -lshell32
> ld: cannot find -luser32
> ld: cannot find -lkernel32
> ld: cannot find -lmingw32
> ld: cannot find -lgcc_s
> ld: cannot find -lgcc
> ld: cannot find -lmoldname
> ld: cannot find -lmingwex
> ld: cannot find -lmsvcrt
> ld: cannot find -lkernel32
> ld: cannot find crtend.o: No such file or directory

GCC should have instructed ld.exe where to look for the libraries
mentioned in this error message.  If crtbegin.o exists at the location
where GCC expects it to be (and I presume the rest of the libraries
mentioned in the above message also exist there -- do they?), then why
didn't ld.exe find them?

Is there a way to invoke JIT compiler so that the -v switch is passed
to ld.exe as well?  On the command line, "gcc -v" accomplishes that.
This will show you which directories ld.exe searches for its
libraries.

Also note that in that invocation GCC didn't find as.exe, so it could
be that these error messages are triggered by that and not by actually
missing the libraries.

> > Alternatively, could it be that under the mingw64 shell Emacs invokes
> > a different copy of gcc/libgccjit, the one that is in its "normal"
> > installation tree and not in the Emacs tree?
> 
> Yes. I do not know how much or how little of the mingw64 install that
> Emacs is using; it could be a missing DLL, config file anything at
> all. But, the -no-deps version of Emacs happily does native-comp when
> launched from inside a mingw64 shell.

Then maybe you should start by duplicating the full installation of
GCC, Binutils, and MinGW runtime under the same directory where you
have bin/emacs.exe.  Once you install that, see if native-comp works
outside of the mingw64 shell.  If it does, then you can start removing
files and directories that are not needed, which will hopefully leave
you with the minimum subset of the mingw64 installation required for
native-comp.

Re: [feature/native-comp] breakage on build

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:

>> > Alternatively, could it be that under the mingw64 shell Emacs invokes
>> > a different copy of gcc/libgccjit, the one that is in its "normal"
>> > installation tree and not in the Emacs tree?
>> 
>> Yes. I do not know how much or how little of the mingw64 install that
>> Emacs is using; it could be a missing DLL, config file anything at
>> all. But, the -no-deps version of Emacs happily does native-comp when
>> launched from inside a mingw64 shell.
>
> Then maybe you should start by duplicating the full installation of
> GCC, Binutils, and MinGW runtime under the same directory where you
> have bin/emacs.exe.  Once you install that, see if native-comp works
> outside of the mingw64 shell.  If it does, then you can start removing
> files and directories that are not needed, which will hopefully leave
> you with the minimum subset of the mingw64 installation required for
> native-comp.


So, I did this:

 - Installed a fresh msys to a clean directory on my desktop.
 - Dropped the emacs-no-deps.zip directory on top of the mingw64 directory.
 - clicked on runemacs.exe from windows shell -- Emacs fails to launch
   because of libgmp
 - pacman installed all the dependencies of Emacs
 - clicked on runemacs.exe -- Emacs launches, native-comp fails as
   described
 - run mingw64 shell form the mys installation
 - ./runemacs.exe in that shell
 - Emacs launches, native-comp works
 - ~/ and thus ~/.emacs.d/eln-cache is a different location with the two
   launch methods

I believe that msys is mostly portable and a new installation is nearly
clean (except for .bashrc and such like).

Conclusions:

 - emacs is picking up dependencies from mingw64 run in either way,
   because otherwise it wouldn't run at all from the shell
 - Emacs is picking up the environment from mingw64 when launched from
   within it, hence the differences in location of ~
 - Native-comp has everything it needs to run, or it couldn't run at
   all.
 - It is missing something from the environment when run in the windows
   shell.

This is export run inside the mingw64 window. So, lots of options there.

Phil


$ export
declare -x ACLOCAL_PATH="/mingw64/share/aclocal:/usr/share/aclocal"
declare -x ALLUSERSPROFILE="C:\\ProgramData"
declare -x APPDATA="C:\\Users\\IEUser\\AppData\\Roaming"
declare -x COMMONPROGRAMFILES="C:\\Program Files\\Common Files"
declare -x COMPUTERNAME="MSEDGEWIN10"
declare -x COMSPEC="C:\\Windows\\system32\\cmd.exe"
declare -x CONFIG_SITE="/mingw64/etc/config.site"
declare -x ChocolateyInstall="C:\\ProgramData\\chocolatey"
declare -x ChocolateyLastPathUpdate="131974685787905910"
declare -x CommonProgramW6432="C:\\Program Files\\Common Files"
declare -x DriverData="C:\\Windows\\System32\\Drivers\\DriverData"
declare -x FPS_BROWSER_APP_PROFILE_STRING="Internet Explorer"
declare -x FPS_BROWSER_USER_PROFILE_STRING="Default"
declare -x HOME="/home/IEUser"
declare -x HOMEDRIVE="C:"
declare -x HOMEPATH="\\Users\\IEUser"
declare -x HOSTNAME="MSEDGEWIN10"
declare -x INFOPATH="/usr/local/info:/usr/share/info:/usr/info:/share/info"
declare -x LC_CTYPE="en_GB.UTF-8"
declare -x LOCALAPPDATA="C:\\Users\\IEUser\\AppData\\Local"
declare -x LOGONSERVER="\\\\MSEDGEWIN10"
declare -x MANPATH="/mingw64/local/man:/mingw64/share/man:/usr/local/man:/usr/share/man:/usr/man:/share/man"
declare -x MINGW_CHOST="x86_64-w64-mingw32"
declare -x MINGW_PACKAGE_PREFIX="mingw-w64-x86_64"
declare -x MINGW_PREFIX="/mingw64"
declare -x MSYSCON="mintty.exe"
declare -x MSYSTEM="MINGW64"
declare -x MSYSTEM_CARCH="x86_64"
declare -x MSYSTEM_CHOST="x86_64-w64-mingw32"
declare -x MSYSTEM_PREFIX="/mingw64"
declare -x NUMBER_OF_PROCESSORS="1"
declare -x OLDPWD="/mingw64/bin"
declare -x ORIGINAL_PATH="/c/Windows/System32:/c/Windows:/c/Windows/System32/Wbem:/c/Windows/System32/WindowsPowerShell/v1.0/"
declare -x ORIGINAL_TEMP="/c/Users/IEUser/AppData/Local/Temp"
declare -x ORIGINAL_TMP="/c/Users/IEUser/AppData/Local/Temp"
declare -x OS="Windows_NT"
declare -x OneDrive="C:\\Users\\IEUser\\OneDrive"
declare -x OneDriveConsumer="C:\\Users\\IEUser\\OneDrive"
declare -x PATH="/mingw64/bin:/usr/local/bin:/usr/bin:/bin:/c/Windows/System32:/c/Windows:/c/Windows/System32/Wbem:/c/Windows/System32/WindowsPowerShell/v1.0/:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl"
declare -x PATHEXT=".COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC"
declare -x PKG_CONFIG_PATH="/mingw64/lib/pkgconfig:/mingw64/share/pkgconfig"
declare -x PRINTER="OneNote for Windows 10"
declare -x PROCESSOR_ARCHITECTURE="AMD64"
declare -x PROCESSOR_IDENTIFIER="AMD64 Family 21 Model 56 Stepping 1, AuthenticAMD"
declare -x PROCESSOR_LEVEL="21"
declare -x PROCESSOR_REVISION="3801"
declare -x PROGRAMFILES="C:\\Program Files"
declare -x PS1="\\[\\e]0;\\w\\a\\]\\n\\[\\e[32m\\]\\u@\\h \\[\\e[35m\\]\$MSYSTEM\\[\\e[0m\\] \\[\\e[33m\\]\\w\\[\\e[0m\\]\\n\\\$ "
declare -x PSModulePath="C:\\Program Files\\WindowsPowerShell\\Modules;C:\\Windows\\system32\\WindowsPowerShell\\v1.0\\Modules"
declare -x PUBLIC="C:\\Users\\Public"
declare -x PWD="/home/IEUser"
declare -x ProgramData="C:\\ProgramData"
declare -x ProgramW6432="C:\\Program Files"
declare -x SESSIONNAME="Console"
declare -x SHELL="/usr/bin/bash"
declare -x SHLVL="1"
declare -x SYSTEMDRIVE="C:"
declare -x SYSTEMROOT="C:\\Windows"
declare -x TEMP="/tmp"
declare -x TERM="xterm"
declare -x TERM_PROGRAM="mintty"
declare -x TERM_PROGRAM_VERSION="3.4.4"
declare -x TMP="/tmp"
declare -x TZ="America/Los_Angeles"
declare -x USER="IEUser"
declare -x USERDOMAIN="MSEDGEWIN10"
declare -x USERDOMAIN_ROAMINGPROFILE="MSEDGEWIN10"
declare -x USERNAME="IEUser"
declare -x USERPROFILE="C:\\Users\\IEUser"
declare -x WINDIR="C:\\Windows"
declare -x temp="C:\\Users\\IEUser\\AppData\\Local\\Temp"
declare -x tmp="C:\\Users\\IEUser\\AppData\\Local\\Temp"

  

 

Re: [feature/native-comp] breakage on build

[Eli Zaretskii]

> From: Phillip Lord <phillip.lord@russet.org.uk>
> Cc: akrl@sdf.org,  emacs-devel@gnu.org
> Date: Sat, 06 Feb 2021 12:58:59 +0000
> 
> So, I did this:
> 
>  - Installed a fresh msys to a clean directory on my desktop.
>  - Dropped the emacs-no-deps.zip directory on top of the mingw64 directory.
>  - clicked on runemacs.exe from windows shell -- Emacs fails to launch
>    because of libgmp
>  - pacman installed all the dependencies of Emacs
>  - clicked on runemacs.exe -- Emacs launches, native-comp fails as
>    described

Fails with exactly the same error messages?  Even though now as.exe
should be on PATH and GCC should be able to find it?

>  - run mingw64 shell form the mys installation
>  - ./runemacs.exe in that shell
>  - Emacs launches, native-comp works
>  - ~/ and thus ~/.emacs.d/eln-cache is a different location with the two
>    launch methods

Please show the full environment as Emacs sees it (look in
process-environment) both when it's launched from mingw64 shell and
outside of it.

> Conclusions:
> 
>  - emacs is picking up dependencies from mingw64 run in either way,
>    because otherwise it wouldn't run at all from the shell
>  - Emacs is picking up the environment from mingw64 when launched from
>    within it, hence the differences in location of ~
>  - Native-comp has everything it needs to run, or it couldn't run at
>    all.
>  - It is missing something from the environment when run in the windows
>    shell.
> 
> This is export run inside the mingw64 window. So, lots of options there.
> [...]
> declare -x MINGW_PREFIX="/mingw64"

What is the meaning and the use of MINGW_PREFIX?

> declare -x PATH="/mingw64/bin:/usr/local/bin:/usr/bin:/bin:/c/Windows/System32:/c/Windows:/c/Windows/System32/Wbem:/c/Windows/System32/WindowsPowerShell/v1.0/:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl"

Could it be that Emacs when started from the mingw64 shell picks up
GCC components from this PATH, and thus GCC will then look for the
libraries and object files for native-comp in the "normal" mingw64
installation, not in the Emacs tree you created?  In that case the
fact that native-comp works from the mingw64 shell is not interesting,
as ll it says that your duplicate GCC installation is somehow
incomplete.

I don't see any other variable that I think could be related.  (But I
don't use MinGW64, so I could miss something.)  However, you show the
exported variables from the MSYS Bash, whereas what we need to look at
is the environment variables as seen by GCC, when it is run from
Emacs.  The value of process-environment in both cases should be a
good first step.

Re: [feature/native-comp] breakage on build

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Phillip Lord <phillip.lord@russet.org.uk>
>> Cc: akrl@sdf.org,  emacs-devel@gnu.org
>> Date: Sat, 06 Feb 2021 12:58:59 +0000
>> 
> Could it be that Emacs when started from the mingw64 shell picks up
> GCC components from this PATH, and thus GCC will then look for the
> libraries and object files for native-comp in the "normal" mingw64
> installation, not in the Emacs tree you created?  In that case the
> fact that native-comp works from the mingw64 shell is not interesting,
> as ll it says that your duplicate GCC installation is somehow
> incomplete.
>
> I don't see any other variable that I think could be related.  (But I
> don't use MinGW64, so I could miss something.)  However, you show the
> exported variables from the MSYS Bash, whereas what we need to look at
> is the environment variables as seen by GCC, when it is run from
> Emacs.  The value of process-environment in both cases should be a
> good first step.


Eli, Oscar, Andrea

Apologies, I had forgotten that I had moved machines during testing. It
was simply the path (and exec-path) that was missing. My own time is
mine to waste, but apologies for wasting yours.

I am coming to the conclusion that I am going the wrong path here. I
think supporting native-comp and keeping a sanely sized Emacs
distribution are not compatible objectives. I do not want to return to
the Emacs-26 distribution, which included a lot of msys2.

I think that the base distribution of Emacs-28 should not include
JIT native-comp. If people want this, they should get it through the
msys2 co-installation that Oscar and I have been talking of.

However, we could just do a full AOT native-comp, so at least the
Emacs-28 packages would all benefit. Andrea, can the native-comp branch
still load *eln files even if it can't produce them?

Phil

Re: [feature/native-comp] breakage on build

[Andrea Corallo via Emacs development discussions.]

Phillip Lord <phillip.lord@russet.org.uk> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> From: Phillip Lord <phillip.lord@russet.org.uk>
>>> Cc: akrl@sdf.org,  emacs-devel@gnu.org
>>> Date: Sat, 06 Feb 2021 12:58:59 +0000
>>> 
>> Could it be that Emacs when started from the mingw64 shell picks up
>> GCC components from this PATH, and thus GCC will then look for the
>> libraries and object files for native-comp in the "normal" mingw64
>> installation, not in the Emacs tree you created?  In that case the
>> fact that native-comp works from the mingw64 shell is not interesting,
>> as ll it says that your duplicate GCC installation is somehow
>> incomplete.
>>
>> I don't see any other variable that I think could be related.  (But I
>> don't use MinGW64, so I could miss something.)  However, you show the
>> exported variables from the MSYS Bash, whereas what we need to look at
>> is the environment variables as seen by GCC, when it is run from
>> Emacs.  The value of process-environment in both cases should be a
>> good first step.
>
>
> Eli, Oscar, Andrea
>
> Apologies, I had forgotten that I had moved machines during testing. It
> was simply the path (and exec-path) that was missing. My own time is
> mine to waste, but apologies for wasting yours.
>
> I am coming to the conclusion that I am going the wrong path here. I
> think supporting native-comp and keeping a sanely sized Emacs
> distribution are not compatible objectives. I do not want to return to
> the Emacs-26 distribution, which included a lot of msys2.
>
> I think that the base distribution of Emacs-28 should not include
> JIT native-comp. If people want this, they should get it through the
> msys2 co-installation that Oscar and I have been talking of.
>
> However, we could just do a full AOT native-comp, so at least the
> Emacs-28 packages would all benefit. Andrea, can the native-comp branch
> still load *eln files even if it can't produce them?

Yes, libgccjit is not needed for loading elns.  There might be some minors to
look at as I've never tested this configuration but should be no
problem.

  Andrea

Re: [feature/native-comp] breakage on build

[Phillip Lord]

Andrea Corallo <akrl@sdf.org> writes:

> Phillip Lord <phillip.lord@russet.org.uk> writes:
>> Eli, Oscar, Andrea
>>
>> Apologies, I had forgotten that I had moved machines during testing. It
>> was simply the path (and exec-path) that was missing. My own time is
>> mine to waste, but apologies for wasting yours.
>>
>> I am coming to the conclusion that I am going the wrong path here. I
>> think supporting native-comp and keeping a sanely sized Emacs
>> distribution are not compatible objectives. I do not want to return to
>> the Emacs-26 distribution, which included a lot of msys2.
>>
>> I think that the base distribution of Emacs-28 should not include
>> JIT native-comp. If people want this, they should get it through the
>> msys2 co-installation that Oscar and I have been talking of.
>>
>> However, we could just do a full AOT native-comp, so at least the
>> Emacs-28 packages would all benefit. Andrea, can the native-comp branch
>> still load *eln files even if it can't produce them?
>
> Yes, libgccjit is not needed for loading elns.  There might be some minors to
> look at as I've never tested this configuration but should be no
> problem.


For an Emacs compiled --with-nativecomp, how do I stop it attempting to
native compile?

Phil

Re: [feature/native-comp] breakage on build

[Andrea Corallo via Emacs development discussions.]

Phillip Lord <phillip.lord@russet.org.uk> writes:

> Andrea Corallo <akrl@sdf.org> writes:
>
>> Phillip Lord <phillip.lord@russet.org.uk> writes:
>>> Eli, Oscar, Andrea
>>>
>>> Apologies, I had forgotten that I had moved machines during testing. It
>>> was simply the path (and exec-path) that was missing. My own time is
>>> mine to waste, but apologies for wasting yours.
>>>
>>> I am coming to the conclusion that I am going the wrong path here. I
>>> think supporting native-comp and keeping a sanely sized Emacs
>>> distribution are not compatible objectives. I do not want to return to
>>> the Emacs-26 distribution, which included a lot of msys2.
>>>
>>> I think that the base distribution of Emacs-28 should not include
>>> JIT native-comp. If people want this, they should get it through the
>>> msys2 co-installation that Oscar and I have been talking of.
>>>
>>> However, we could just do a full AOT native-comp, so at least the
>>> Emacs-28 packages would all benefit. Andrea, can the native-comp branch
>>> still load *eln files even if it can't produce them?
>>
>> Yes, libgccjit is not needed for loading elns.  There might be some minors to
>> look at as I've never tested this configuration but should be no
>> problem.
>
>
> For an Emacs compiled --with-nativecomp, how do I stop it attempting to
> native compile?

`comp-deferred-compilation'

But I was wrong there's another issue.  If we can't compile we cannot
synthesize trampolines, as a consequence we can't redefine or advice
primitive functions reliably... Sorry

  Andrea

Re: [feature/native-comp] breakage on build

[Phillip Lord]

Andrea Corallo <akrl@sdf.org> writes:

> Phillip Lord <phillip.lord@russet.org.uk> writes:
>
>> Andrea Corallo <akrl@sdf.org> writes:
>>
>>> Phillip Lord <phillip.lord@russet.org.uk> writes:
>>>
>>> Yes, libgccjit is not needed for loading elns.  There might be some minors to
>>> look at as I've never tested this configuration but should be no
>>> problem.
>>
>>
>> For an Emacs compiled --with-nativecomp, how do I stop it attempting to
>> native compile?
>
> `comp-deferred-compilation'
>
> But I was wrong there's another issue.  If we can't compile we cannot
> synthesize trampolines, as a consequence we can't redefine or advice
> primitive functions reliably... Sorry


I didn't think we could do this anyway. I guess this means that
native-compiled lisp to primitive adviced functions will skip the
advice? So, that would be a change from the current situation where
C calls to C functions don't get adviced, but lisp calls to C functions
do.

Phil

Re: [feature/native-comp] breakage on build

[Andrea Corallo via Emacs development discussions.]

Phillip Lord <phillip.lord@russet.org.uk> writes:

> Andrea Corallo <akrl@sdf.org> writes:
>
>> Phillip Lord <phillip.lord@russet.org.uk> writes:
>>
>>> Andrea Corallo <akrl@sdf.org> writes:
>>>
>>>> Phillip Lord <phillip.lord@russet.org.uk> writes:
>>>>
>>>> Yes, libgccjit is not needed for loading elns.  There might be some minors to
>>>> look at as I've never tested this configuration but should be no
>>>> problem.
>>>
>>>
>>> For an Emacs compiled --with-nativecomp, how do I stop it attempting to
>>> native compile?
>>
>> `comp-deferred-compilation'
>>
>> But I was wrong there's another issue.  If we can't compile we cannot
>> synthesize trampolines, as a consequence we can't redefine or advice
>> primitive functions reliably... Sorry
>
>
> I didn't think we could do this anyway. I guess this means that
> native-compiled lisp to primitive adviced functions will skip the
> advice? So, that would be a change from the current situation where
> C calls to C functions don't get adviced, but lisp calls to C functions
> do.

Apologies let me be more precise.

- Advising C to C primitives is not possible in both implementations

- With bytecode you can effectively redefine primitives with _no_
  dedicated op-bytecode.

- With native code you can redefine all of these except for the (very
  few) primitives we open-code directly in the generated code.  This is
  a smaller set than the previous one but still not perfect.

  Anyway, we can do this but relying on this trampoline syntheses
  mechanism.

  Removing this mechanism is therfore ATM not possible, unless we
  precompile all trampolines AOT bug#44238.  This should be rather easy
  tho, we lack only some integration into the build system lets say.


  Andrea

Re: [feature/native-comp] breakage on build

[Stefan Monnier]

> - With native code you can redefine all of these except for the (very
>   few) primitives we open-code directly in the generated code.  This is
>   a smaller set than the previous one but still not perfect.

But if we can't build trampolines, currently, the situation is much
worse: many calls to native-compiled Lisp functions from native-compiled Lisp
functions can't be advised any more.


        Stefan

Re: [feature/native-comp] breakage on build

[Andrea Corallo via Emacs development discussions.]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> - With native code you can redefine all of these except for the (very
>>   few) primitives we open-code directly in the generated code.  This is
>>   a smaller set than the previous one but still not perfect.
>
> But if we can't build trampolines, currently, the situation is much
> worse: many calls to native-compiled Lisp functions from native-compiled Lisp
> functions can't be advised any more.

Correct, but if we can build eln files during the initial compilation we
can also build trampolines AOT.  This is just a matter of running:

(mapatoms (λ (s)    
            (when (subr-primitive-p (symbol-function s))
              (comp-trampoline-compile s))))

Probably we should wire this in the build system so it's accessible as
make target?

  Andrea

Re: [feature/native-comp] breakage on build

[Phillip Lord]

Andrea Corallo <akrl@sdf.org> writes:

> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>
>>> - With native code you can redefine all of these except for the (very
>>>   few) primitives we open-code directly in the generated code.  This is
>>>   a smaller set than the previous one but still not perfect.
>>
>> But if we can't build trampolines, currently, the situation is much
>> worse: many calls to native-compiled Lisp functions from native-compiled Lisp
>> functions can't be advised any more.
>
> Correct, but if we can build eln files during the initial compilation we
> can also build trampolines AOT.  This is just a matter of running:
>
> (mapatoms (λ (s)    
>             (when (subr-primitive-p (symbol-function s))
>               (comp-trampoline-compile s))))
>
> Probably we should wire this in the build system so it's accessible as
> make target?


Why not wire it into the AOT compilation always? Obviously, if it
doubles the AOT compilation, that might not be such a great idea, but
otherwise?

Phil

Re: [feature/native-comp] breakage on build

[Andrea Corallo via Emacs development discussions.]

Phillip Lord <phillip.lord@russet.org.uk> writes:

> Andrea Corallo <akrl@sdf.org> writes:
>
>> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>>
>>>> - With native code you can redefine all of these except for the (very
>>>>   few) primitives we open-code directly in the generated code.  This is
>>>>   a smaller set than the previous one but still not perfect.
>>>
>>> But if we can't build trampolines, currently, the situation is much
>>> worse: many calls to native-compiled Lisp functions from native-compiled Lisp
>>> functions can't be advised any more.
>>
>> Correct, but if we can build eln files during the initial compilation we
>> can also build trampolines AOT.  This is just a matter of running:
>>
>> (mapatoms (λ (s)    
>>             (when (subr-primitive-p (symbol-function s))
>>               (comp-trampoline-compile s))))
>>
>> Probably we should wire this in the build system so it's accessible as
>> make target?
>
>
> Why not wire it into the AOT compilation always? Obviously, if it
> doubles the AOT compilation, that might not be such a great idea, but
> otherwise?
>
> Phil

That's a great question.

Compiling a trampoline is about 0.2s each (mostly GCC gas ld startup
time).  Say we have ~1400 primitives this should translate into ~4.5 min
of build time for something that in the average case is largely unused.

The trouble is that the simple code snippet I posted is not parallel and
there's no trivial way to make make aware of all these targets to have
it handling the parallelism.

  Andrea

Re: [feature/native-comp] breakage on build

[Stefan Monnier]

> Compiling a trampoline is about 0.2s each (mostly GCC gas ld startup
> time).  Say we have ~1400 primitives this should translate into ~4.5 min
> of build time for something that in the average case is largely unused.

But do we really need 1400 different trampolines?
Can't these all just be copies of a small number of "base" trampolines
(presumably one needs a different trampoline for every number of
arguments, but that's about it, right)?


        Stefan

Re: [feature/native-comp] breakage on build

[Andrea Corallo via Emacs development discussions.]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Compiling a trampoline is about 0.2s each (mostly GCC gas ld startup
>> time).  Say we have ~1400 primitives this should translate into ~4.5 min
>> of build time for something that in the average case is largely unused.
>
> But do we really need 1400 different trampolines?
>
> Can't these all just be copies of a small number of "base" trampolines
> (presumably one needs a different trampoline for every number of
> arguments, but that's about it, right)?

Yes I think they could, the perf penalty should be negligible compared
to the use of an advised primitive.

The question is if is it so important to support this configuration (eln
but no libgccjit) that indeed is achievable only on Windows.

  Andrea

Re: [feature/native-comp] breakage on build

[Phillip Lord]

Andrea Corallo <akrl@sdf.org> writes:

> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>
>>> Compiling a trampoline is about 0.2s each (mostly GCC gas ld startup
>>> time).  Say we have ~1400 primitives this should translate into ~4.5 min
>>> of build time for something that in the average case is largely unused.
>>
>> But do we really need 1400 different trampolines?
>>
>> Can't these all just be copies of a small number of "base" trampolines
>> (presumably one needs a different trampoline for every number of
>> arguments, but that's about it, right)?
>
> Yes I think they could, the perf penalty should be negligible compared
> to the use of an advised primitive.
>
> The question is if is it so important to support this configuration (eln
> but no libgccjit) that indeed is achievable only on Windows.


It's a good question.

I am siding toward the idea of having a windows distribution with the
DLLs as now, and then another which is simply integrated into msys2 for
all the bells and whistles. It would be perfectly tenable to have
native-comp only available on the msys2 version.

Whether this is a good compromise I do not know, since it's not clear
how much difference native-comp will make in practice or on windows to
the feeling of speed.

Phil

Re: [feature/native-comp] breakage on build

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>,
>         Andrea Corallo via "Emacs
>  development discussions." <emacs-devel@gnu.org>,
>         Eli Zaretskii
>  <eliz@gnu.org>
> Date: Wed, 10 Feb 2021 08:44:57 +0000
> 
> Compiling a trampoline is about 0.2s each (mostly GCC gas ld startup
> time).  Say we have ~1400 primitives this should translate into ~4.5 min
> of build time for something that in the average case is largely unused.

4.5 min out of how much?  Isn't building Emacs with native-comp take a
long time anyway?

> The trouble is that the simple code snippet I posted is not parallel and
> there's no trivial way to make make aware of all these targets to have
> it handling the parallelism.

How about if we make the necessary assembly snippet part of the DEFUN
macro?  Then compiling each .c file will produce those trampolines
ready to be used, no?  (I admit that I don't really understand the
details of those trampolines, so maybe the above makes no sense.)

Re: [feature/native-comp] breakage on build

[Andrea Corallo via Emacs development discussions.]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <akrl@sdf.org>
>> Cc: Stefan Monnier <monnier@iro.umontreal.ca>,
>>         Andrea Corallo via "Emacs
>>  development discussions." <emacs-devel@gnu.org>,
>>         Eli Zaretskii
>>  <eliz@gnu.org>
>> Date: Wed, 10 Feb 2021 08:44:57 +0000
>> 
>> Compiling a trampoline is about 0.2s each (mostly GCC gas ld startup
>> time).  Say we have ~1400 primitives this should translate into ~4.5 min
>> of build time for something that in the average case is largely unused.
>
> 4.5 min out of how much?  Isn't building Emacs with native-comp take a
> long time anyway?

IIRC the full AOT build on my dev machine (8 core 16 threads) is ~19mins.

>> The trouble is that the simple code snippet I posted is not parallel and
>> there's no trivial way to make make aware of all these targets to have
>> it handling the parallelism.
>
> How about if we make the necessary assembly snippet part of the DEFUN
> macro?  Then compiling each .c file will produce those trampolines
> ready to be used, no?  (I admit that I don't really understand the
> details of those trampolines, so maybe the above makes no sense.)

Yes, that would be equivalent for a primitve foo to expand also
something like.

Lisp_Object
Ffoo_trampoline (Lisp_Object arg1, Lisp_Object arg2)
{
  return CALLN (Ffuncall, Qfoo, arg1, arg2);
}

At this point one need also a mechanism set the Ffoo_trampoline address
in our link table when foo is redefined.

One downside of this approach is that the binary is statically bloated
by rarely used code.

  Andrea

Re: [feature/native-comp] breakage on build

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: phillip.lord@russet.org.uk, monnier@iro.umontreal.ca, emacs-devel@gnu.org
> Date: Wed, 10 Feb 2021 20:41:07 +0000
> 
> > 4.5 min out of how much?  Isn't building Emacs with native-comp take a
> > long time anyway?
> 
> IIRC the full AOT build on my dev machine (8 core 16 threads) is ~19mins.

That's 20%, not too bad.

> > How about if we make the necessary assembly snippet part of the DEFUN
> > macro?  Then compiling each .c file will produce those trampolines
> > ready to be used, no?  (I admit that I don't really understand the
> > details of those trampolines, so maybe the above makes no sense.)
> 
> Yes, that would be equivalent for a primitve foo to expand also
> something like.
> 
> Lisp_Object
> Ffoo_trampoline (Lisp_Object arg1, Lisp_Object arg2)
> {
>   return CALLN (Ffuncall, Qfoo, arg1, arg2);
> }
> 
> At this point one need also a mechanism set the Ffoo_trampoline address
> in our link table when foo is redefined.
> 
> One downside of this approach is that the binary is statically bloated
> by rarely used code.

How many bytes is the trampoline?  With 1500 primitives, we are
talking perhaps about 20KB bloat?  That's not very significant, IMO.

Re: [feature/native-comp] breakage on build

[Andrea Corallo via Emacs development discussions.]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <akrl@sdf.org>
>> Cc: phillip.lord@russet.org.uk, monnier@iro.umontreal.ca, emacs-devel@gnu.org
>> Date: Wed, 10 Feb 2021 20:41:07 +0000
>> 
>> > 4.5 min out of how much?  Isn't building Emacs with native-comp take a
>> > long time anyway?
>> 
>> IIRC the full AOT build on my dev machine (8 core 16 threads) is ~19mins.
>
> That's 20%, not too bad.

Agreed, we could also improve this with some parallelism later on, BTW I
think most people will not build these AOT.

>> > How about if we make the necessary assembly snippet part of the DEFUN
>> > macro?  Then compiling each .c file will produce those trampolines
>> > ready to be used, no?  (I admit that I don't really understand the
>> > details of those trampolines, so maybe the above makes no sense.)
>> 
>> Yes, that would be equivalent for a primitve foo to expand also
>> something like.
>> 
>> Lisp_Object
>> Ffoo_trampoline (Lisp_Object arg1, Lisp_Object arg2)
>> {
>>   return CALLN (Ffuncall, Qfoo, arg1, arg2);
>> }
>> 
>> At this point one need also a mechanism set the Ffoo_trampoline address
>> in our link table when foo is redefined.
>> 
>> One downside of this approach is that the binary is statically bloated
>> by rarely used code.
>
> How many bytes is the trampoline?  With 1500 primitives, we are
> talking perhaps about 20KB bloat?  That's not very significant, IMO.

On my system each trampoline is 45K, this translates in ~65M bloat that
is probably more problematic.  OTOH this should be measured on Windows
as it really depends on the object format I guess.

  Andrea

Re: [feature/native-comp] breakage on build

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: phillip.lord@russet.org.uk, monnier@iro.umontreal.ca, emacs-devel@gnu.org
> Date: Thu, 11 Feb 2021 14:51:53 +0000
> 
> On my system each trampoline is 45K, this translates in ~65M bloat that
> is probably more problematic.

Why is it so large? what does a trampoline include that it takes so
much code?

Re: [feature/native-comp] breakage on build

[Andrea Corallo via Emacs development discussions.]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <akrl@sdf.org>
>> Cc: phillip.lord@russet.org.uk, monnier@iro.umontreal.ca, emacs-devel@gnu.org
>> Date: Thu, 11 Feb 2021 14:51:53 +0000
>> 
>> On my system each trampoline is 45K, this translates in ~65M bloat that
>> is probably more problematic.
>
> Why is it so large? what does a trampoline include that it takes so
> much code?

Good point,

Interestingly I've disassembled one trampoline and look like we are
still outputting in the binary a number of static functions that we
define but in this case are indeed not used.

I've to look into GCC what's going on, this is indeed undesired.

  Andrea

Re: [feature/native-comp] breakage on build

[Stefan Monnier]

>>> On my system each trampoline is 45K, this translates in ~65M bloat that
>>> is probably more problematic.

IIUC each trampoline gets its own .eln file, right?
If so, if we decide to prebuild all trampolines (and to keep not
sharing their code) we should likely build a single .eln file containing
them all.


        Stefan

Re: [feature/native-comp] breakage on build

[Andrea Corallo via Emacs development discussions.]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>>> On my system each trampoline is 45K, this translates in ~65M bloat that
>>>> is probably more problematic.
>
> IIUC each trampoline gets its own .eln file, right?
> If so, if we decide to prebuild all trampolines (and to keep not
> sharing their code) we should likely build a single .eln file containing
> them all.

That's a good solution.  It should also cut the compilation time that is
ATM mostly startup time.

OTOH I realized the reason of the 45K, trampolines are not optimized ATM
so GCC does not prune unused functions.  I'll modify to have them
compiled at least at O1 and follow-up on the corresponding footprint.

  Andrea

Re: [feature/native-comp] breakage on build

[Andrea Corallo via Emacs development discussions.]

Andrea Corallo via "Emacs development discussions."
<emacs-devel@gnu.org> writes:

> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>
>>>>> On my system each trampoline is 45K, this translates in ~65M bloat that
>>>>> is probably more problematic.
>>
>> IIUC each trampoline gets its own .eln file, right?
>> If so, if we decide to prebuild all trampolines (and to keep not
>> sharing their code) we should likely build a single .eln file containing
>> them all.
>
> That's a good solution.  It should also cut the compilation time that is
> ATM mostly startup time.
>
> OTOH I realized the reason of the 45K, trampolines are not optimized ATM
> so GCC does not prune unused functions.  I'll modify to have them
> compiled at least at O1 and follow-up on the corresponding footprint.

Okay compiling at -O1 makes each trampoline on my system 17K, so ~25M in
total for all ~1500 primitives.

The compilation time is almost the same, therfore I've set it as default
for trampolines.

If we want to go further into saving space Stefan's suggestion (the
single eln containing all trampolines) has to be implemented.

  Andrea

Re: [feature/native-comp] breakage on build

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>, Eli Zaretskii
>  <eliz@gnu.org>,
>         phillip.lord@russet.org.uk
> Date: Thu, 11 Feb 2021 20:52:20 +0000
> 
> > OTOH I realized the reason of the 45K, trampolines are not optimized ATM
> > so GCC does not prune unused functions.  I'll modify to have them
> > compiled at least at O1 and follow-up on the corresponding footprint.
> 
> Okay compiling at -O1 makes each trampoline on my system 17K, so ~25M in
> total for all ~1500 primitives.
> 
> The compilation time is almost the same, therfore I've set it as default
> for trampolines.

Thanks, but I'd still like to know what kind of code makes it so
large.  Can you describe what the trampoline does?

> If we want to go further into saving space Stefan's suggestion (the
> single eln containing all trampolines) has to be implemented.

Is it a lot of work?  If not, I think we should do it (unless we could
make the trampolines significantly smaller).

Re: [feature/native-comp] breakage on build

[Phillip Lord]

Andrea Corallo <akrl@sdf.org> writes:

> Phillip Lord <phillip.lord@russet.org.uk> writes:
>>> Correct, but if we can build eln files during the initial compilation we
>>> can also build trampolines AOT.  This is just a matter of running:
>>>
>>> (mapatoms (λ (s)    
>>>             (when (subr-primitive-p (symbol-function s))
>>>               (comp-trampoline-compile s))))
>>>
>>> Probably we should wire this in the build system so it's accessible as
>>> make target?
>>
>>
>> Why not wire it into the AOT compilation always? Obviously, if it
>> doubles the AOT compilation, that might not be such a great idea, but
>> otherwise?
>>
>> Phil
>
> That's a great question.
>
> Compiling a trampoline is about 0.2s each (mostly GCC gas ld startup
> time).  Say we have ~1400 primitives this should translate into ~4.5 min
> of build time for something that in the average case is largely unused.
>
> The trouble is that the simple code snippet I posted is not parallel and
> there's no trivial way to make make aware of all these targets to have
> it handling the parallelism.


Yeah, that's not good at all.

Phil

Re: [feature/native-comp] breakage on build

[Andrea Corallo via Emacs development discussions.]

Phillip Lord <phillip.lord@russet.org.uk> writes:

> Andrea Corallo <akrl@sdf.org> writes:
>
>> Phillip Lord <phillip.lord@russet.org.uk> writes:
>>>> Correct, but if we can build eln files during the initial compilation we
>>>> can also build trampolines AOT.  This is just a matter of running:
>>>>
>>>> (mapatoms (λ (s)    
>>>>             (when (subr-primitive-p (symbol-function s))
>>>>               (comp-trampoline-compile s))))
>>>>
>>>> Probably we should wire this in the build system so it's accessible as
>>>> make target?
>>>
>>>
>>> Why not wire it into the AOT compilation always? Obviously, if it
>>> doubles the AOT compilation, that might not be such a great idea, but
>>> otherwise?
>>>
>>> Phil
>>
>> That's a great question.
>>
>> Compiling a trampoline is about 0.2s each (mostly GCC gas ld startup
>> time).  Say we have ~1400 primitives this should translate into ~4.5 min
>> of build time for something that in the average case is largely unused.
>>
>> The trouble is that the simple code snippet I posted is not parallel and
>> there's no trivial way to make make aware of all these targets to have
>> it handling the parallelism.
>
>
> Yeah, that's not good at all.

Maybe one option is to invoke Emacs for this target passing the -J
number to it so we can handle the parallelism directly there.

  Andrea

Re: [feature/native-comp] breakage on build

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org
> Date: Mon, 08 Feb 2021 14:28:46 +0000
> 
> If we can't compile we cannot synthesize trampolines, as a
> consequence we can't redefine or advice primitive functions
> reliably...

Can you elaborate on this issue?  What exactly won't work, and how
hard would it be to try to fix that?

Thanks.

Re: [feature/native-comp] breakage on build

[Eli Zaretskii]

> From: Phillip Lord <phillip.lord@russet.org.uk>
> Cc: Eli Zaretskii <eliz@gnu.org>,  emacs-devel@gnu.org
> Date: Mon, 08 Feb 2021 12:08:57 +0000
> 
> For an Emacs compiled --with-nativecomp, how do I stop it attempting to
> native compile?

When libgccjit is available or when it isn't?  In the latter case it
should ideally happen automatically.

Re: [feature/native-comp] breakage on build

[Óscar Fuentes]

Phillip Lord <phillip.lord@russet.org.uk> writes:

> So, I did this:
>
>  - Installed a fresh msys to a clean directory on my desktop.
>  - Dropped the emacs-no-deps.zip directory on top of the mingw64 directory.
>  - clicked on runemacs.exe from windows shell -- Emacs fails to launch
>    because of libgmp
>  - pacman installed all the dependencies of Emacs
>  - clicked on runemacs.exe -- Emacs launches, native-comp fails as
>    described
>  - run mingw64 shell form the mys installation
>  - ./runemacs.exe in that shell
>  - Emacs launches, native-comp works
>  - ~/ and thus ~/.emacs.d/eln-cache is a different location with the two
>    launch methods
>
> I believe that msys is mostly portable and a new installation is nearly
> clean (except for .bashrc and such like).
>
> Conclusions:
>
>  - emacs is picking up dependencies from mingw64 run in either way,
>    because otherwise it wouldn't run at all from the shell
>  - Emacs is picking up the environment from mingw64 when launched from
>    within it, hence the differences in location of ~
>  - Native-comp has everything it needs to run, or it couldn't run at
>    all.
>  - It is missing something from the environment when run in the windows
>    shell.
>
> This is export run inside the mingw64 window. So, lots of options there.

If, at the top of .emacs.el, you add the binary directory of emacs.exe
to exec-path, does native-comp work when you start emacs from the
desktop's icon?

Also, some days ago I sent you an e-mail off-list. Have you seen it?

Re: [feature/native-comp] breakage on build

[Óscar Fuentes]

Óscar Fuentes <ofv@wanadoo.es> writes:

> Phillip Lord <phillip.lord@russet.org.uk> writes:
>
>> So, I did this:
>>
>>  - Installed a fresh msys to a clean directory on my desktop.
>>  - Dropped the emacs-no-deps.zip directory on top of the mingw64 directory.
>>  - clicked on runemacs.exe from windows shell -- Emacs fails to launch
>>    because of libgmp
>>  - pacman installed all the dependencies of Emacs
>>  - clicked on runemacs.exe -- Emacs launches, native-comp fails as
>>    described
>>  - run mingw64 shell form the mys installation
>>  - ./runemacs.exe in that shell
>>  - Emacs launches, native-comp works
>>  - ~/ and thus ~/.emacs.d/eln-cache is a different location with the two
>>    launch methods
>>
>> I believe that msys is mostly portable and a new installation is nearly
>> clean (except for .bashrc and such like).
>>
>> Conclusions:
>>
>>  - emacs is picking up dependencies from mingw64 run in either way,
>>    because otherwise it wouldn't run at all from the shell
>>  - Emacs is picking up the environment from mingw64 when launched from
>>    within it, hence the differences in location of ~
>>  - Native-comp has everything it needs to run, or it couldn't run at
>>    all.
>>  - It is missing something from the environment when run in the windows
>>    shell.
>>
>> This is export run inside the mingw64 window. So, lots of options there.
>
> If, at the top of .emacs.el, you add the binary directory of emacs.exe
> to exec-path, does native-comp work when you start emacs from the
> desktop's icon?
>
> Also, some days ago I sent you an e-mail off-list. Have you seen it?

On second thought, exec-path solves the problem of emacs.exe running
other executables, but it is not enough when those executables run other
executables on turn. So emacs.exe will launch gcc.exe just fine, but
gcc.exe will fail to run as.exe, ld.exe, etc.

So we must set PATH too.

Re: [feature/native-comp] breakage on build

[Andrea Corallo via Emacs development discussions.]

Phillip Lord <phillip.lord@russet.org.uk> writes:

> Andrea Corallo <akrl@sdf.org> writes:
>
>> Phillip Lord <phillip.lord@russet.org.uk> writes:
>>
>>> Andrea Corallo <akrl@sdf.org> writes:
>>>
>>>> Phillip Lord <phillip.lord@russet.org.uk> writes:
>>>>
>>> I've tried adding ld.exe and as.exe, and altering the path (at runtime
>>> which Eli says is bad).
>>>
>>> I am now getting this error message; it's different so that's obviously
>>> part of the problem.
>>>
>>>    Warning (comp): Debugger entered--Lisp error: (native-ice
>>> "c:/msys64/home/Administrator/emacs-upload/emacs-28..." "failed to
>>> compile" "c:/Users/Administrator/AppData/Roaming/.emacs.d/el..."
>>> "error invoking gcc driver") Disable showing Disable logging
>>>
>>>
>>> I'm try following up on jit@gcc as you suggest. They may well know what
>>> I need for a minimal functional install. Once I have that, I can work
>>> out how to achieve it in the package. At this rate, though, I fear that
>>> I am moving back to shipping half of msys.
>>
>> Hi Phillip,
>>
>> you should see the full error in the *Async-native-compile-log*
>> buffer.
>
>
> Oh, yes, looking in the wrong buffer, silly me.
>
> It's ld/as not finding things that is causing the problem.
>
> Phil

Probably saying something stupid here as I've no idea how this packaging
process works in Windows or exactly what it means but anyway...

In GNU/linux I do something similar in my Docker recipe where I need to
move all the libgccjit stuff from the build container to the one I ship.

How I do it is just to configure and compile libgccjit using --prefix
and then in the target directory I know I've everything I need for that
[1].

Just an idea, hope it helps.

Thanks

  Andrea

[1] <https://gitlab.com/koral/emacs-nativecomp-dockerfile/-/blob/master/Dockerfile>

Re: [feature/native-comp] breakage on build

[Phillip Lord]

Andrea Corallo <akrl@sdf.org> writes:

> Phillip Lord <phillip.lord@russet.org.uk> writes:
>
>>>
>>> Hi Phillip,
>>>
>>> you should see the full error in the *Async-native-compile-log*
>>> buffer.
>>
>>
>> Oh, yes, looking in the wrong buffer, silly me.
>>
>> It's ld/as not finding things that is causing the problem.
>>
>> Phil
>
> Probably saying something stupid here as I've no idea how this packaging
> process works in Windows or exactly what it means but anyway...

It means that I compile under mingw64, then I have a hacky shell script
that works out what Emacs' dependencies are which zips them all
up. In essence, what I am really doing is creating a minimal mingw64
installation with just enough stuff to run Emacs correctly.

There would be a certain amount of sense in not doing it, and just
assume people will use chocolaty or msys2 directly. But that's not very
windowish.



> In GNU/linux I do something similar in my Docker recipe where I need to
> move all the libgccjit stuff from the build container to the one I ship.
>
> How I do it is just to configure and compile libgccjit using --prefix
> and then in the target directory I know I've everything I need for that
> [1].
>
> Just an idea, hope it helps.


I think that this is not enough. You'll have libgccgit inside your
--prefix, but are you not reliant on the rest of your container for any
dependencies of libgccjit?

Phil

Re: [feature/native-comp] breakage on build

[Eli Zaretskii]

> From: Phillip Lord <phillip.lord@russet.org.uk>
> Cc: akrl--- via "Emacs development discussions." <emacs-devel@gnu.org>,  Eli
>  Zaretskii <eliz@gnu.org>
> Date: Thu, 04 Feb 2021 21:07:40 +0000
> 
> At this rate, though, I fear that I am moving back to shipping half
> of msys.

If it comes close to that, my suggestion would be to separate the
parts required for JIT native compilation into a separate zip.  Then
people who already have the development environment installed will not
need to download a huge distribution full of stuff they already have,
and others will have a choice of deciding whether the ability to
natively-compile Lisp files not included in the tarball is worth the
hassle of downloading a large distribution.

The need to have all these tools handy is indeed the main disadvantage
of native-compilation implementation that uses a JIT compiler.  So it
comes as no surprise to me that we bump into these difficulties.

Re: [feature/native-comp] breakage on build

[Óscar Fuentes]

akrl--- via "Emacs development discussions." <emacs-devel@gnu.org>
writes:

> The lenght of these hashes is coming directly from the algo we use
> (md5), I think collision is really not an issue so we can easlily short
> these.

A non-cryptographic 32-bit hash (64-bit if you are paranoid) should do
fine.

Re: [feature/native-comp] breakage on build

[Stefan Monnier]

>> The lenght of these hashes is coming directly from the algo we use
>> (md5), I think collision is really not an issue so we can easlily short
>> these.
> A non-cryptographic 32-bit hash (64-bit if you are paranoid) should do
> fine.

We can also use base64 encoding instead of hex, so as to pack 50% more
bits per byte.


        Stefan

Re: [feature/native-comp] breakage on build

[Óscar Fuentes]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> The lenght of these hashes is coming directly from the algo we use
>>> (md5), I think collision is really not an issue so we can easlily short
>>> these.
>> A non-cryptographic 32-bit hash (64-bit if you are paranoid) should do
>> fine.
>
> We can also use base64 encoding instead of hex, so as to pack 50% more
> bits per byte.

That looks like a good idea until you take into account case-insensitive
file systems.

Re: [feature/native-comp] breakage on build

[Stefan Monnier]

> That looks like a good idea until you take into account case-insensitive
> file systems.

AFAICT it'd still work on those filesystems (and even more so if they
preserve the case information, as seems to be the standard behavior).


        Stefan

Re: [feature/native-comp] breakage on build

[Andy Moreton]

On Sat 30 Jan 2021, akrl--- via "Emacs development discussions." wrote:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> Date: Sat, 30 Jan 2021 18:38:10 +0200
>>> From: Eli Zaretskii <eliz@gnu.org>
>>> Cc: akrl@sdf.org, emacs-devel@gnu.org
>>> 
>>>   compiling to
>>> c:/msys64/home/Administrator/emacs-build/build/emacs-28.0.50-snapshot-feature_native-comp-windows/native-lisp/28.0.50-x86_64-w64-mingw32-3459bc949453c7833f94b407f8ee7c17/titdic-cnv-765ac3beca7090ba17f799d9600c18e6-a5dbcc84a969a94717512a079e1cc96a.eln
>>> 
>>> The name of that file is suspiciously close to the 260-byte limit of
>>> file-name length that Win32 APIs support.  So my guess is that the
>>> temporary file had a few more characters (like the 6 characters which
>>> replace the XXXXXX, perhaps?), and that caused the failure.
>>
>> Btw, Andrea: why do we need no less than 3 32-character hashes in that
>> file name?  With 26-byte limit on file names that libgccjit can use,
>> these 96 characters use up almost half of that space, and that will
>> bite us time and again down the road on MS-Windows.  Can we use just
>> one hash?
>>
>
> Say ATM we have:
>
> native-lisp/28.0.50-x86_64-w64-mingw32-HASH1/titdic-cnv-HASH2-HASH3.eln
>
> - HASH1 disambiguate triplet, Emacs configuration, version etc.  We can
>   say this is disambiguating everything that influence the ABI the eln
>   expects and exposes.
>
> - HASH2 is just hashing the full patch of the original source file.

This is unclear. Did you mean the content of the source, or its filename ?

> - HASH3 is hashing the content of the file.
>
> HASH1 is handy because one can remove selectively the eln files of say a
> previous Emacs version or an old configuration.  I often remove all of
> these folders but the most recent one.  That said technically HASH1 and
> HASH3 could be merged.

This should be done: it is better to simplify the filesystem layout and
provide some utility functions to help with removing stale cache entries.

> HASH2 is handy because there must be within this folder at most one file
> with each HASH2.  IOW recompiling we remove all the other .eln sharing
> the same HASH2 if present as indeed these files are obsolete.

So if the same source same rebuilt with different eln compile flags then
only one of them is retained ? That would need to change if HASH1 and
HASH3 are merged for the .eln filename.

    AndyM

Re: [feature/native-comp] breakage on build

[akrl--- via Emacs development discussions.]

Andy Moreton <andrewjmoreton@gmail.com> writes:

> On Sat 30 Jan 2021, akrl--- via "Emacs development discussions." wrote:
>
>> Eli Zaretskii <eliz@gnu.org> writes:
>>
>>>> Date: Sat, 30 Jan 2021 18:38:10 +0200
>>>> From: Eli Zaretskii <eliz@gnu.org>
>>>> Cc: akrl@sdf.org, emacs-devel@gnu.org
>>>> 
>>>>   compiling to
>>>> c:/msys64/home/Administrator/emacs-build/build/emacs-28.0.50-snapshot-feature_native-comp-windows/native-lisp/28.0.50-x86_64-w64-mingw32-3459bc949453c7833f94b407f8ee7c17/titdic-cnv-765ac3beca7090ba17f799d9600c18e6-a5dbcc84a969a94717512a079e1cc96a.eln
>>>> 
>>>> The name of that file is suspiciously close to the 260-byte limit of
>>>> file-name length that Win32 APIs support.  So my guess is that the
>>>> temporary file had a few more characters (like the 6 characters which
>>>> replace the XXXXXX, perhaps?), and that caused the failure.
>>>
>>> Btw, Andrea: why do we need no less than 3 32-character hashes in that
>>> file name?  With 26-byte limit on file names that libgccjit can use,
>>> these 96 characters use up almost half of that space, and that will
>>> bite us time and again down the road on MS-Windows.  Can we use just
>>> one hash?
>>>
>>
>> Say ATM we have:
>>
>> native-lisp/28.0.50-x86_64-w64-mingw32-HASH1/titdic-cnv-HASH2-HASH3.eln
>>
>> - HASH1 disambiguate triplet, Emacs configuration, version etc.  We can
>>   say this is disambiguating everything that influence the ABI the eln
>>   expects and exposes.
>>
>> - HASH2 is just hashing the full patch of the original source file.
>
> This is unclear. Did you mean the content of the source, or its filename ?

Sorry typo; wanted to write path, so please read filename.

>> - HASH3 is hashing the content of the file.
>>
>> HASH1 is handy because one can remove selectively the eln files of say a
>> previous Emacs version or an old configuration.  I often remove all of
>> these folders but the most recent one.  That said technically HASH1 and
>> HASH3 could be merged.
>
> This should be done: it is better to simplify the filesystem layout and
> provide some utility functions to help with removing stale cache entries.

Please design and implement such a system, when will be proved it works
well I'm sure these patches will be welcome.

>> HASH2 is handy because there must be within this folder at most one file
>> with each HASH2.  IOW recompiling we remove all the other .eln sharing
>> the same HASH2 if present as indeed these files are obsolete.
>
> So if the same source same rebuilt with different eln compile flags then
> only one of them is retained ?

No, HASH2 is not influenced by eln compilation flags.

  Andrea

Re: [feature/native-comp] breakage on build

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: phillip.lord@russet.org.uk,  emacs-devel@gnu.org
> Date: Mon, 01 Feb 2021 16:20:17 +0000
> 
> > OK, thanks.  I understand the reasons now.  However, I'm not sure we
> > should encode all this information in the .eln file's name.  For
> > starters, some of those reasons are also true for *.elc files, right?
> > For example, suppose that a primitive that the .el file calls changed
> > its signature in backward-incompatible ways -- the .el files which
> > call it need to be recompiled, or else they will fail, either silently
> > or noisily.  Right?  But we don't encode those details in the names of
> > the *.elc files, do we?  Why should the *.eln files be different?
> 
> I think with eln the situation is considerably more sentitive than the
> elc one in this respect.  The reason is that we (on purpose) skip
> Ffuncall to call directly into C code, as a consequence an
> incompatibility will most certainly lead to a crash and not a runtime
> error.

Crash aside, I don't see the crucial difference, sorry.  And if we
detect incompatibility and prevent the crash, I seed no difference at
all, as that can happen with *.elc files as well.

> 2- We move HASH1 into the eln as metadata.
> 
>   - The limitation of 1 persists
> 
>   - To check that each eln is loadable one has: to 'dlopen' it, read the
>     metadata ('dlsym'), verify and in case reject.  This is certainly
>     way more expansive than directly searching for a given filename and
>     might impact negatively startup time, especially on certain OSes
>     where opening files is an expensive operation.

How much more expensive?  I'd be surprised to hear it's significant,
since in-memory operations are much faster than file I/O.  Is it
possible to measure the difference, so that we could make a decision
based on facts?

>   - We'd have name clashes for different eln coming from different
>     Emacs configurations/versions.

A different version will have a different directory, like it does now
with *.elc files.  As for different configurations: if you mean the
same architecture, how probable is that, for normal users?  If the
issue happens with Emacs developers, wed could design something
special for that case.

I guess I'm saying that using the file names to record this
information is not necessarily justified by the issues it is supposed
to solve, and there are alternative solutions with smaller costs.

> Shall we pick a length?  8 characters is probably more than okay but we
> can go for 16 if we prefer to stay on the safe side.

If 8 is enough, then why go for 16?  How is that safer?