bug#60996: 29.0.60; Native compile fails to remove temp file for trampoline

bug#60996: 29.0.60; Native compile fails to remove temp file for trampoline

[Andy Moreton]

Recently emacs 29 (and master) has started showing an error and
backtrace during startup:

Debugger entered--Lisp error: (permission-denied "Removing old name" 
"Permission denied" "c:/Users/ajm/AppData/Local/Temp/comp-lambda-MTAMbr...")
delete-file("c:/Users/ajm/AppData/Local/Temp/comp-lambda-MTAMbr...")
   #f(compiled-function () #<bytecode 0xb2aa5c0c9e24501>)()
   comp--native-compile((lambda (arg0 &optional arg1 arg2 arg3) (let ((f 
#'format-mode-line)) (funcall f arg0 arg1 arg2 arg3))) nil nil)
   comp-trampoline-compile(format-mode-line)
   comp-subr-trampoline-install(format-mode-line)
   advice--add-function(:around (#f(compiled-function () #<bytecode 
0x3c5d8019df7e91>) . #f(compiled-function (gv--val) #<bytecode 
0x8fdecbbba7cb3c2>)) delight--format-mode-line nil)
   advice-add(format-mode-line :around delight--format-mode-line)
   require(delight nil t)
   (not (require 'delight nil t))
   (if (not (require 'delight nil t)) (display-warning 'use-package 
(format "Cannot load %s" 'delight) :error))
   (prog1 (if (not (require 'delight nil t)) (display-warning 
'use-package (format "Cannot load %s" 'delight) :error)) (let ((elapsed 
(float-time (time-subtract (current-time) now)))) (if (> elapsed 0.1) 
(message "%s...done (%.3fs)" "Loading package delight" elapsed) (message 
"%s...done" "Loading package delight"))))
   (let ((now (current-time))) (message "%s..." "Loading package 
delight") (prog1 (if (not (require 'delight nil t)) (display-warning 
'use-package (format "Cannot load %s" 'delight) :error)) (let ((elapsed 
(float-time (time-subtract (current-time) now)))) (if (> elapsed 0.1) 
(message "%s...done (%.3fs)" "Loading package delight" elapsed) (message 
"%s...done" "Loading package delight")))))
   (condition-case err (let ((now (current-time))) (message "%s..." 
"Loading package delight") (prog1 (if (not (require 'delight nil t)) 
(display-warning 'use-package (format "Cannot load %s" 'delight) 
:error)) (let ((elapsed (float-time (time-subtract ... now)))) (if (> 
elapsed 0.1) (message "%s...done (%.3fs)" "Loading package delight" 
elapsed) (message "%s...done" "Loading package delight"))))) ((debug 
error) (funcall use-package--warning1 :catch err)))
   load-with-code-conversion("c:/home/ajm/.emacs.d/init.el" 
"c:/home/ajm/.emacs.d/init.el" t t)
   load("c:/home/ajm/.emacs.d/init" noerror nomessage)
   startup--load-user-init-file(#f(compiled-function () #<bytecode 
-0x1cd3443936cf717e>) #f(compiled-function () #<bytecode 
-0x1f3c61addc0c4f35>) t)
   command-line()
   normal-top-level()

Tracing execution of emacs with Process Explorer shows that the temp
file used to native compile trampolines is opened and closed repeatedly
by emacs, and at the point of the backtrace is still open by the same
emacs process.

Adding a workaround to .emacs.d/init.el mitigates this problem:
  (setq comp-enable-subr-trampolines nil)

Running "emacs -Q" does not show the problem, but is then less likely to
native compile trampolines during startup.

I am not sure excactly when this issue started, but I did not see it in
emacs-29 or master bootstrapped before this month.

     AndyM



In GNU Emacs 29.0.60 (build 8, x86_64-w64-mingw32) of 2023-01-21 built
  on QUIETUS
Repository revision: b875c9bf67ebf858648a00307c370d7a196aab56
Repository branch: emacs-29
Windowing system distributor 'Microsoft Corp.', version 10.0.19045
System Description: Microsoft Windows 10 Pro (v10.0.2009.19045.2486)

Configured using:
  'configure --prefix=c:/emacs/29.0.60/mingw64-x86_64-O2-native-aot
  --cache-file=/c/emacs/git/emacs/emacs-29/build/mingw64-x86_64-O2-native-aot/config.cache
  --without-compress-install --without-dbus --with-gif --with-gnutls
  --without-imagemagick --with-jpeg --with-json --with-lcms2
  --with-modules --with-native-compilation=aot --with-png --without-pop
  --with-rsvg --with-sqlite3 --with-tiff --with-tree-sitter --with-xml2
  --with-xpm --enable-checking 'CPPFLAGS= -DNO_SHLWAPI_ISOS=1' 'CFLAGS=
  -O2 -g3 -gdwarf-4 -fdiagnostics-color=never'
  PKG_CONFIG_PATH=/mingw64/lib/pkgconfig:/mingw64/share/pkgconfig'

Configured features:
ACL GIF GMP GNUTLS HARFBUZZ JPEG JSON LCMS2 LIBXML2 MODULES NATIVE_COMP
NOTIFY W32NOTIFY PDUMPER PNG RSVG SOUND SQLITE3 THREADS TIFF
TOOLKIT_SCROLL_BARS TREE_SITTER WEBP XPM ZLIB

Important settings:
   value of $LANG: ENG
   locale-coding-system: cp1252

Major mode: ELisp/l

Minor modes in effect:
   hexl-follow-ascii: t
   which-function-mode: t
   global-so-long-mode: t
   display-fill-column-indicator-mode: t
   desktop-save-mode: t
   auto-image-file-mode: t
   minibuffer-electric-default-mode: t
   override-global-mode: t
   tooltip-mode: t
   global-eldoc-mode: t
   eldoc-mode: t
   show-paren-mode: t
   mouse-wheel-mode: t
   menu-bar-mode: t
   file-name-shadow-mode: t
   global-font-lock-mode: t
   font-lock-mode: t
   blink-cursor-mode: t
   column-number-mode: t
   line-number-mode: t
   transient-mark-mode: t
   auto-composition-mode: t
   auto-compression-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message yank-media rfc822 mml mml-sec
epa derived gnus-util mm-decode mm-bodies mm-encode mail-parse rfc2231
mailabbrev gmm-utils mailheader sendmail rfc2047 rfc2045 ietf-drums
mm-util mail-prsvr mail-utils add-log time mule-util jka-compr
bug-reference dired-aux xcscope autorevert dired-x dired dired-loaddefs
sh-script treesit executable time-date vc-svn rnc-mode rng-nxml
rng-valid rng-loc rng-uri rng-parse nxml-parse rng-match rng-dt rng-util
rng-pttrn nxml-ns nxml-mode nxml-outln nxml-rap sgml-mode facemenu dom
nxml-util nxml-enc xmltok lua-mode advice rust-utils rust-mode
rust-rustfmt rust-playpen rust-compile rust-cargo meson-mode smie eglot
external-completion array filenotify jsonrpc ert ewoc debug backtrace
find-func xref flymake-proc flymake thingatpt project hexl grep vc-git
diff-mode vc-dispatcher graphviz-dot-mode compile text-property-search
xr which-func imenu so-long display-fill-column-indicator desktop
frameset cygwin-mount ange-ftp comint ansi-osc ansi-color ring hl-line
pcase image-file image-converter edmacro kmacro use-package-bind-key
use-package-delight minibuf-eldef delight comp comp-cstr warnings rx
bind-key easy-mmode cl-extra help-mode use-package-ensure
use-package-core finder-inf yaml-mode-autoloads xr-autoloads
rust-mode-autoloads rnc-mode-autoloads powershell-autoloads
plantuml-mode-autoloads meson-mode-autoloads markdown-mode-autoloads
lua-mode-autoloads htmlize-autoloads haskell-mode-autoloads
graphviz-dot-mode-autoloads gnuplot-autoloads
gnu-elpa-keyring-update-autoloads epg rfc6068 epg-config
gnu-elpa-keyring-update delight-autoloads debbugs-autoloads info
dash-autoloads cus-edit pp cus-load icons wid-edit package browse-url
url url-proxy url-privacy url-expand url-methods url-history url-cookie
generate-lisp-file url-domsuf url-util mailcap url-handlers url-parse
auth-source cl-seq eieio eieio-core cl-macs password-cache json url-vars
nsm map byte-opt gv bytecomp byte-compile subr-x gnutls puny cl-loaddefs
cl-lib rmc iso-transl tooltip cconv eldoc paren electric uniquify
ediff-hook vc-hooks lisp-float-type elisp-mode mwheel dos-w32 ls-lisp
disp-table term/w32-win w32-win w32-vars term/common-win tool-bar dnd
fontset image regexp-opt fringe tabulated-list replace newcomment
text-mode lisp-mode prog-mode register page tab-bar menu-bar rfn-eshadow
isearch easymenu timer select scroll-bar mouse jit-lock font-lock syntax
font-core term/tty-colors frame minibuffer nadvice seq simple cl-generic
indonesian philippine cham georgian utf-8-lang misc-lang vietnamese
tibetan thai tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek
romanian slovak czech european ethiopic indian cyrillic chinese
composite emoji-zwj charscript charprop case-table epa-hook
jka-cmpr-hook help abbrev obarray oclosure cl-preloaded button loaddefs
theme-loaddefs faces cus-face macroexp files window text-properties
overlay sha1 md5 base64 format env code-pages mule custom widget keymap
hashtable-print-readable backquote threads w32notify w32 lcms2 multi-tty
make-network-process native-compile emacs)

Memory information:
((conses 16 388657 216671)
  (symbols 48 25171 486)
  (strings 32 119123 37613)
  (string-bytes 1 3096676)
  (vectors 16 49749)
  (vector-slots 8 1397743 432730)
  (floats 8 85 750)
  (intervals 56 5657 2026)
  (buffers 984 25))

bug#60996: 29.0.60; Native compile fails to remove temp file for trampoline

[Eli Zaretskii]

> Date: Sat, 21 Jan 2023 22:12:10 +0000
> From: Andy Moreton <andrewjmoreton@gmail.com>
> 
> Recently emacs 29 (and master) has started showing an error and
> backtrace during startup:
> 
> Debugger entered--Lisp error: (permission-denied "Removing old name" 
> "Permission denied" "c:/Users/ajm/AppData/Local/Temp/comp-lambda-MTAMbr...")
> delete-file("c:/Users/ajm/AppData/Local/Temp/comp-lambda-MTAMbr...")

We need a reproducible recipe to investigate this, or results of such
investigation by you: which code has the file open when we try
deleting it, and why that other code has it open?

For a recipe, it should be enough to present a minimal init file which
causes the problem (but pleased make it really minimal: as few lines
as strictly needed for reproduction)

Btw, "comp-lambda-MTAMbr..." seems to tell that it's some part of
comp.el, which sounds strange: comp.el is supposed to be
natively-compiled during the build, and that includes the trampolines
for it.  Hmm...

> Tracing execution of emacs with Process Explorer shows that the temp
> file used to native compile trampolines is opened and closed repeatedly
> by emacs, and at the point of the backtrace is still open by the same
> emacs process.

We need to know which code opened it the last time and didn't close
it.  Can you figure that out?  All the files Emacs opens go through 2
functions in w32.c: sys_fopen and sys_open, so by running with 2
breakpoints there that show the backtrace and continue, you should be
able to see the culprit, and we can then take it from there.

> I am not sure excactly when this issue started, but I did not see it in
> emacs-29 or master bootstrapped before this month.

Could be because we now compile trampolines differently (to avoid the
danger of the "fork bomb" due to recursive compilation of
trampolines by async subprocesses).

Andrea, any suggestions or comments?

Thanks.

bug#60996: 29.0.60; Native compile fails to remove temp file for trampoline

[Andy Moreton]

On Sun 22 Jan 2023, Eli Zaretskii wrote:

>> Date: Sat, 21 Jan 2023 22:12:10 +0000
>> From: Andy Moreton <andrewjmoreton@gmail.com>
>> 
>> Recently emacs 29 (and master) has started showing an error and
>> backtrace during startup:
>> 
>> Debugger entered--Lisp error: (permission-denied "Removing old name" 
>> "Permission denied" "c:/Users/ajm/AppData/Local/Temp/comp-lambda-MTAMbr...")
>> delete-file("c:/Users/ajm/AppData/Local/Temp/comp-lambda-MTAMbr...")
>
> We need a reproducible recipe to investigate this, or results of such
> investigation by you: which code has the file open when we try
> deleting it, and why that other code has it open?
>
> For a recipe, it should be enough to present a minimal init file which
> causes the problem (but pleased make it really minimal: as few lines
> as strictly needed for reproduction)

It may take considerable time and effort to reduce my init file down to a
simple reproducer...

> Btw, "comp-lambda-MTAMbr..." seems to tell that it's some part of
> comp.el, which sounds strange: comp.el is supposed to be
> natively-compiled during the build, and that includes the trampolines
> for it.  Hmm...

The backtrace always seems to contain:

  delete file("path/to/comp-lambda-XXXXX.eln")
  comp--native-compile((lambda (...) ...))
  comp-trampoline-compile(function-name)
  comp-subr-trampoline-install(function-name)
  advice--add-function(...)
  advice-add(function-name ...)

So it looks very much like compiling trampolines is involved.
  
>> Tracing execution of emacs with Process Explorer shows that the temp
>> file used to native compile trampolines is opened and closed repeatedly
>> by emacs, and at the point of the backtrace is still open by the same
>> emacs process.
>
> We need to know which code opened it the last time and didn't close
> it.  Can you figure that out?  All the files Emacs opens go through 2
> functions in w32.c: sys_fopen and sys_open, so by running with 2
> breakpoints there that show the backtrace and continue, you should be
> able to see the culprit, and we can then take it from there.

Thanks. I'll take a look in gdb and report back if I get further info.

>> I am not sure excactly when this issue started, but I did not see it in
>> emacs-29 or master bootstrapped before this month.
>
> Could be because we now compile trampolines differently (to avoid the
> danger of the "fork bomb" due to recursive compilation of
> trampolines by async subprocesses).
>
> Andrea, any suggestions or comments?
>
> Thanks.

bug#60996: 29.0.60; Native compile fails to remove temp file for trampoline

[Andrea Corallo]

Andy Moreton <andrewjmoreton@gmail.com> writes:

> On Sun 22 Jan 2023, Eli Zaretskii wrote:
>
>>> Date: Sat, 21 Jan 2023 22:12:10 +0000
>>> From: Andy Moreton <andrewjmoreton@gmail.com>
>>> 
>>> Recently emacs 29 (and master) has started showing an error and
>>> backtrace during startup:
>>> 
>>> Debugger entered--Lisp error: (permission-denied "Removing old name" 
>>> "Permission denied" "c:/Users/ajm/AppData/Local/Temp/comp-lambda-MTAMbr...")
>>> delete-file("c:/Users/ajm/AppData/Local/Temp/comp-lambda-MTAMbr...")
>>
>> We need a reproducible recipe to investigate this, or results of such
>> investigation by you: which code has the file open when we try
>> deleting it, and why that other code has it open?
>>
>> For a recipe, it should be enough to present a minimal init file which
>> causes the problem (but pleased make it really minimal: as few lines
>> as strictly needed for reproduction)
>
> It may take considerable time and effort to reduce my init file down to a
> simple reproducer...
>
>> Btw, "comp-lambda-MTAMbr..." seems to tell that it's some part of
>> comp.el, which sounds strange: comp.el is supposed to be
>> natively-compiled during the build, and that includes the trampolines
>> for it.  Hmm...
>
> The backtrace always seems to contain:
>
>   delete file("path/to/comp-lambda-XXXXX.eln")
>   comp--native-compile((lambda (...) ...))
>   comp-trampoline-compile(function-name)
>   comp-subr-trampoline-install(function-name)
>   advice--add-function(...)
>   advice-add(function-name ...)
>
> So it looks very much like compiling trampolines is involved.

Hello all,

yes I confirm that's a trampoline compilation.  The file starts with
"lambda" as the mechanism is the same used for compiling function not
not defined in source files.

  Andrea

bug#60996: 29.0.60; Native compile fails to remove temp file for trampoline

[Eli Zaretskii]

> Cc: 60996@debbugs.gnu.org
> From: Andrea Corallo <akrl@sdf.org>
> Date: Mon, 23 Jan 2023 17:04:30 +0000
> 
> yes I confirm that's a trampoline compilation.  The file starts with
> "lambda" as the mechanism is the same used for compiling function not
> not defined in source files.

Thanks.  Could the problem be caused by the fact that we nowadays
compile trampolines in the same session where it is needed, instead of
in a separate sub-process?  IOW, could this cause a situation where we
are still using the renamed .eln.tmp trampoline when we want to delete
it?

bug#60996: 29.0.60; Native compile fails to remove temp file for trampoline

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> Cc: 60996@debbugs.gnu.org
>> From: Andrea Corallo <akrl@sdf.org>
>> Date: Mon, 23 Jan 2023 17:04:30 +0000
>> 
>> yes I confirm that's a trampoline compilation.  The file starts with
>> "lambda" as the mechanism is the same used for compiling function not
>> not defined in source files.
>
> Thanks.  Could the problem be caused by the fact that we nowadays
> compile trampolines in the same session where it is needed, instead of
> in a separate sub-process?  IOW, could this cause a situation where we
> are still using the renamed .eln.tmp trampoline when we want to delete
> it?

I don't know, still I've to understand why we try to delete this
trampoline.

What I've understood from the trace is that `comp--native-compile' is
trying to delete the trampoline that was just compiled.

This is not expected in normal conditions so we have to understand why.

The only case where we might do that AFAIR is when
`inhibit-automatic-native-compilation' is used.  This was the infamous
mechanism that was installed by Lars where, if I'm not wrong, we are
supposed to compile a trampoline, load it, and remove it to pretend we
didn't compiled anything :x

If (as I understand) in Windows we can't delete a mapped file we have a
problem more to solve.

Andrew do you happen to have `inhibit-automatic-native-compilation' set?

If not could we debug why we reach delete-file in this piece of code at
the end of `comp--native-compile'?

        (when (and (not (stringp function-or-file))
                   comp-ctxt
                   (comp-ctxt-output comp-ctxt)
                   (file-exists-p (comp-ctxt-output comp-ctxt)))
          (delete-file (comp-ctxt-output comp-ctxt)))

Thanks

  Andrea

bug#60996: 29.0.60; Native compile fails to remove temp file for trampoline

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: andrewjmoreton@gmail.com, 60996@debbugs.gnu.org
> Date: Thu, 26 Jan 2023 16:50:59 +0000
> 
> What I've understood from the trace is that `comp--native-compile' is
> trying to delete the trampoline that was just compiled.
> 
> This is not expected in normal conditions so we have to understand why.
> 
> The only case where we might do that AFAIR is when
> `inhibit-automatic-native-compilation' is used.  This was the infamous
> mechanism that was installed by Lars where, if I'm not wrong, we are
> supposed to compile a trampoline, load it, and remove it to pretend we
> didn't compiled anything :x

OK, this seems to be what is happening here, because we compile the
trampoline to a temporary directory.  Otherwise, I don't see why we
would do that, and why we would delete a trampoline we just compiled.

> If (as I understand) in Windows we can't delete a mapped file we have a
> problem more to solve.

Yes.  If this is what happens, then I think we will have to maintain a
list of such trampolines, and delete them just before exiting, after
calling dynlib_close for them.

bug#60996: 29.0.60; Native compile fails to remove temp file for trampoline

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <akrl@sdf.org>
>> Cc: andrewjmoreton@gmail.com, 60996@debbugs.gnu.org
>> Date: Thu, 26 Jan 2023 16:50:59 +0000
>> 
>> What I've understood from the trace is that `comp--native-compile' is
>> trying to delete the trampoline that was just compiled.
>> 
>> This is not expected in normal conditions so we have to understand why.
>> 
>> The only case where we might do that AFAIR is when
>> `inhibit-automatic-native-compilation' is used.  This was the infamous
>> mechanism that was installed by Lars where, if I'm not wrong, we are
>> supposed to compile a trampoline, load it, and remove it to pretend we
>> didn't compiled anything :x
>
> OK, this seems to be what is happening here, because we compile the
> trampoline to a temporary directory.  Otherwise, I don't see why we
> would do that, and why we would delete a trampoline we just compiled.

Agree, or at least I hope (otherwise we have two problems in place of
one :).

>> If (as I understand) in Windows we can't delete a mapped file we have a
>> problem more to solve.
>
> Yes.  If this is what happens, then I think we will have to maintain a
> list of such trampolines, and delete them just before exiting, after
> calling dynlib_close for them.

Mmmhh, and what if another Emacs tries to delete or overwrite the same
file?

But my question is: is this mechanism _really_ necessary?

From my POV it's just a kludge that was, is and will be source of
problems.  Was never clear to me for which specific reason this was
implemented as, at the time, I had the impression that all Debian
requirements could be handled with what we already offered.

At the time I firmly opposed to this change, but I was told by Lars it
went into master "for discussion" (!?), indeed the review discussion
never happened...  and now we are left with this present.

Unless we have a very strong reason to keep it, I believe we should just
get rid of this mechanism, otherwise it will be source of pain for us
again and again in the future.

Best Regards

  Andrea

bug#60996: 29.0.60; Native compile fails to remove temp file for trampoline

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: andrewjmoreton@gmail.com, 60996@debbugs.gnu.org
> Date: Thu, 26 Jan 2023 19:46:25 +0000
> 
> >> If (as I understand) in Windows we can't delete a mapped file we have a
> >> problem more to solve.
> >
> > Yes.  If this is what happens, then I think we will have to maintain a
> > list of such trampolines, and delete them just before exiting, after
> > calling dynlib_close for them.
> 
> Mmmhh, and what if another Emacs tries to delete or overwrite the same
> file?

This shouldn't happen, since the *.eln files are generated with
make-temp-file.

> But my question is: is this mechanism _really_ necessary?
> 
> >From my POV it's just a kludge that was, is and will be source of
> problems.  Was never clear to me for which specific reason this was
> implemented as, at the time, I had the impression that all Debian
> requirements could be handled with what we already offered.
> 
> At the time I firmly opposed to this change, but I was told by Lars it
> went into master "for discussion" (!?), indeed the review discussion
> never happened...  and now we are left with this present.
> 
> Unless we have a very strong reason to keep it, I believe we should just
> get rid of this mechanism, otherwise it will be source of pain for us
> again and again in the future.

I'll have to refresh my memory about the reasons and the actual
changeset.

bug#60996: 29.0.60; Native compile fails to remove temp file for trampoline

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <akrl@sdf.org>
>> Cc: andrewjmoreton@gmail.com, 60996@debbugs.gnu.org
>> Date: Thu, 26 Jan 2023 19:46:25 +0000
>> 
>> >> If (as I understand) in Windows we can't delete a mapped file we have a
>> >> problem more to solve.
>> >
>> > Yes.  If this is what happens, then I think we will have to maintain a
>> > list of such trampolines, and delete them just before exiting, after
>> > calling dynlib_close for them.
>> 
>> Mmmhh, and what if another Emacs tries to delete or overwrite the same
>> file?
>
> This shouldn't happen, since the *.eln files are generated with
> make-temp-file.

Right sorry

>> But my question is: is this mechanism _really_ necessary?
>> 
>> >From my POV it's just a kludge that was, is and will be source of
>> problems.  Was never clear to me for which specific reason this was
>> implemented as, at the time, I had the impression that all Debian
>> requirements could be handled with what we already offered.
>> 
>> At the time I firmly opposed to this change, but I was told by Lars it
>> went into master "for discussion" (!?), indeed the review discussion
>> never happened...  and now we are left with this present.
>> 
>> Unless we have a very strong reason to keep it, I believe we should just
>> get rid of this mechanism, otherwise it will be source of pain for us
>> again and again in the future.
>
> I'll have to refresh my memory about the reasons and the actual
> changeset.

Thanks

BR

  Andrea

bug#60996: 29.0.60; Native compile fails to remove temp file for trampoline

[Eli Zaretskii]

> Cc: andrewjmoreton@gmail.com, 60996@debbugs.gnu.org
> Date: Thu, 26 Jan 2023 22:03:39 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> 
> > But my question is: is this mechanism _really_ necessary?
> > 
> > >From my POV it's just a kludge that was, is and will be source of
> > problems.  Was never clear to me for which specific reason this was
> > implemented as, at the time, I had the impression that all Debian
> > requirements could be handled with what we already offered.
> > 
> > At the time I firmly opposed to this change, but I was told by Lars it
> > went into master "for discussion" (!?), indeed the review discussion
> > never happened...  and now we are left with this present.
> > 
> > Unless we have a very strong reason to keep it, I believe we should just
> > get rid of this mechanism, otherwise it will be source of pain for us
> > again and again in the future.
> 
> I'll have to refresh my memory about the reasons and the actual
> changeset.

I started a discussion on emacs-devel about this, let's see where it
leads us.

I think for now I should simply wrap the delete-file call at the end
of 'comp--native-compile' with 'ignore-errors'.  WDYT?

bug#60996: 29.0.60; Native compile fails to remove temp file for trampoline

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> Cc: andrewjmoreton@gmail.com, 60996@debbugs.gnu.org
>> Date: Thu, 26 Jan 2023 22:03:39 +0200
>> From: Eli Zaretskii <eliz@gnu.org>
>> 
>> > But my question is: is this mechanism _really_ necessary?
>> > 
>> > >From my POV it's just a kludge that was, is and will be source of
>> > problems.  Was never clear to me for which specific reason this was
>> > implemented as, at the time, I had the impression that all Debian
>> > requirements could be handled with what we already offered.
>> > 
>> > At the time I firmly opposed to this change, but I was told by Lars it
>> > went into master "for discussion" (!?), indeed the review discussion
>> > never happened...  and now we are left with this present.
>> > 
>> > Unless we have a very strong reason to keep it, I believe we should just
>> > get rid of this mechanism, otherwise it will be source of pain for us
>> > again and again in the future.
>> 
>> I'll have to refresh my memory about the reasons and the actual
>> changeset.
>
> I started a discussion on emacs-devel about this, let's see where it
> leads us.

Thanks

> I think for now I should simply wrap the delete-file call at the end
> of 'comp--native-compile' with 'ignore-errors'.  WDYT?

Yeah agree, I'm fine with that as a temporary solution.  I hope we'll
not forget it there as definitive and we'll be able come up with a
decision on this.

Best Regards

  Andrea

bug#60996: 29.0.60; Native compile fails to remove temp file for trampoline

[Eli Zaretskii]

> Cc: andrewjmoreton@gmail.com, 60996@debbugs.gnu.org
> Date: Thu, 26 Jan 2023 20:38:45 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> 
> > The only case where we might do that AFAIR is when
> > `inhibit-automatic-native-compilation' is used.  This was the infamous
> > mechanism that was installed by Lars where, if I'm not wrong, we are
> > supposed to compile a trampoline, load it, and remove it to pretend we
> > didn't compiled anything :x
> 
> OK, this seems to be what is happening here, because we compile the
> trampoline to a temporary directory.  Otherwise, I don't see why we
> would do that, and why we would delete a trampoline we just compiled.

Actually, I don't think I see where we delete the trampoline that we
generated when inhibit-automatic-native-compilation is non-nil.  Can
you point me to the code which does that?

bug#60996: 29.0.60; Native compile fails to remove temp file for trampoline

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> Cc: andrewjmoreton@gmail.com, 60996@debbugs.gnu.org
>> Date: Thu, 26 Jan 2023 20:38:45 +0200
>> From: Eli Zaretskii <eliz@gnu.org>
>> 
>> > The only case where we might do that AFAIR is when
>> > `inhibit-automatic-native-compilation' is used.  This was the infamous
>> > mechanism that was installed by Lars where, if I'm not wrong, we are
>> > supposed to compile a trampoline, load it, and remove it to pretend we
>> > didn't compiled anything :x
>> 
>> OK, this seems to be what is happening here, because we compile the
>> trampoline to a temporary directory.  Otherwise, I don't see why we
>> would do that, and why we would delete a trampoline we just compiled.
>
> Actually, I don't think I see where we delete the trampoline that we
> generated when inhibit-automatic-native-compilation is non-nil.  Can
> you point me to the code which does that?

Sure, the code behind this mechanism is not straightforward and took me
a bit to decipher it as well yesterday.

In `comp-trampoline-compile' when `inhibit-automatic-native-compilation'
is not nil we invoke `comp--native-compile' with the `output' parameter
set to null.

`comp--native-compile' after compiling does two things:

1- If the compilation input was a file returns the .eln filename

2- If the input was a lambda (the case for trampolines) it does return
the compiled subr.  To do that it preforms a load of the eln before
returning.

So in general what `comp--native-compile' returns is:

              (if (stringp function-or-file)
                   data
                 ;; So we return the compiled function.
                 (native-elisp-load data)))

But at the end of `comp--native-compile' this when was added

(when (and (not (stringp function-or-file))
                      (not output)
                      comp-ctxt
                      (comp-ctxt-output comp-ctxt)
                      (file-exists-p (comp-ctxt-output comp-ctxt)))
             (delete-file (comp-ctxt-output comp-ctxt)))


Took me a while to actually realize this is the unwindform of an
enormous `unwind-protect'.  Anyway this is the code that when `output'
is null (the case for trampolines) tries to performs the file
deletetion.

Best Regards

  Andrea

bug#60996: 29.0.60; Native compile fails to remove temp file for trampoline

[Andy Moreton]

On Fri 27 Jan 2023, Andrea Corallo wrote:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> Cc: andrewjmoreton@gmail.com, 60996@debbugs.gnu.org
>>> Date: Thu, 26 Jan 2023 20:38:45 +0200
>>> From: Eli Zaretskii <eliz@gnu.org>
>>> 
>>> > The only case where we might do that AFAIR is when
>>> > `inhibit-automatic-native-compilation' is used.  This was the infamous
>>> > mechanism that was installed by Lars where, if I'm not wrong, we are
>>> > supposed to compile a trampoline, load it, and remove it to pretend we
>>> > didn't compiled anything :x
>>> 
>>> OK, this seems to be what is happening here, because we compile the
>>> trampoline to a temporary directory.  Otherwise, I don't see why we
>>> would do that, and why we would delete a trampoline we just compiled.

Sorry that this bug report has consumed so much of everyone's time.
After a while where I could not reproduce the problem, a new rebuild on
master provoked it again.

From more experimentation, the recipe seems to be:
1) Delete the "subr-trampoline-*.eln" files from the eln-cache dir
2) Start emacs with inhibit-automatic-native-compilation non-nil


>> Actually, I don't think I see where we delete the trampoline that we
>> generated when inhibit-automatic-native-compilation is non-nil.  Can
>> you point me to the code which does that?
>
> Sure, the code behind this mechanism is not straightforward and took me
> a bit to decipher it as well yesterday.

Perhaps an opportunity to refactor it at some point, to make it easier
for future maintainers to reason about this code.

> In `comp-trampoline-compile' when `inhibit-automatic-native-compilation'
> is not nil we invoke `comp--native-compile' with the `output' parameter
> set to null.
>
> `comp--native-compile' after compiling does two things:
>
> 1- If the compilation input was a file returns the .eln filename
>
> 2- If the input was a lambda (the case for trampolines) it does return
> the compiled subr.  To do that it preforms a load of the eln before
> returning.
>
> So in general what `comp--native-compile' returns is:
>
>               (if (stringp function-or-file)
>                    data
>                  ;; So we return the compiled function.
>                  (native-elisp-load data)))
>
> But at the end of `comp--native-compile' this when was added
>
> (when (and (not (stringp function-or-file))
>                       (not output)
>                       comp-ctxt
>                       (comp-ctxt-output comp-ctxt)
>                       (file-exists-p (comp-ctxt-output comp-ctxt)))
>              (delete-file (comp-ctxt-output comp-ctxt)))
>
>
> Took me a while to actually realize this is the unwindform of an
> enormous `unwind-protect'.  Anyway this is the code that when `output'
> is null (the case for trampolines) tries to performs the file
> deletetion.

I think you have identified the cause of this issue - as the .eln has
been loaded, the delete-file will never succeed on Windows.

I know Eli is not a fan of inhibit-automatic-native-compilation, but I
think there is a place for it. Users may want to use precompiled .eln
files but not waste effort on native compiling other code.

If this error can be suppressed such that the only effect is for a temp
file to be left around, that is a reasonable solution to this.

    AndyM

bug#60996: 29.0.60; Native compile fails to remove temp file for trampoline

[Eli Zaretskii]

> From: Andy Moreton <andrewjmoreton@gmail.com>
> Date: Sat, 28 Jan 2023 21:15:02 +0000
> 
> I know Eli is not a fan of inhibit-automatic-native-compilation, but I
> think there is a place for it. Users may want to use precompiled .eln
> files but not waste effort on native compiling other code.
> 
> If this error can be suppressed such that the only effect is for a temp
> file to be left around, that is a reasonable solution to this.

What I dislike is that inhibit-automatic-native-compilation promises
not to start compilation, but it actually does start compilation, for
trampolines, and writes them to a directory we invent behind user's
back.  What I'd like to do is to go to the previous situation, where
compilation of trampolines was disabled at startup if
native-compilation is not available (something that happens on Windows
only), and leave the rest to the user: if they want the trampolines to
be compiled, but not written to eln-cache, let them tweak
native-comp-eln-load-path to direct trampolines to another place,
whether the temporary directory or any other directory.

bug#60996: 29.0.60; Native compile fails to remove temp file for trampoline

[Eli Zaretskii]

> Cc: 60996@debbugs.gnu.org
> Date: Sun, 29 Jan 2023 09:01:16 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> 
> and leave the rest to the user: if they want the trampolines to
> be compiled, but not written to eln-cache, let them tweak
> native-comp-eln-load-path to direct trampolines to another place,
> whether the temporary directory or any other directory.

Alternatively, we could modify comp-enable-subr-trampolines to be able
to have a string value, which would then be a directory to which to
write the trampolines.

bug#60996: 29.0.60; Native compile fails to remove temp file for trampoline

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> Cc: 60996@debbugs.gnu.org
>> Date: Sun, 29 Jan 2023 09:01:16 +0200
>> From: Eli Zaretskii <eliz@gnu.org>
>> 
>> and leave the rest to the user: if they want the trampolines to
>> be compiled, but not written to eln-cache, let them tweak
>> native-comp-eln-load-path to direct trampolines to another place,
>> whether the temporary directory or any other directory.
>
> Alternatively, we could modify comp-enable-subr-trampolines to be able
> to have a string value, which would then be a directory to which to
> write the trampolines.

Yes please, I'm on the same line.

  Andrea

bug#60996: 29.0.60; Native compile fails to remove temp file for trampoline

[Eli Zaretskii]

> From: Andy Moreton <andrewjmoreton@gmail.com>
> Date: Sat, 28 Jan 2023 21:15:02 +0000
> 
> If this error can be suppressed such that the only effect is for a temp
> file to be left around, that is a reasonable solution to this.

I've now done so, please see if your scenario now silently fails,
leaving the temporary compiled trampolines in %TEMP%.

bug#60996: 29.0.60; Native compile fails to remove temp file for trampoline

[Andy Moreton]

On Sun 29 Jan 2023, Eli Zaretskii wrote:

>> From: Andy Moreton <andrewjmoreton@gmail.com>
>> Date: Sat, 28 Jan 2023 21:15:02 +0000
>> 
>> If this error can be suppressed such that the only effect is for a temp
>> file to be left around, that is a reasonable solution to this.
>
> I've now done so, please see if your scenario now silently fails,
> leaving the temporary compiled trampolines in %TEMP%.

Yes, that fixes the problem on emacs-29. The problem still exists on
master (which does not yet have your patch), so all looks good.

Your commit message said:

  Fix spurious errors on Windows when deleting temporary *.eln files

I think that is a mischaracterisation: the errors are not spurious, and
are documented by Microsoft. The behaviour is different to POSIX, but
that makes it different, not spurious.

    AndyM

bug#60996: 29.0.60; Native compile fails to remove temp file for trampoline

[Eli Zaretskii]

> From: Andy Moreton <andrewjmoreton@gmail.com>
> Date: Sun, 29 Jan 2023 11:37:09 +0000
> 
> Your commit message said:
> 
>   Fix spurious errors on Windows when deleting temporary *.eln files
> 
> I think that is a mischaracterisation: the errors are not spurious, and
> are documented by Microsoft. The behaviour is different to POSIX, but
> that makes it different, not spurious.

They are spurious from the user POV.

Anyway, the commit message is now carved in stone and cannot be
changed.

bug#60996: 29.0.60; Native compile fails to remove temp file for trampoline

[Eli Zaretskii]

Closing.

bug#60996: 29.0.60; Native compile fails to remove temp file for trampoline

[Andy Moreton]

[-- Attachment #1: Type: text/plain, Size: 2241 bytes --]

On Sun 22 Jan 2023, Eli Zaretskii wrote:

>> Date: Sat, 21 Jan 2023 22:12:10 +0000
>> From: Andy Moreton <andrewjmoreton@gmail.com>
>> 
>> Recently emacs 29 (and master) has started showing an error and
>> backtrace during startup:
>> 
>> Debugger entered--Lisp error: (permission-denied "Removing old name" 
>> "Permission denied" "c:/Users/ajm/AppData/Local/Temp/comp-lambda-MTAMbr...")
>> delete-file("c:/Users/ajm/AppData/Local/Temp/comp-lambda-MTAMbr...")
>
> We need a reproducible recipe to investigate this, or results of such
> investigation by you: which code has the file open when we try
> deleting it, and why that other code has it open?
>
> For a recipe, it should be enough to present a minimal init file which
> causes the problem (but pleased make it really minimal: as few lines
> as strictly needed for reproduction)

Agreed, but this appear to be tiing sensitive, so a minimised reproducer
may take a while to reduce down from my init file.

> Btw, "comp-lambda-MTAMbr..." seems to tell that it's some part of
> comp.el, which sounds strange: comp.el is supposed to be
> natively-compiled during the build, and that includes the trampolines
> for it.  Hmm...

From more expermenting with a debug build in gdb, it seem to be related
to this code in native-elisp-load:

      /* If in this session there was ever a file loaded with this
	 name, rename it before loading, to make sure we always get a
	 new handle!  */
      Lisp_Object tmp_filename =
	Fmake_temp_file_internal (filename, Qnil, build_string (".eln.tmp"),
				  Qnil);
      if (NILP (Ffile_writable_p (tmp_filename)))
	comp_u->handle = dynlib_open_for_eln (SSDATA (encoded_filename));
      else
	{
	  Frename_file (filename, tmp_filename, Qt);
	  comp_u->handle = dynlib_open_for_eln (SSDATA (ENCODE_FILE (tmp_filename)));
	  Frename_file (tmp_filename, filename, Qnil);
	}

The renaming results in the ".eln.tmp" suffixed name having an open
handle. That handle is still open when other code tries to delete
the renamed ".eln" file, which fails.

The attached .csv file shows filesystem activity captured by Process
Monitor for the relevant "...\comp-lambda-*" temp files during emacs
startup. (It may be easier to read if imported into a spreadsheet).


[-- Attachment #2: filesystem activity --]
[-- Type: text/plain, Size: 19632 bytes --]

"Time of Day","Process Name","PID","Operation","Path","Result","Detail"
"14:15:21.8965246","emacs.exe","6452","CreateFile","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7.eln","SUCCESS","Desired Access: Generic Read/Write, Disposition: Create, Options: Synchronous IO Non-Alert, Non-Directory File, Attributes: N, ShareMode: Read, Write, AllocationSize: 0, OpenResult: Created"
"14:15:21.8968228","emacs.exe","6452","CloseFile","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7.eln","SUCCESS",""
"14:15:24.4141153","emacs.exe","7976","CreateFile","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7vca7Uc.eln.tmp","SUCCESS","Desired Access: Generic Read/Write, Disposition: Create, Options: Synchronous IO Non-Alert, Non-Directory File, Attributes: N, ShareMode: Read, Write, AllocationSize: 0, OpenResult: Created"
"14:15:24.4143658","emacs.exe","7976","CloseFile","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7vca7Uc.eln.tmp","SUCCESS",""
"14:15:24.8245971","emacs.exe","7976","CreateFile","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7vca7Uc.eln.tmp","SUCCESS","Desired Access: Generic Write, Read Attributes, Disposition: OverwriteIf, Options: Synchronous IO Non-Alert, Non-Directory File, Attributes: N, ShareMode: Read, Write, AllocationSize: 0, OpenResult: Overwritten"
"14:15:24.8248010","emacs.exe","7976","WriteFile","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7vca7Uc.eln.tmp","SUCCESS","Offset: 0, Length: 4,096, Priority: Normal"
"14:15:24.8250025","emacs.exe","7976","WriteFile","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7vca7Uc.eln.tmp","SUCCESS","Offset: 4,096, Length: 4,096, Priority: Normal"
"14:15:24.8251398","emacs.exe","7976","WriteFile","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7vca7Uc.eln.tmp","SUCCESS","Offset: 8,192, Length: 4,096, Priority: Normal"
"14:15:24.8252491","emacs.exe","7976","WriteFile","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7vca7Uc.eln.tmp","SUCCESS","Offset: 12,288, Length: 4,096"
"14:15:24.8253186","emacs.exe","7976","WriteFile","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7vca7Uc.eln.tmp","SUCCESS","Offset: 16,384, Length: 4,096"
"14:15:24.8253869","emacs.exe","7976","WriteFile","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7vca7Uc.eln.tmp","SUCCESS","Offset: 20,480, Length: 4,096"
"14:15:24.8254553","emacs.exe","7976","WriteFile","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7vca7Uc.eln.tmp","SUCCESS","Offset: 24,576, Length: 4,096"
"14:15:24.8255238","emacs.exe","7976","WriteFile","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7vca7Uc.eln.tmp","SUCCESS","Offset: 28,672, Length: 4,096"
"14:15:24.8256255","emacs.exe","7976","WriteFile","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7vca7Uc.eln.tmp","SUCCESS","Offset: 32,768, Length: 4,096, Priority: Normal"
"14:15:24.8257619","emacs.exe","7976","WriteFile","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7vca7Uc.eln.tmp","SUCCESS","Offset: 36,864, Length: 4,096, Priority: Normal"
"14:15:24.8259024","emacs.exe","7976","WriteFile","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7vca7Uc.eln.tmp","SUCCESS","Offset: 40,960, Length: 4,096, Priority: Normal"
"14:15:24.8260399","emacs.exe","7976","WriteFile","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7vca7Uc.eln.tmp","SUCCESS","Offset: 45,056, Length: 4,096, Priority: Normal"
"14:15:24.8261815","emacs.exe","7976","WriteFile","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7vca7Uc.eln.tmp","SUCCESS","Offset: 49,152, Length: 4,096, Priority: Normal"
"14:15:24.8262978","emacs.exe","7976","WriteFile","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7vca7Uc.eln.tmp","SUCCESS","Offset: 53,248, Length: 4,096"
"14:15:24.8263671","emacs.exe","7976","WriteFile","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7vca7Uc.eln.tmp","SUCCESS","Offset: 57,344, Length: 4,096"
"14:15:24.8264377","emacs.exe","7976","WriteFile","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7vca7Uc.eln.tmp","SUCCESS","Offset: 61,440, Length: 4,096"
"14:15:24.8265084","emacs.exe","7976","WriteFile","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7vca7Uc.eln.tmp","SUCCESS","Offset: 65,536, Length: 4,096"
"14:15:24.8265771","emacs.exe","7976","WriteFile","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7vca7Uc.eln.tmp","SUCCESS","Offset: 69,632, Length: 4,096"
"14:15:24.8266454","emacs.exe","7976","WriteFile","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7vca7Uc.eln.tmp","SUCCESS","Offset: 73,728, Length: 4,096"
"14:15:24.8267138","emacs.exe","7976","WriteFile","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7vca7Uc.eln.tmp","SUCCESS","Offset: 77,824, Length: 4,096"
"14:15:24.8267821","emacs.exe","7976","WriteFile","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7vca7Uc.eln.tmp","SUCCESS","Offset: 81,920, Length: 4,096"
"14:15:24.8268504","emacs.exe","7976","WriteFile","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7vca7Uc.eln.tmp","SUCCESS","Offset: 86,016, Length: 4,096"
"14:15:24.8271045","emacs.exe","7976","WriteFile","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7vca7Uc.eln.tmp","SUCCESS","Offset: 90,112, Length: 2,139"
"14:15:24.8271425","emacs.exe","7976","CloseFile","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7vca7Uc.eln.tmp","SUCCESS",""
"14:15:24.8289434","emacs.exe","7976","CreateFile","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7.eln","SUCCESS","Desired Access: Read Attributes, Disposition: Open, Options: Open Reparse Point, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a, OpenResult: Opened"
"14:15:24.8289892","emacs.exe","7976","QueryBasicInformationFile","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7.eln","SUCCESS","CreationTime: 22/01/2023 14:15:21, LastAccessTime: 22/01/2023 14:15:21, LastWriteTime: 22/01/2023 14:15:21, ChangeTime: 22/01/2023 14:15:21, FileAttributes: A"
"14:15:24.8290252","emacs.exe","7976","CloseFile","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7.eln","SUCCESS",""
"14:15:24.8293467","emacs.exe","7976","CreateFile","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7.eln","SUCCESS","Desired Access: Read Attributes, Disposition: Open, Options: Open Reparse Point, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a, OpenResult: Opened"
"14:15:24.8293902","emacs.exe","7976","QueryBasicInformationFile","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7.eln","SUCCESS","CreationTime: 22/01/2023 14:15:21, LastAccessTime: 22/01/2023 14:15:21, LastWriteTime: 22/01/2023 14:15:21, ChangeTime: 22/01/2023 14:15:21, FileAttributes: A"
"14:15:24.8294253","emacs.exe","7976","CloseFile","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7.eln","SUCCESS",""
"14:15:24.8297901","emacs.exe","7976","CreateFile","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7.eln","SUCCESS","Desired Access: Read Attributes, Disposition: Open, Options: Open Reparse Point, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a, OpenResult: Opened"
"14:15:24.8298328","emacs.exe","7976","QueryBasicInformationFile","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7.eln","SUCCESS","CreationTime: 22/01/2023 14:15:21, LastAccessTime: 22/01/2023 14:15:21, LastWriteTime: 22/01/2023 14:15:21, ChangeTime: 22/01/2023 14:15:21, FileAttributes: A"
"14:15:24.8298674","emacs.exe","7976","CloseFile","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7.eln","SUCCESS",""
"14:15:24.8300493","emacs.exe","7976","CreateFile","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7.eln","SUCCESS","Desired Access: Write Attributes, Synchronize, Disposition: Open, Options: Synchronous IO Non-Alert, Open Reparse Point, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a, OpenResult: Opened"
"14:15:24.8301106","emacs.exe","7976","SetBasicInformationFile","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7.eln","SUCCESS","CreationTime: 01/01/1601 00:00:00, LastAccessTime: 01/01/1601 00:00:00, LastWriteTime: 01/01/1601 00:00:00, ChangeTime: 01/01/1601 00:00:00, FileAttributes: AN"
"14:15:24.8301488","emacs.exe","7976","CloseFile","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7.eln","SUCCESS",""
"14:15:24.8303279","emacs.exe","7976","CreateFile","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7.eln","SUCCESS","Desired Access: Read Attributes, Delete, Disposition: Open, Options: Non-Directory File, Open Reparse Point, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a, OpenResult: Opened"
"14:15:24.8303853","emacs.exe","7976","QueryAttributeTagFile","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7.eln","SUCCESS","Attributes: A, ReparseTag: 0x0"
"14:15:24.8304243","emacs.exe","7976","SetDispositionInformationEx","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7.eln","SUCCESS","Flags: FILE_DISPOSITION_DELETE, FILE_DISPOSITION_POSIX_SEMANTICS, FILE_DISPOSITION_FORCE_IMAGE_SECTION_CHECK"
"14:15:24.8304640","emacs.exe","7976","CloseFile","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7.eln","SUCCESS",""
"14:15:24.8309423","emacs.exe","7976","CreateFile","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7.eln","NAME NOT FOUND","Desired Access: Read Attributes, Disposition: Open, Options: Open Reparse Point, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a"
"14:15:24.8312547","emacs.exe","7976","CreateFile","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7.eln","NAME NOT FOUND","Desired Access: Read Attributes, Disposition: Open, Options: Open Reparse Point, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a"
"14:15:24.8316060","emacs.exe","7976","CreateFile","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7vca7Uc.eln.tmp","SUCCESS","Desired Access: Read Attributes, Delete, Synchronize, Disposition: Open, Options: Synchronous IO Non-Alert, Attributes: n/a, ShareMode: None, AllocationSize: n/a, OpenResult: Opened"
"14:15:24.8316786","emacs.exe","7976","QueryBasicInformationFile","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7vca7Uc.eln.tmp","SUCCESS","CreationTime: 22/01/2023 14:15:24, LastAccessTime: 22/01/2023 14:15:24, LastWriteTime: 22/01/2023 14:15:24, ChangeTime: 22/01/2023 14:15:24, FileAttributes: A"
"14:15:24.8321739","emacs.exe","7976","SetRenameInformationFile","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7vca7Uc.eln.tmp","SUCCESS","ReplaceIfExists: False, FileName: C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7.eln"
"14:15:24.8327212","emacs.exe","7976","CloseFile","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7.eln","SUCCESS",""
"14:15:24.8724898","emacs.exe","6452","CreateFile","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7.eln","SUCCESS","Desired Access: Read Attributes, Disposition: Open, Options: Open Reparse Point, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a, OpenResult: Opened"
"14:15:24.8725598","emacs.exe","6452","QueryBasicInformationFile","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7.eln","SUCCESS","CreationTime: 22/01/2023 14:15:21, LastAccessTime: 22/01/2023 14:15:24, LastWriteTime: 22/01/2023 14:15:24, ChangeTime: 22/01/2023 14:15:24, FileAttributes: A"
"14:15:24.8726035","emacs.exe","6452","CloseFile","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7.eln","SUCCESS",""
"14:15:24.8729415","emacs.exe","6452","CreateFile","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7.eln","SUCCESS","Desired Access: Read Attributes, Disposition: Open, Options: Open Reparse Point, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a, OpenResult: Opened"
"14:15:24.8729962","emacs.exe","6452","QueryBasicInformationFile","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7.eln","SUCCESS","CreationTime: 22/01/2023 14:15:21, LastAccessTime: 22/01/2023 14:15:24, LastWriteTime: 22/01/2023 14:15:24, ChangeTime: 22/01/2023 14:15:24, FileAttributes: A"
"14:15:24.8730581","emacs.exe","6452","CloseFile","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7.eln","SUCCESS",""
"14:15:24.8734177","emacs.exe","6452","CreateFile","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7.eln","SUCCESS","Desired Access: Read Attributes, Disposition: Open, Options: Open Reparse Point, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a, OpenResult: Opened"
"14:15:24.8734703","emacs.exe","6452","QueryBasicInformationFile","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7.eln","SUCCESS","CreationTime: 22/01/2023 14:15:21, LastAccessTime: 22/01/2023 14:15:24, LastWriteTime: 22/01/2023 14:15:24, ChangeTime: 22/01/2023 14:15:24, FileAttributes: A"
"14:15:24.8735148","emacs.exe","6452","CloseFile","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7.eln","SUCCESS",""
"14:15:24.8737115","emacs.exe","6452","CreateFile","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7.eln","SUCCESS","Desired Access: Read Data/List Directory, Execute/Traverse, Synchronize, Disposition: Open, Options: Synchronous IO Non-Alert, Non-Directory File, Attributes: n/a, ShareMode: Read, Delete, AllocationSize: n/a, OpenResult: Opened"
"14:15:24.8737860","emacs.exe","6452","CreateFileMapping","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7.eln","FILE LOCKED WITH ONLY READERS","SyncType: SyncTypeCreateSection, PageProtection: PAGE_EXECUTE_READWRITE|PAGE_NOCACHE"
"14:15:24.8738324","emacs.exe","6452","QueryStandardInformationFile","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7.eln","SUCCESS","AllocationSize: 94,208, EndOfFile: 92,251, NumberOfLinks: 1, DeletePending: False, Directory: False"
"14:15:24.8740966","emacs.exe","6452","CreateFileMapping","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7.eln","SUCCESS","SyncType: SyncTypeOther"
"14:15:24.8742065","emacs.exe","6452","Load Image","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7.eln","SUCCESS","Image Base: 0x7ffaacb50000, Image Size: 0x21000"
"14:15:24.8743184","emacs.exe","6452","CloseFile","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7.eln","SUCCESS",""
"14:15:24.8747892","emacs.exe","6452","CreateFile","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7.eln","SUCCESS","Desired Access: Read Attributes, Disposition: Open, Options: Open Reparse Point, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a, OpenResult: Opened"
"14:15:24.8748127","emacs.exe","6452","QueryBasicInformationFile","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7.eln","SUCCESS","CreationTime: 22/01/2023 14:15:21, LastAccessTime: 22/01/2023 14:15:24, LastWriteTime: 22/01/2023 14:15:24, ChangeTime: 22/01/2023 14:15:24, FileAttributes: A"
"14:15:24.8748286","emacs.exe","6452","CloseFile","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7.eln","SUCCESS",""
"14:15:24.8751046","emacs.exe","6452","CreateFile","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7.eln","SUCCESS","Desired Access: Read Attributes, Disposition: Open, Options: Open Reparse Point, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a, OpenResult: Opened"
"14:15:24.8751292","emacs.exe","6452","QueryBasicInformationFile","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7.eln","SUCCESS","CreationTime: 22/01/2023 14:15:21, LastAccessTime: 22/01/2023 14:15:24, LastWriteTime: 22/01/2023 14:15:24, ChangeTime: 22/01/2023 14:15:24, FileAttributes: A"
"14:15:24.8751445","emacs.exe","6452","CloseFile","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7.eln","SUCCESS",""
"14:15:24.8754961","emacs.exe","6452","CreateFile","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7.eln","SUCCESS","Desired Access: Read Attributes, Disposition: Open, Options: Open Reparse Point, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a, OpenResult: Opened"
"14:15:24.8755282","emacs.exe","6452","QueryBasicInformationFile","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7.eln","SUCCESS","CreationTime: 22/01/2023 14:15:21, LastAccessTime: 22/01/2023 14:15:24, LastWriteTime: 22/01/2023 14:15:24, ChangeTime: 22/01/2023 14:15:24, FileAttributes: A"
"14:15:24.8755437","emacs.exe","6452","CloseFile","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7.eln","SUCCESS",""
"14:15:24.8757797","emacs.exe","6452","CreateFile","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7.eln","SUCCESS","Desired Access: Read Attributes, Disposition: Open, Options: Open Reparse Point, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a, OpenResult: Opened"
"14:15:24.8758013","emacs.exe","6452","QueryBasicInformationFile","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7.eln","SUCCESS","CreationTime: 22/01/2023 14:15:21, LastAccessTime: 22/01/2023 14:15:24, LastWriteTime: 22/01/2023 14:15:24, ChangeTime: 22/01/2023 14:15:24, FileAttributes: A"
"14:15:24.8758154","emacs.exe","6452","CloseFile","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7.eln","SUCCESS",""
"14:15:24.8764948","emacs.exe","6452","CreateFile","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7.eln","SUCCESS","Desired Access: Read Attributes, Disposition: Open, Options: Open Reparse Point, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a, OpenResult: Opened"
"14:15:24.8765354","emacs.exe","6452","QueryBasicInformationFile","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7.eln","SUCCESS","CreationTime: 22/01/2023 14:15:21, LastAccessTime: 22/01/2023 14:15:24, LastWriteTime: 22/01/2023 14:15:24, ChangeTime: 22/01/2023 14:15:24, FileAttributes: A"
"14:15:24.8765646","emacs.exe","6452","CloseFile","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7.eln","SUCCESS",""
"14:15:24.8768883","emacs.exe","6452","CreateFile","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7.eln","SUCCESS","Desired Access: Write Attributes, Synchronize, Disposition: Open, Options: Synchronous IO Non-Alert, Open Reparse Point, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a, OpenResult: Opened"
"14:15:24.8769622","emacs.exe","6452","SetBasicInformationFile","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7.eln","SUCCESS","CreationTime: 01/01/1601 00:00:00, LastAccessTime: 01/01/1601 00:00:00, LastWriteTime: 01/01/1601 00:00:00, ChangeTime: 01/01/1601 00:00:00, FileAttributes: AN"
"14:15:24.8770113","emacs.exe","6452","CloseFile","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7.eln","SUCCESS",""
"14:15:24.8772176","emacs.exe","6452","CreateFile","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7.eln","SUCCESS","Desired Access: Read Attributes, Delete, Disposition: Open, Options: Non-Directory File, Open Reparse Point, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a, OpenResult: Opened"
"14:15:24.8772934","emacs.exe","6452","QueryAttributeTagFile","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7.eln","SUCCESS","Attributes: A, ReparseTag: 0x0"
"14:15:24.8773402","emacs.exe","6452","SetDispositionInformationEx","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7.eln","CANNOT DELETE","Flags: FILE_DISPOSITION_DELETE, FILE_DISPOSITION_POSIX_SEMANTICS, FILE_DISPOSITION_FORCE_IMAGE_SECTION_CHECK"
"14:15:24.8774207","emacs.exe","6452","CloseFile","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7.eln","SUCCESS",""
"14:15:24.8777610","emacs.exe","6452","CreateFile","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7.eln","SUCCESS","Desired Access: Read Attributes, Disposition: Open, Options: Open Reparse Point, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a, OpenResult: Opened"
"14:15:24.8778135","emacs.exe","6452","QueryBasicInformationFile","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7.eln","SUCCESS","CreationTime: 22/01/2023 14:15:21, LastAccessTime: 22/01/2023 14:15:24, LastWriteTime: 22/01/2023 14:15:24, ChangeTime: 22/01/2023 14:15:24, FileAttributes: A"
"14:15:24.8778576","emacs.exe","6452","CloseFile","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7.eln","SUCCESS",""

[-- Attachment #3: Type: text/plain, Size: 1236 bytes --]



>> Tracing execution of emacs with Process Explorer shows that the temp
>> file used to native compile trampolines is opened and closed repeatedly
>> by emacs, and at the point of the backtrace is still open by the same
>> emacs process.
>
> We need to know which code opened it the last time and didn't close
> it.  Can you figure that out?  All the files Emacs opens go through 2
> functions in w32.c: sys_fopen and sys_open, so by running with 2
> breakpoints there that show the backtrace and continue, you should be
> able to see the culprit, and we can then take it from there.

As noted above, I think it is the code in native-elisp-load interacting
woth the way that the files are renamed (and if those operations have
specified posix semantics or not). 

>> I am not sure excactly when this issue started, but I did not see it in
>> emacs-29 or master bootstrapped before this month.
>
> Could be because we now compile trampolines differently (to avoid the
> danger of the "fork bomb" due to recursive compilation of
> trampolines by async subprocesses).

Any idea when those changes were committed, and which changesets ? IT
doesn't apper obvios from the commit history.

> Andrea, any suggestions or comments?
>
> Thanks.

bug#60996: 29.0.60; Native compile fails to remove temp file for trampoline

[Eli Zaretskii]

> From: Andy Moreton <andrewjmoreton@gmail.com>
> Date: Mon, 23 Jan 2023 02:30:43 +0000
> 
> >From more expermenting with a debug build in gdb, it seem to be related
> to this code in native-elisp-load:
> 
>       /* If in this session there was ever a file loaded with this
> 	 name, rename it before loading, to make sure we always get a
> 	 new handle!  */
>       Lisp_Object tmp_filename =
> 	Fmake_temp_file_internal (filename, Qnil, build_string (".eln.tmp"),
> 				  Qnil);
>       if (NILP (Ffile_writable_p (tmp_filename)))
> 	comp_u->handle = dynlib_open_for_eln (SSDATA (encoded_filename));
>       else
> 	{
> 	  Frename_file (filename, tmp_filename, Qt);
> 	  comp_u->handle = dynlib_open_for_eln (SSDATA (ENCODE_FILE (tmp_filename)));
> 	  Frename_file (tmp_filename, filename, Qnil);
> 	}
> 
> The renaming results in the ".eln.tmp" suffixed name having an open
> handle. That handle is still open when other code tries to delete
> the renamed ".eln" file, which fails.

The question is: why is that .eln.tmp file still open when we are
trying to delete it?  I hoped we'd be able to establish that, so that
we could decide what to do about it.  But I don't yet see the
information which would explain why the file is still open.

Do these *.eln.tmp file remain in %TEMP% after Emacs startup?  If they
remain there, then what happens if you exit Emacs and restart it? do
these files remain there or are they deleted?  IOW, if we just ignore
these errors (e.g., by ignore-error), would the problem be fixed, or
will there be left-overs?

> The attached .csv file shows filesystem activity captured by Process
> Monitor for the relevant "...\comp-lambda-*" temp files during emacs
> startup. (It may be easier to read if imported into a spreadsheet).

Maybe I don't understand how to read this, but I cannot find any
traces for the failed deletion of a .eln.tmp file.  The only failure
to delete is this:

> "14:15:24.8773402","emacs.exe","6452","SetDispositionInformationEx","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7.eln","CANNOT DELETE","Flags: FILE_DISPOSITION_DELETE, FILE_DISPOSITION_POSIX_SEMANTICS, FILE_DISPOSITION_FORCE_IMAGE_SECTION_CHECK"

and it names a .eln file, not .eln.tmp.  I also don't see any
RenameFile calls, but maybe they appear under different names in this
captured activity?

> > We need to know which code opened it the last time and didn't close
> > it.  Can you figure that out?  All the files Emacs opens go through 2
> > functions in w32.c: sys_fopen and sys_open, so by running with 2
> > breakpoints there that show the backtrace and continue, you should be
> > able to see the culprit, and we can then take it from there.
> 
> As noted above, I think it is the code in native-elisp-load interacting
> woth the way that the files are renamed (and if those operations have
> specified posix semantics or not). 

I don't think I follow.  The snippet from native-elisp-load you show
doesn't call Fdelete_file, it only renames files, and renaming should
work on MS-Windows like it works on Posix systems in this case.  The
problem, as the error message evidences, is in Fdelete_file.

Frename_file indeed can call Fdelete_file, but rename-file is not in
the backtraces you posted, so I don't think this code is involved
directly.  You seem to say this code somehow "interferes" with
deleting files, but how does it interfere with that?  Also, AFAIK this
part of comp.c didn't change since Emacs 28, so whatever caused the
problem is not this code directly, but something else.

IOW, I'd still like to understand in more detail what changed lately
that causes these errors in your case.

> >> I am not sure excactly when this issue started, but I did not see it in
> >> emacs-29 or master bootstrapped before this month.
> >
> > Could be because we now compile trampolines differently (to avoid the
> > danger of the "fork bomb" due to recursive compilation of
> > trampolines by async subprocesses).
> 
> Any idea when those changes were committed, and which changesets ? IT
> doesn't apper obvios from the commit history.

See commit 1a8015b837.  Maybe also commit 5fec9182db.

Thanks.

bug#60996: 29.0.60; Native compile fails to remove temp file for trampoline

[Andy Moreton]

On Mon 23 Jan 2023, Eli Zaretskii wrote:

>> From: Andy Moreton <andrewjmoreton@gmail.com>
>> Date: Mon, 23 Jan 2023 02:30:43 +0000
>> 
>> >From more expermenting with a debug build in gdb, it seem to be related
>> to this code in native-elisp-load:
>> 
>>       /* If in this session there was ever a file loaded with this
>> 	 name, rename it before loading, to make sure we always get a
>> 	 new handle!  */
>>       Lisp_Object tmp_filename =
>> 	Fmake_temp_file_internal (filename, Qnil, build_string (".eln.tmp"),
>> 				  Qnil);
>>       if (NILP (Ffile_writable_p (tmp_filename)))
>> 	comp_u->handle = dynlib_open_for_eln (SSDATA (encoded_filename));
>>       else
>> 	{
>> 	  Frename_file (filename, tmp_filename, Qt);
>> 	  comp_u->handle = dynlib_open_for_eln (SSDATA (ENCODE_FILE (tmp_filename)));
>> 	  Frename_file (tmp_filename, filename, Qnil);
>> 	}
>> 
>> The renaming results in the ".eln.tmp" suffixed name having an open
>> handle. That handle is still open when other code tries to delete
>> the renamed ".eln" file, which fails.
>
> The question is: why is that .eln.tmp file still open when we are
> trying to delete it?  I hoped we'd be able to establish that, so that
> we could decide what to do about it.  But I don't yet see the
> information which would explain why the file is still open.

The file handle is returned from dynlib_open_for_eln (which came from a
LoadLibrary call in dylib_open). More below.

> Do these *.eln.tmp file remain in %TEMP% after Emacs startup?  If they
> remain there, then what happens if you exit Emacs and restart it? do
> these files remain there or are they deleted?  IOW, if we just ignore
> these errors (e.g., by ignore-error), would the problem be fixed, or
> will there be left-overs?

The ".eln.tmp" suffixed files are only renamed temporarily, and the files
only fleetingly have that name. The ".eln" suffixed file names persist
after the failed delete, so will accumulate unless removed manually.

>> The attached .csv file shows filesystem activity captured by Process
>> Monitor for the relevant "...\comp-lambda-*" temp files during emacs
>> startup. (It may be easier to read if imported into a spreadsheet).
>
> Maybe I don't understand how to read this, but I cannot find any
> traces for the failed deletion of a .eln.tmp file.  The only failure
> to delete is this:
>
>> "14:15:24.8773402","emacs.exe","6452","SetDispositionInformationEx","C:\Users\ajm\AppData\Local\Temp\comp-lambda-SGvFx7.eln","CANNOT
>> DELETE","Flags: FILE_DISPOSITION_DELETE, FILE_DISPOSITION_POSIX_SEMANTICS,
>> FILE_DISPOSITION_FORCE_IMAGE_SECTION_CHECK"

That line is the failed delete-file (for "comp-lambda-SGvFx7.eln") at the
end of comp--native-compile.

From the Microsoft kernel docs at:
https://learn.microsoft.com/en-us/windows-hardware/drivers/ddi/ntddk/ns-ntddk-_file_disposition_information_ex

  "A return value of STATUS_CANNOT_DELETE indicates that either the file
  is read-only, or there is an existing mapped view to the file."

The earlier lines in the trace with CreateFileMapping and Load Image for
"comp-lambda-SGvFx7.eln" to suggest that existing mapping created from
dynlib_open (by LoadLibrary) may be the problem.

>> Any idea when those changes were committed, and which changesets ? IT
>> doesn't apper obvios from the commit history.
>
> See commit 1a8015b837.  Maybe also commit 5fec9182db.

Thanks for the additional details.

   AndyM

bug#60996: 29.0.60; Native compile fails to remove temp file for trampoline

[Eli Zaretskii]

> From: Andy Moreton <andrewjmoreton@gmail.com>
> Date: Tue, 24 Jan 2023 01:18:01 +0000
> 
> >>       /* If in this session there was ever a file loaded with this
> >> 	 name, rename it before loading, to make sure we always get a
> >> 	 new handle!  */
> >>       Lisp_Object tmp_filename =
> >> 	Fmake_temp_file_internal (filename, Qnil, build_string (".eln.tmp"),
> >> 				  Qnil);
> >>       if (NILP (Ffile_writable_p (tmp_filename)))
> >> 	comp_u->handle = dynlib_open_for_eln (SSDATA (encoded_filename));
> >>       else
> >> 	{
> >> 	  Frename_file (filename, tmp_filename, Qt);
> >> 	  comp_u->handle = dynlib_open_for_eln (SSDATA (ENCODE_FILE (tmp_filename)));
> >> 	  Frename_file (tmp_filename, filename, Qnil);
> >> 	}
> >> 
> >> The renaming results in the ".eln.tmp" suffixed name having an open
> >> handle. That handle is still open when other code tries to delete
> >> the renamed ".eln" file, which fails.
> >
> > The question is: why is that .eln.tmp file still open when we are
> > trying to delete it?  I hoped we'd be able to establish that, so that
> > we could decide what to do about it.  But I don't yet see the
> > information which would explain why the file is still open.
> 
> The file handle is returned from dynlib_open_for_eln (which came from a
> LoadLibrary call in dylib_open). More below.
> 
> > Do these *.eln.tmp file remain in %TEMP% after Emacs startup?  If they
> > remain there, then what happens if you exit Emacs and restart it? do
> > these files remain there or are they deleted?  IOW, if we just ignore
> > these errors (e.g., by ignore-error), would the problem be fixed, or
> > will there be left-overs?
> 
> The ".eln.tmp" suffixed files are only renamed temporarily, and the files
> only fleetingly have that name. The ".eln" suffixed file names persist
> after the failed delete, so will accumulate unless removed manually.

So we are trying to delete a .eln file that is still being loaded into
this same Emacs session?  That sounds like a bug to me.

Andrea, could you see if this situation can happen?  IIUC, it should
also happen on Unix when we compile a trampoline on the flight
(perhaps when a previous outdated version of the trampoline exists?),
except that on Unix the deletion silently succeeds and the file is
removed when the session ends, which doesn't happen on Windows.  On
Windows, we need to unload the .eln file first, but can we do that
with a trampoline?

But first, I'd like to understand whether indeed it could happen that
we are deleting a temporary .eln file for a trampoline we just
compiled when we are still using that .eln file.  If this happens,
we'd need to find a way to delay the deletion till Emacs is about to
exit or something.

bug#60996: 29.0.60; Native compile fails to remove temp file for trampoline

[Eli Zaretskii]

> Cc: 60996@debbugs.gnu.org
> Date: Tue, 24 Jan 2023 14:56:00 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> 
> But first, I'd like to understand whether indeed it could happen that
> we are deleting a temporary .eln file for a trampoline we just
> compiled when we are still using that .eln file.  If this happens,
> we'd need to find a way to delay the deletion till Emacs is about to
> exit or something.

I keep looking at the backtraces you provided, this:

> Debugger entered--Lisp error: (permission-denied "Removing old name" 
> "Permission denied" "c:/Users/ajm/AppData/Local/Temp/comp-lambda-MTAMbr...")
> delete-file("c:/Users/ajm/AppData/Local/Temp/comp-lambda-MTAMbr...")
>    #f(compiled-function () #<bytecode 0xb2aa5c0c9e24501>)()
>    comp--native-compile((lambda (arg0 &optional arg1 arg2 arg3) (let ((f 
> #'format-mode-line)) (funcall f arg0 arg1 arg2 arg3))) nil nil)
>    comp-trampoline-compile(format-mode-line)
>    comp-subr-trampoline-install(format-mode-line)
>    advice--add-function(:around (#f(compiled-function () #<bytecode 
> 0x3c5d8019df7e91>) . #f(compiled-function (gv--val) #<bytecode 
> 0x8fdecbbba7cb3c2>)) delight--format-mode-line nil)
>    advice-add(format-mode-line :around delight--format-mode-line)
>    require(delight nil t)

And this:

> delete file("path/to/comp-lambda-XXXXX.eln")
> comp--native-compile((lambda (...) ...))
> comp-trampoline-compile(function-name)
> comp-subr-trampoline-install(function-name)
> advice--add-function(...)
> advice-add(function-name ...)

and I still don't have a clear picture regarding which code calls
delete-file and why.  For example, the first backtrace above says that
comp--native-compile calls delete-file via some byte-compiled
function, not directly.  But what is that byte-compiled function? can
you figure that out?

There is a direct call to delete-file in comp--native-compile, but is
that the call which fails?  And if so, why does it fail?  Which code
created the .eln file that comp--native-compile tries to delete?

Andrea, can you please describe in detail what happens, step by step,
when comp-subr-trampoline-install is called like above, to compile a
trampoline for an advised function?  What is that file which
comp--native-compile is deleting at the end?

Thanks.

bug#60996: 29.0.60; Native compile fails to remove temp file for trampoline

[Eli Zaretskii]

> Cc: 60996@debbugs.gnu.org
> Date: Tue, 24 Jan 2023 19:50:27 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> 
> and I still don't have a clear picture regarding which code calls
> delete-file and why.  For example, the first backtrace above says that
> comp--native-compile calls delete-file via some byte-compiled
> function, not directly.  But what is that byte-compiled function? can
> you figure that out?
> 
> There is a direct call to delete-file in comp--native-compile, but is
> that the call which fails?  And if so, why does it fail?  Which code
> created the .eln file that comp--native-compile tries to delete?

With this simple .emacs:

  (require 'delight nil t)

I cannot reproduce these errors from delete-file, although the
trampoline for advice-add in delight.el does get natively-compiled.
After the compilation finishes, I see no left-over *.eln files in my
system temporary directory.  So there's definitely more in your case
than meets the eye, and just getting Emacs to compile a trampoline is
not enough to trigger the problem.

bug#60996: 29.0.60; Native compile fails to remove temp file for trampoline

[Andy Moreton]

On Tue 24 Jan 2023, Eli Zaretskii wrote:

>> Cc: 60996@debbugs.gnu.org
>> Date: Tue, 24 Jan 2023 19:50:27 +0200
>> From: Eli Zaretskii <eliz@gnu.org>
>> 
>> and I still don't have a clear picture regarding which code calls
>> delete-file and why.  For example, the first backtrace above says that
>> comp--native-compile calls delete-file via some byte-compiled
>> function, not directly.  But what is that byte-compiled function? can
>> you figure that out?

I cannot reproduce the problem any more - trying to examine things in
gdb did not produce enlightenment, as the problem seems to be timing
sensitive, so adding breakpoints or tracing in my experiments often
stopped the problem.

>> There is a direct call to delete-file in comp--native-compile, but is
>> that the call which fails?  And if so, why does it fail?  Which code
>> created the .eln file that comp--native-compile tries to delete?

From memory I think that involved compiling a trampoline, and
comp-spill-lap calling one of the comp-spill-lap methods, which was
probably
  (cl-defmethod comp-spill-lap-function ((form list))

I cannot reproduce this any longer as I have updated various software,
so this bug can probably be closed (and reopened later if the problem
becomes visible again).

    AndyM

bug#60996: 29.0.60; Native compile fails to remove temp file for trampoline

[Eli Zaretskii]

> From: Andy Moreton <andrewjmoreton@gmail.com>
> Date: Tue, 24 Jan 2023 22:32:35 +0000
> 
> On Tue 24 Jan 2023, Eli Zaretskii wrote:
> 
> >> There is a direct call to delete-file in comp--native-compile, but is
> >> that the call which fails?  And if so, why does it fail?  Which code
> >> created the .eln file that comp--native-compile tries to delete?
> 
> >From memory I think that involved compiling a trampoline, and
> comp-spill-lap calling one of the comp-spill-lap methods, which was
> probably
>   (cl-defmethod comp-spill-lap-function ((form list))
> 
> I cannot reproduce this any longer as I have updated various software,
> so this bug can probably be closed (and reopened later if the problem
> becomes visible again).

Before we close this, I suggest that you try reproducing again, after
removing the *.eln files from your eln-cache.  Basically, as more and
more *.eln files are produced and deposited there, the lower the
probability of seeing any problems with native compilation, since
Emacs stops compiling stuff because it already has what it needs.

(But if you already tried that and still failed, then OK, let's close
this issue.)

Thanks.

bug#60996: 29.0.60; Native compile fails to remove temp file for trampoline

[Andy Moreton]

On Wed 25 Jan 2023, Eli Zaretskii wrote:

>> From: Andy Moreton <andrewjmoreton@gmail.com>
>> Date: Tue, 24 Jan 2023 22:32:35 +0000
>> 
>> On Tue 24 Jan 2023, Eli Zaretskii wrote:
>> 
>> >> There is a direct call to delete-file in comp--native-compile, but is
>> >> that the call which fails?  And if so, why does it fail?  Which code
>> >> created the .eln file that comp--native-compile tries to delete?
>> 
>> >From memory I think that involved compiling a trampoline, and
>> comp-spill-lap calling one of the comp-spill-lap methods, which was
>> probably
>>   (cl-defmethod comp-spill-lap-function ((form list))
>> 
>> I cannot reproduce this any longer as I have updated various software,
>> so this bug can probably be closed (and reopened later if the problem
>> becomes visible again).
>
> Before we close this, I suggest that you try reproducing again, after
> removing the *.eln files from your eln-cache.  Basically, as more and
> more *.eln files are produced and deposited there, the lower the
> probability of seeing any problems with native compilation, since
> Emacs stops compiling stuff because it already has what it needs.

I tried various combinations of cleaning the .eln cache, bootstrap from
a clean tree etc, and leaving or removing trampoline .eln files from
%TEMP% and did not get a reproducibe effect.
>
> (But if you already tried that and still failed, then OK, let's close
> this issue.)

I also tried running with anti-virus disabled (in case that affected
filesystem activity or timing) without any change.

Please close this, and we can start again if the problem reappears.

    AndyM

bug#60996: 29.0.60; Native compile fails to remove temp file for trampoline

[Eli Zaretskii]

> From: Andy Moreton <andrewjmoreton@gmail.com>
> Date: Wed, 25 Jan 2023 23:49:04 +0000
> 
> On Wed 25 Jan 2023, Eli Zaretskii wrote:
> 
> > Before we close this, I suggest that you try reproducing again, after
> > removing the *.eln files from your eln-cache.  Basically, as more and
> > more *.eln files are produced and deposited there, the lower the
> > probability of seeing any problems with native compilation, since
> > Emacs stops compiling stuff because it already has what it needs.
> 
> I tried various combinations of cleaning the .eln cache, bootstrap from
> a clean tree etc, and leaving or removing trampoline .eln files from
> %TEMP% and did not get a reproducibe effect.
> >
> > (But if you already tried that and still failed, then OK, let's close
> > this issue.)
> 
> I also tried running with anti-virus disabled (in case that affected
> filesystem activity or timing) without any change.
> 
> Please close this, and we can start again if the problem reappears.

OK, done.  Thanks for your efforts in trying to decipher this.

bug#60996: 29.0.60; Native compile fails to remove temp file for trampoline

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> Cc: 60996@debbugs.gnu.org
>> Date: Tue, 24 Jan 2023 14:56:00 +0200
>> From: Eli Zaretskii <eliz@gnu.org>
>> 
>> But first, I'd like to understand whether indeed it could happen that
>> we are deleting a temporary .eln file for a trampoline we just
>> compiled when we are still using that .eln file.  If this happens,
>> we'd need to find a way to delay the deletion till Emacs is about to
>> exit or something.
>
> I keep looking at the backtraces you provided, this:
>
>> Debugger entered--Lisp error: (permission-denied "Removing old name" 
>> "Permission denied" "c:/Users/ajm/AppData/Local/Temp/comp-lambda-MTAMbr...")
>> delete-file("c:/Users/ajm/AppData/Local/Temp/comp-lambda-MTAMbr...")
>>    #f(compiled-function () #<bytecode 0xb2aa5c0c9e24501>)()
>>    comp--native-compile((lambda (arg0 &optional arg1 arg2 arg3) (let ((f 
>> #'format-mode-line)) (funcall f arg0 arg1 arg2 arg3))) nil nil)
>>    comp-trampoline-compile(format-mode-line)
>>    comp-subr-trampoline-install(format-mode-line)
>>    advice--add-function(:around (#f(compiled-function () #<bytecode 
>> 0x3c5d8019df7e91>) . #f(compiled-function (gv--val) #<bytecode 
>> 0x8fdecbbba7cb3c2>)) delight--format-mode-line nil)
>>    advice-add(format-mode-line :around delight--format-mode-line)
>>    require(delight nil t)
>
> And this:
>
>> delete file("path/to/comp-lambda-XXXXX.eln")
>> comp--native-compile((lambda (...) ...))
>> comp-trampoline-compile(function-name)
>> comp-subr-trampoline-install(function-name)
>> advice--add-function(...)
>> advice-add(function-name ...)
>
> and I still don't have a clear picture regarding which code calls
> delete-file and why.  For example, the first backtrace above says that
> comp--native-compile calls delete-file via some byte-compiled
> function, not directly.  But what is that byte-compiled function? can
> you figure that out?
>
> There is a direct call to delete-file in comp--native-compile, but is
> that the call which fails?  And if so, why does it fail?  Which code
> created the .eln file that comp--native-compile tries to delete?
>
> Andrea, can you please describe in detail what happens, step by step,
> when comp-subr-trampoline-install is called like above, to compile a
> trampoline for an advised function?  What is that file which
> comp--native-compile is deleting at the end?

Hi Eli,

comp-subr-trampoline-install just search for an existing trampoline with
comp-trampoline-search, if this is not found is using
comp-trampoline-compile to produce that.  Once the trampoline is found
or created it gets installed.

In this case as the trampoline is not found and so
comp-trampoline-compile is invoking comp--native-compile to produce one.

Please see my other reply into this thread for more, as there might be
more to investigate here.

BR

  Andrea