bug#48141: 28.0.50; Files left over by native compiler

bug#48141: 28.0.50; Files left over by native compiler

[Stefan Monnier]

Package: Emacs
Version: 28.0.50


I have a few files apparently left over from failed native compilations.
They look like `lisp/foo.elcABCDEF` where `ABCDEF` is random
(presumably chosen by `make-temp-file`).

I haven't found a way to reproduce the problem, so I'm not sure exactly
how they got here but maybe we need to take more precautions to remove
them, and maybe we should put them in /tmp?


        Stefan

bug#48141: 28.0.50; Files left over by native compiler

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Sat, 01 May 2021 10:36:22 -0400
> 
> I have a few files apparently left over from failed native compilations.
> They look like `lisp/foo.elcABCDEF` where `ABCDEF` is random
> (presumably chosen by `make-temp-file`).
> 
> I haven't found a way to reproduce the problem, so I'm not sure exactly
> how they got here but maybe we need to take more precautions to remove
> them, and maybe we should put them in /tmp?

These are created by byte compiling, they aren't specific to native
compilations.  (AFAIR, we already had them in /tmp at some point, but
moved away of that for some good reasons that I cannot recall?)

I wouldn't be bothered by them, as failed native compilations will
sooner or later pass from the world...

bug#48141: 28.0.50; Files left over by native compiler

[Glenn Morris]

Stefan Monnier wrote:

> I have a few files apparently left over from failed native compilations.
> They look like `lisp/foo.elcABCDEF` where `ABCDEF` is random
> (presumably chosen by `make-temp-file`).

This is not directly related to native compilation.
This is so creation of .elc files can be atomic (bug#4196).

> I haven't found a way to reproduce the problem, so I'm not sure exactly
> how they got here but maybe we need to take more precautions to remove
> them

I would guess that the only way this can happen is if Emacs dies during
byte-compilation, which is obviously not a normal situation.

> and maybe we should put them in /tmp?

IIUC, the temp files must be on the same partition as the destination
.elc files, else the move would not be atomic.

bug#48141: 28.0.50; Files left over by native compiler

[Lars Ingebrigtsen]

Glenn Morris <rgm@gnu.org> writes:

>> and maybe we should put them in /tmp?
>
> IIUC, the temp files must be on the same partition as the destination
> .elc files, else the move would not be atomic.

Well, we could create them in /tmp, and then move them to
lisp/foo.elcABCDEF, and then move them to lisp/foo.elc -- and it's
unlikely that things go wrong between the first and the second move,
which should lead to fewer of these files being left behind them Emacs
segfaults and stuff when generating the .elc files.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#48141: 28.0.50; Files left over by native compiler

[Gregory Heytings]

>> I have a few files apparently left over from failed native 
>> compilations. They look like `lisp/foo.elcABCDEF` where `ABCDEF` is 
>> random (presumably chosen by `make-temp-file`).
>
> This is not directly related to native compilation. This is so creation 
> of .elc files can be atomic (bug#4196).
>
>> and maybe we should put them in /tmp?
>
> IIUC, the temp files must be on the same partition as the destination 
> .elc files, else the move would not be atomic.
>

Given that requirement (atomicity) and its implication (same partition), 
would it not be better to create these temporary files in a 'tmp' 
subdirectory of their target directory (e.g. 'lisp/tmp' for el files in 
'lisp' and 'lisp/calc/tmp' for el files in 'lisp/calc') instead of using 
random filenames in the target directory itself?

bug#48141: 28.0.50; Files left over by native compiler

[Eli Zaretskii]

> Date: Mon, 03 May 2021 08:42:08 +0000
> From: Gregory Heytings <gregory@heytings.org>
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>, 48141@debbugs.gnu.org
> 
> Given that requirement (atomicity) and its implication (same partition), 
> would it not be better to create these temporary files in a 'tmp' 
> subdirectory of their target directory (e.g. 'lisp/tmp' for el files in 
> 'lisp' and 'lisp/calc/tmp' for el files in 'lisp/calc') instead of using 
> random filenames in the target directory itself?

Since they will be in the tree anyway, why does it matter in what
subdirectory they will be deposited?

Using a subdirectory has a disadvantage: you need to create it and
delete it, which can hamper atomicity, and also cause trouble in
parallel builds.  So if the advantage is unclear, I'd rather not go
there.

There's nothing wrong with what we do now, IMO.  We arrived at that
after several iterations, which did try some of the suggestions in
this thread.  If Magit users are annoyed by their popping up
unexpectedly, adding them to .gitignore is TRT and will solve the
issue cleanly and safely.

bug#48141: 28.0.50; Files left over by native compiler

[Gregory Heytings]

>> Given that requirement (atomicity) and its implication (same 
>> partition), would it not be better to create these temporary files in a 
>> 'tmp' subdirectory of their target directory (e.g. 'lisp/tmp' for el 
>> files in 'lisp' and 'lisp/calc/tmp' for el files in 'lisp/calc') 
>> instead of using random filenames in the target directory itself?
>
> Since they will be in the tree anyway, why does it matter in what 
> subdirectory they will be deposited?
>

IMO it's easier to manage, both in .gitignore and to clean the temporary 
files.  But it's just a suggestion (inspired by the way maildirs work).