bug#48079: Temporary files while building after native-comp merge

bug#48079: Temporary files while building after native-comp merge

[Stefan Kangas]

Severity: minor

After the native-comp merge, I see temporary files like this while
building:

    lisp/emacs-lisp/cl-generic.elcsEFkYM
    lisp/emacs-lisp/lisp-mode.elcghtoKr
    lisp/emacs-lisp/macroexp.elco2VM6n
    lisp/emacs-lisp/nadvice.elcCtU6Tw
    lisp/emacs-lisp/syntax.elcM42AoI
    lisp/emacs-lisp/tabulated-list.elc0hWK4o

These files show up in "git status" (in my case, more specifically, in
my `magit-status' buffer) if you run that command at the wrong time.

It would be nice if these files could be added to .gitignore or
something so that one doesn't have to see them, even briefly.

bug#48079: Temporary files while building after native-comp merge

[Eli Zaretskii]

> From: Stefan Kangas <stefan@marxist.se>
> Date: Wed, 28 Apr 2021 06:10:06 -0500
> Cc: Andrea Corallo <akrl@sdf.org>
> 
> After the native-comp merge, I see temporary files like this while
> building:
> 
>     lisp/emacs-lisp/cl-generic.elcsEFkYM
>     lisp/emacs-lisp/lisp-mode.elcghtoKr
>     lisp/emacs-lisp/macroexp.elco2VM6n
>     lisp/emacs-lisp/nadvice.elcCtU6Tw
>     lisp/emacs-lisp/syntax.elcM42AoI
>     lisp/emacs-lisp/tabulated-list.elc0hWK4o
> 
> These files show up in "git status" (in my case, more specifically, in
> my `magit-status' buffer) if you run that command at the wrong time.
> 
> It would be nice if these files could be added to .gitignore or
> something so that one doesn't have to see them, even briefly.

This is due to some bug, so ignoring these files would not be TRT.
I have no such files, FWIW.

Can you tell by looking at the times of these files when they were
created? is that during your bootstrap following the merge of the
branch?  Is it possible some native-compilation crashed at some point
(in which case these files could be left behind)?

bug#48079: Temporary files while building after native-comp merge

[Eli Zaretskii]

> Date: Wed, 28 Apr 2021 15:10:43 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: 48079@debbugs.gnu.org, akrl@sdf.org
> 
> >     lisp/emacs-lisp/cl-generic.elcsEFkYM
> >     lisp/emacs-lisp/lisp-mode.elcghtoKr
> >     lisp/emacs-lisp/macroexp.elco2VM6n
> >     lisp/emacs-lisp/nadvice.elcCtU6Tw
> >     lisp/emacs-lisp/syntax.elcM42AoI
> >     lisp/emacs-lisp/tabulated-list.elc0hWK4o
> > 
> > These files show up in "git status" (in my case, more specifically, in
> > my `magit-status' buffer) if you run that command at the wrong time.

Oh, you mean they appear momentarily, and then disappear?

But then it should also happen when a simple byte-compilation runs,
right?  IOW, it isn't specific to native-comp, right?

bug#48079: Temporary files while building after native-comp merge

[Stefan Kangas]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Wed, 28 Apr 2021 15:10:43 +0300
>> From: Eli Zaretskii <eliz@gnu.org>
>> Cc: 48079@debbugs.gnu.org, akrl@sdf.org
>>
>> >     lisp/emacs-lisp/cl-generic.elcsEFkYM
>> >     lisp/emacs-lisp/lisp-mode.elcghtoKr
>> >     lisp/emacs-lisp/macroexp.elco2VM6n
>> >     lisp/emacs-lisp/nadvice.elcCtU6Tw
>> >     lisp/emacs-lisp/syntax.elcM42AoI
>> >     lisp/emacs-lisp/tabulated-list.elc0hWK4o
>> >
>> > These files show up in "git status" (in my case, more specifically, in
>> > my `magit-status' buffer) if you run that command at the wrong time.
>
> Oh, you mean they appear momentarily, and then disappear?

Correct, sorry for not being more clear.

> But then it should also happen when a simple byte-compilation runs,
> right?  IOW, it isn't specific to native-comp, right?

Hmm, I have never noticed any such files before the native-comp merge
but it is possible that they were there before.  They might just be
staying around longer now, perhaps?

I've now seen them several times over the last couple of days, whereas
I've never seen them before, so something must have changed.

bug#48079: Temporary files while building after native-comp merge

[Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Wed, 28 Apr 2021 15:10:43 +0300
>> From: Eli Zaretskii <eliz@gnu.org>
>> Cc: 48079@debbugs.gnu.org, akrl@sdf.org
>> 
>> >     lisp/emacs-lisp/cl-generic.elcsEFkYM
>> >     lisp/emacs-lisp/lisp-mode.elcghtoKr
>> >     lisp/emacs-lisp/macroexp.elco2VM6n
>> >     lisp/emacs-lisp/nadvice.elcCtU6Tw
>> >     lisp/emacs-lisp/syntax.elcM42AoI
>> >     lisp/emacs-lisp/tabulated-list.elc0hWK4o
>> > 
>> > These files show up in "git status" (in my case, more specifically, in
>> > my `magit-status' buffer) if you run that command at the wrong time.
>
> Oh, you mean they appear momentarily, and then disappear?

Confirm

> But then it should also happen when a simple byte-compilation runs,
> right?  IOW, it isn't specific to native-comp, right?

Exacly, it's just that native takes longer to complete.

Regards

  Andrea

bug#48079: Temporary files while building after native-comp merge

[Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Stefan Kangas <stefan@marxist.se> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> Date: Wed, 28 Apr 2021 15:10:43 +0300
>>> From: Eli Zaretskii <eliz@gnu.org>
>>> Cc: 48079@debbugs.gnu.org, akrl@sdf.org
>>>
>>> >     lisp/emacs-lisp/cl-generic.elcsEFkYM
>>> >     lisp/emacs-lisp/lisp-mode.elcghtoKr
>>> >     lisp/emacs-lisp/macroexp.elco2VM6n
>>> >     lisp/emacs-lisp/nadvice.elcCtU6Tw
>>> >     lisp/emacs-lisp/syntax.elcM42AoI
>>> >     lisp/emacs-lisp/tabulated-list.elc0hWK4o
>>> >
>>> > These files show up in "git status" (in my case, more specifically, in
>>> > my `magit-status' buffer) if you run that command at the wrong time.
>>
>> Oh, you mean they appear momentarily, and then disappear?
>
> Correct, sorry for not being more clear.
>
>> But then it should also happen when a simple byte-compilation runs,
>> right?  IOW, it isn't specific to native-comp, right?
>
> Hmm, I have never noticed any such files before the native-comp merge
> but it is possible that they were there before.  They might just be
> staying around longer now, perhaps?

Nothing should be changed in this respect.

> I've now seen them several times over the last couple of days, whereas
> I've never seen them before, so something must have changed.

I'm wondering, is it a real issue not to have in .gitignore files that
are showing up temporary?

Regards

  Andrea

bug#48079: Temporary files while building after native-comp merge

[Stefan Kangas]

Andrea Corallo <akrl@sdf.org> writes:

> I'm wondering, is it a real issue not to have in .gitignore files that
> are showing up temporary?

That depends on your definition of "real issue", I suppose.  ;-)

For me it becomes a real issue because I'm used to be working in the
magit-status buffer while compiling, and that buffer shows all
uncommitted files at the top and any changes at the bottom.  It is
unsettling to have these files show up there occasionally as it makes
the changes "jump" up and down.

But perhaps the fix is as simple as:

diff --git a/.gitignore b/.gitignore
index fcbc9cd7f4..e27ebe36d0 100644
--- a/.gitignore
+++ b/.gitignore
@@ -135,6 +135,7 @@ src/gl-stamp
 *.dll
 *.core
 *.elc
+*.elc[0-9A-Za-z][0-9A-Za-z][0-9A-Za-z][0-9A-Za-z][0-9A-Za-z][0-9A-Za-z]
 *.eln
 *.o
 *.res

bug#48079: Temporary files while building after native-comp merge

[Eli Zaretskii]

> From: Stefan Kangas <stefan@marxist.se>
> Date: Wed, 28 Apr 2021 16:26:30 -0500
> Cc: Eli Zaretskii <eliz@gnu.org>, 48079@debbugs.gnu.org
> 
> Andrea Corallo <akrl@sdf.org> writes:
> 
> > I'm wondering, is it a real issue not to have in .gitignore files that
> > are showing up temporary?
> 
> That depends on your definition of "real issue", I suppose.  ;-)
> 
> For me it becomes a real issue because I'm used to be working in the
> magit-status buffer while compiling, and that buffer shows all
> uncommitted files at the top and any changes at the bottom.  It is
> unsettling to have these files show up there occasionally as it makes
> the changes "jump" up and down.

Do you have some kind of auto-revert feature turned on?  If not, how
come magit-status even knows these files are created?

> But perhaps the fix is as simple as:
> 
> diff --git a/.gitignore b/.gitignore
> index fcbc9cd7f4..e27ebe36d0 100644
> --- a/.gitignore
> +++ b/.gitignore
> @@ -135,6 +135,7 @@ src/gl-stamp
>  *.dll
>  *.core
>  *.elc
> +*.elc[0-9A-Za-z][0-9A-Za-z][0-9A-Za-z][0-9A-Za-z][0-9A-Za-z][0-9A-Za-z]
>  *.eln
>  *.o
>  *.res

Fine with me, if no better solution emerges.

bug#48079: Temporary files while building after native-comp merge

[Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Stefan Kangas <stefan@marxist.se> writes:

> Andrea Corallo <akrl@sdf.org> writes:
>
>> I'm wondering, is it a real issue not to have in .gitignore files that
>> are showing up temporary?
>
> That depends on your definition of "real issue", I suppose.  ;-)
>
> For me it becomes a real issue because I'm used to be working in the
> magit-status buffer while compiling, and that buffer shows all
> uncommitted files at the top and any changes at the bottom.  It is
> unsettling to have these files show up there occasionally as it makes
> the changes "jump" up and down.

Probably yes :) I use magit while compiling all the time and I never
cared about those files temporary showing up under "Untracked files" (it
can also be collabsed with tab).

> But perhaps the fix is as simple as:
>
> diff --git a/.gitignore b/.gitignore
> index fcbc9cd7f4..e27ebe36d0 100644
> --- a/.gitignore
> +++ b/.gitignore
> @@ -135,6 +135,7 @@ src/gl-stamp
>  *.dll
>  *.core
>  *.elc
> +*.elc[0-9A-Za-z][0-9A-Za-z][0-9A-Za-z][0-9A-Za-z][0-9A-Za-z][0-9A-Za-z]
>  *.eln
>  *.o
>  *.res

FWIW LGTM

  Andrea

bug#48079: Temporary files while building after native-comp merge

[Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Stefan Kangas <stefan@marxist.se>
>> Date: Wed, 28 Apr 2021 16:26:30 -0500
>> Cc: Eli Zaretskii <eliz@gnu.org>, 48079@debbugs.gnu.org
>> 
>> Andrea Corallo <akrl@sdf.org> writes:
>> 
>> > I'm wondering, is it a real issue not to have in .gitignore files that
>> > are showing up temporary?
>> 
>> That depends on your definition of "real issue", I suppose.  ;-)
>> 
>> For me it becomes a real issue because I'm used to be working in the
>> magit-status buffer while compiling, and that buffer shows all
>> uncommitted files at the top and any changes at the bottom.  It is
>> unsettling to have these files show up there occasionally as it makes
>> the changes "jump" up and down.
>
> Do you have some kind of auto-revert feature turned on?  If not, how
> come magit-status even knows these files are created?

I believe he's referring to a new invocation of `magit-status' while
compilation is happening.

  Andrea

bug#48079: Temporary files while building after native-comp merge

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: Stefan Kangas <stefan@marxist.se>, 48079@debbugs.gnu.org
> Date: Thu, 29 Apr 2021 08:19:31 +0000
> 
> > Do you have some kind of auto-revert feature turned on?  If not, how
> > come magit-status even knows these files are created?
> 
> I believe he's referring to a new invocation of `magit-status' while
> compilation is happening.

Btw, what is the reason that these temporary *.elc files live longer
with the native compilation?  Does the native compilation need the
*.elc file for its processing?  If not, perhaps we could rename or
delete them earlier?

bug#48079: Temporary files while building after native-comp merge

[Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <akrl@sdf.org>
>> Cc: Stefan Kangas <stefan@marxist.se>, 48079@debbugs.gnu.org
>> Date: Thu, 29 Apr 2021 08:19:31 +0000
>> 
>> > Do you have some kind of auto-revert feature turned on?  If not, how
>> > come magit-status even knows these files are created?
>> 
>> I believe he's referring to a new invocation of `magit-status' while
>> compilation is happening.
>
> Btw, what is the reason that these temporary *.elc files live longer
> with the native compilation?

Yes, essentially just the fact that compilation takes longer.

When we compile Emacs the makefile uses
`batch-byte-native-compile-for-bootstrap' to produce both the .elc and
the .eln.  As the eln is produced as side product of the .elc to have
the makefile dependecy model work we can't rename the .elc before the
.eln is also finished even if we could, otherwise in case of
interruption we may have the .elc produced but not the .eln.

  Andrea

> Does the native compilation need the
> *.elc file for its processing?  If not, perhaps we could rename or
> delete them earlier?
>

bug#48079: Temporary files while building after native-comp merge

[Stefan Kangas]

Andrea Corallo <akrl@sdf.org> writes:

>> Do you have some kind of auto-revert feature turned on?  If not, how
>> come magit-status even knows these files are created?
>
> I believe he's referring to a new invocation of `magit-status' while
> compilation is happening.

Correct, I run that command manually and on purpose.

bug#48079: Temporary files while building after native-comp merge

[Lars Ingebrigtsen]

Stefan Kangas <stefan@marxist.se> writes:

> But perhaps the fix is as simple as:
>
> diff --git a/.gitignore b/.gitignore
> index fcbc9cd7f4..e27ebe36d0 100644
> --- a/.gitignore
> +++ b/.gitignore
> @@ -135,6 +135,7 @@ src/gl-stamp
>  *.dll
>  *.core
>  *.elc
> +*.elc[0-9A-Za-z][0-9A-Za-z][0-9A-Za-z][0-9A-Za-z][0-9A-Za-z][0-9A-Za-z]
>  *.eln
>  *.o
>  *.res

Sounds like a good idea to me.  (But add a comment about what these
files are.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#48079: Temporary files while building after native-comp merge

[Stefan Kangas]

[-- Attachment #1: Type: text/plain, Size: 1356 bytes --]

tags 48079 - patch
thanks

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Stefan Kangas <stefan@marxist.se> writes:
>
>> But perhaps the fix is as simple as:
>>
>> diff --git a/.gitignore b/.gitignore
>> index fcbc9cd7f4..e27ebe36d0 100644
>> --- a/.gitignore
>> +++ b/.gitignore
>> @@ -135,6 +135,7 @@ src/gl-stamp
>>  *.dll
>>  *.core
>>  *.elc
>> +*.elc[0-9A-Za-z][0-9A-Za-z][0-9A-Za-z][0-9A-Za-z][0-9A-Za-z][0-9A-Za-z]
>>  *.eln
>>  *.o
>>  *.res
>
> Sounds like a good idea to me.  (But add a comment about what these
> files are.)

Thanks.

But now I am seeing that some of these files have not been deleted in my
tree:

    lisp/emacs-lisp/comp.elcivrXZS
    lisp/window.elc9aaVMg

This might be because I interrupted the build process at some point; I
don't know.

These files don't get removed when I run extraclean or bootstrap.  So if
we just ignore them there is a risk that these files will slowly build
up over time.

Perhaps we should also make sure they get cleaned up when building, as
in the attached.

Another alternative, arguably more correct, would be to figure out
exactly under which circumstances these files gets left over and delete
them there.  (There is a let-bound `kill-emacs-hook' in
`byte-compile-file', which might be a good place to start looking.)
What makes this a bit tricky is that this is somewhat hard to reproduce.

[-- Attachment #2: 0001-Delete-temporary-.elcXXXXXX-files-created-when-compi.patch --]
[-- Type: text/x-diff, Size: 2084 bytes --]

From 54a57a777968dc5ae12256a661ad2d548cc9bfd0 Mon Sep 17 00:00:00 2001
From: Stefan Kangas <stefan@marxist.se>
Date: Thu, 29 Apr 2021 13:16:13 +0200
Subject: [PATCH] Delete temporary *.elcXXXXXX files created when compiling

* lisp/Makefile.in (compile-clean):
* test/Makefile.in (clean): Delete left-over *.elcXXXXXX files created
when compiling.

* .gitignore: Ignore temporary *.elcXXXXXX files created when
compiling.  (Bug#48079)
---
 .gitignore       | 3 +++
 lisp/Makefile.in | 2 ++
 test/Makefile.in | 2 +-
 3 files changed, 6 insertions(+), 1 deletion(-)

diff --git a/.gitignore b/.gitignore
index fcbc9cd7f4..14882b5353 100644
--- a/.gitignore
+++ b/.gitignore
@@ -148,6 +148,9 @@ oo-spd/
 src/*.map
 vgcore.*[0-9]
 
+# Temporary files when byte-compiling.
+*.elc[0-9A-Za-z][0-9A-Za-z][0-9A-Za-z][0-9A-Za-z][0-9A-Za-z][0-9A-Za-z]
+
 # Tests.
 test/manual/biditest.txt
 test/manual/etags/srclist
diff --git a/lisp/Makefile.in b/lisp/Makefile.in
index b970451dd2..de2c8fcad1 100644
--- a/lisp/Makefile.in
+++ b/lisp/Makefile.in
@@ -345,6 +345,7 @@ compile-main:
 
 .PHONY: compile-clean
 # Erase left-over .elc files that do not have a corresponding .el file.
+# Also erase any left-over temporary files such as "*.elcivrXZS".
 compile-clean:
 	@cd $(lisp) && \
 	elcs=`echo "${SUBDIRS_REL} " | sed -e 's|/\./|/|g' -e 's|/\. | |g' -e 's| |/*.elc |g'`; \
@@ -354,6 +355,7 @@ compile-clean:
 	    rm "$${el}c"; \
 	  fi; \
 	done
+	find $(lisp) -name '*.elc[0-9A-Za-z][0-9A-Za-z][0-9A-Za-z][0-9A-Za-z][0-9A-Za-z][0-9A-Za-z]' $(FIND_DELETE)
 
 .PHONY: gen-lisp leim semantic
 
diff --git a/test/Makefile.in b/test/Makefile.in
index 84ab4e70ae..d04376be56 100644
--- a/test/Makefile.in
+++ b/test/Makefile.in
@@ -339,7 +339,7 @@ mostlyclean:
 	rm -f ./*.tmp
 
 clean:
-	find . '(' -name '*.log' -o -name '*.log~' ')' $(FIND_DELETE)
+	find . '(' -name '*.log' -o -name '*.log~' -o -name '*.elc[0-9A-Za-z][0-9A-Za-z][0-9A-Za-z][0-9A-Za-z][0-9A-Za-z][0-9A-Za-z]' ')' $(FIND_DELETE)
 	rm -f $(test_module_dir)/*.o $(test_module_dir)/*.so \
 	  $(test_module_dir)/*.dll
 
-- 
2.30.2

bug#48079: Temporary files while building after native-comp merge

[Lars Ingebrigtsen]

Stefan Kangas <stefan@marxist.se> writes:

> These files don't get removed when I run extraclean or bootstrap.  So if
> we just ignore them there is a risk that these files will slowly build
> up over time.
>
> Perhaps we should also make sure they get cleaned up when building, as
> in the attached.

Makes sense to me.

> Another alternative, arguably more correct, would be to figure out
> exactly under which circumstances these files gets left over and delete
> them there.  (There is a let-bound `kill-emacs-hook' in
> `byte-compile-file', which might be a good place to start looking.)
> What makes this a bit tricky is that this is somewhat hard to reproduce.

Bug#48141 is about this problem, I think.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#48079: Temporary files while building after native-comp merge

[Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Stefan Kangas <stefan@marxist.se> writes:

> tags 48079 - patch
> thanks
>
> Lars Ingebrigtsen <larsi@gnus.org> writes:
>
>> Stefan Kangas <stefan@marxist.se> writes:
>>
>>> But perhaps the fix is as simple as:
>>>
>>> diff --git a/.gitignore b/.gitignore
>>> index fcbc9cd7f4..e27ebe36d0 100644
>>> --- a/.gitignore
>>> +++ b/.gitignore
>>> @@ -135,6 +135,7 @@ src/gl-stamp
>>>  *.dll
>>>  *.core
>>>  *.elc
>>> +*.elc[0-9A-Za-z][0-9A-Za-z][0-9A-Za-z][0-9A-Za-z][0-9A-Za-z][0-9A-Za-z]
>>>  *.eln
>>>  *.o
>>>  *.res
>>
>> Sounds like a good idea to me.  (But add a comment about what these
>> files are.)
>
> Thanks.
>
> But now I am seeing that some of these files have not been deleted in my
> tree:
>
>     lisp/emacs-lisp/comp.elcivrXZS
>     lisp/window.elc9aaVMg
>
> This might be because I interrupted the build process at some point; I
> don't know.
>
> These files don't get removed when I run extraclean or bootstrap.  So if
> we just ignore them there is a risk that these files will slowly build
> up over time.
>
> Perhaps we should also make sure they get cleaned up when building, as
> in the attached.

I'm not sure is correct to do that in compile-clean, if there's really a
problem IMO we should isolate it and debug, IIUC having this in
compile-clean could just mask it.

  Andrea

bug#48079: Temporary files while building after native-comp merge

[Lars Ingebrigtsen]

Stefan Kangas <stefan@marxist.se> writes:

> But now I am seeing that some of these files have not been deleted in my
> tree:
>
>     lisp/emacs-lisp/comp.elcivrXZS
>     lisp/window.elc9aaVMg
>
> This might be because I interrupted the build process at some point; I
> don't know.

I got this, too (several copies of comp.elc<random>), and I was
momentarily confused -- but then I remembered that I had `C-c'd the
compilation.  Since it takes so long to build these files now, that's
probably be a common occurrence.

I guess we can't set up a signal trap to delete the temp file if we're
being interrupted?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#48079: Temporary files while building after native-comp merge

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Andrea Corallo <akrl@sdf.org>,  Eli Zaretskii <eliz@gnu.org>,
>   48079@debbugs.gnu.org
> Date: Wed, 05 May 2021 10:35:02 +0200
> 
> I guess we can't set up a signal trap to delete the temp file if we're
> being interrupted?

You mean, in the Makefile?  That won't handle the cases where the
compilation is manually invoked (e.g., from a running Emacs).

bug#48079: Temporary files while building after native-comp merge

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> I guess we can't set up a signal trap to delete the temp file if we're
>> being interrupted?
>
> You mean, in the Makefile?  That won't handle the cases where the
> compilation is manually invoked (e.g., from a running Emacs).

I'm not actually sure -- I don't know what the possibilities at our
disposal are here, really.  If we can catch this in Emacs, that'd be
nicer...  but can we?

Another thing I'm wondering about is why we write the subr.elc0EdJIV
file at all, and then apparently don't rename it to .elc immediately?
It seems to linger on in that name for a very long time?  But I haven't
actually looked at the code here.

It's easy to reproduce the behaviour here:

touch lisp/subr.el
make
wait a few seconds
C-c C-c a few times

and then you have a subr.elc0EdJIV file or two.

So is it writing the subr.elc0EdJIV file, then doing the .eln
compilation, and then moving subr.elc0EdJIV to subr.elc?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#48079: Temporary files while building after native-comp merge

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: stefan@marxist.se,  akrl@sdf.org,  48079@debbugs.gnu.org
> Date: Wed, 05 May 2021 14:47:43 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> I guess we can't set up a signal trap to delete the temp file if we're
> >> being interrupted?
> >
> > You mean, in the Makefile?  That won't handle the cases where the
> > compilation is manually invoked (e.g., from a running Emacs).
> 
> I'm not actually sure -- I don't know what the possibilities at our
> disposal are here, really.  If we can catch this in Emacs, that'd be
> nicer...  but can we?

With complicated enough code, sure, we could.

> Another thing I'm wondering about is why we write the subr.elc0EdJIV
> file at all, and then apparently don't rename it to .elc immediately?
> It seems to linger on in that name for a very long time?  But I haven't
> actually looked at the code here.

AFAIR, we rename it atomically when we are done with it.  I guess with
native-compilation it takes longer to "be done with it".

> So is it writing the subr.elc0EdJIV file, then doing the .eln
> compilation, and then moving subr.elc0EdJIV to subr.elc?

Yes, I think so.

bug#48079: Temporary files while building after native-comp merge

[Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Lars Ingebrigtsen <larsi@gnus.org>
>> Cc: stefan@marxist.se,  akrl@sdf.org,  48079@debbugs.gnu.org
>> Date: Wed, 05 May 2021 14:47:43 +0200
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> >> I guess we can't set up a signal trap to delete the temp file if we're
>> >> being interrupted?
>> >
>> > You mean, in the Makefile?  That won't handle the cases where the
>> > compilation is manually invoked (e.g., from a running Emacs).
>> 
>> I'm not actually sure -- I don't know what the possibilities at our
>> disposal are here, really.  If we can catch this in Emacs, that'd be
>> nicer...  but can we?
>
> With complicated enough code, sure, we could.
>
>> Another thing I'm wondering about is why we write the subr.elc0EdJIV
>> file at all, and then apparently don't rename it to .elc immediately?
>> It seems to linger on in that name for a very long time?  But I haven't
>> actually looked at the code here.
>
> AFAIR, we rename it atomically when we are done with it.  I guess with
> native-compilation it takes longer to "be done with it".

Exactly, we can't rename before the native compilation is done as native
compilation is a side product of byte compilation and the build system
knows only the latter.

>> So is it writing the subr.elc0EdJIV file, then doing the .eln
>> compilation, and then moving subr.elc0EdJIV to subr.elc?
>
> Yes, I think so.

Confirm.

  Andrea

bug#48079: Temporary files while building after native-comp merge

[Lars Ingebrigtsen]

Andrea Corallo <akrl@sdf.org> writes:

>>> So is it writing the subr.elc0EdJIV file, then doing the .eln
>>> compilation, and then moving subr.elc0EdJIV to subr.elc?
>>
>> Yes, I think so.
>
> Confirm.

Again, I haven't actually looked at the code, so this is totally
uninformed -- but do we have to write the subr.elc0EdJIV file before
doing the .eln compilation?  Can't the .eln compilation work off of the
contents of a buffer instead?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#48079: Temporary files while building after native-comp merge

[Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Andrea Corallo <akrl@sdf.org> writes:
>
>>>> So is it writing the subr.elc0EdJIV file, then doing the .eln
>>>> compilation, and then moving subr.elc0EdJIV to subr.elc?
>>>
>>> Yes, I think so.
>>
>> Confirm.
>
> Again, I haven't actually looked at the code, so this is totally
> uninformed -- but do we have to write the subr.elc0EdJIV file before
> doing the .eln compilation?  Can't the .eln compilation work off of the
> contents of a buffer instead?

I think it should be possible, ATM is still the byte-complier saving the
.elc buffer and we delay just the final renaming in comp.el, we could
delay the buffer being saved and do it too after the .eln is produced.

  Andrea

bug#48079: Temporary files while building after native-comp merge

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> Btw, what is the reason that these temporary *.elc files live longer
>> with the native compilation?
>
> Yes, essentially just the fact that compilation takes longer.
>
> When we compile Emacs the makefile uses
> `batch-byte-native-compile-for-bootstrap' to produce both the .elc and
> the .eln.  As the eln is produced as side product of the .elc to have
> the makefile dependecy model work we can't rename the .elc before the
> .eln is also finished even if we could, otherwise in case of
> interruption we may have the .elc produced but not the .eln.

Hmm... IIUC the situation is the following:

In the plain old byte-compiler, these files are *very* short lived
because they're not created during the compilation itself but only at
the very end when we save the result to a file (and we just do it by
first saving to `foo.elcNNMMPP` and then renaming that to `foo.elc`).

Now with `batch-byte-native-compile-for-bootstrap` apparently we "suspend
the byte-compiler" right in the middle of this small time window, i.e. after
writing to `foo.elcNNMMPP` but before its renamed.  Then we call the
native compiler and only once the native compiler is done, we resume the
byte compilation which just renames the file and exits.

If that understanding is correct, then I think we may be able to fix the
problem by just changing the moment at which we suspend the byte-compiler:
suspend it *before* it writes to `foo.elcNNMMPP`.


        Stefan

bug#48079: Temporary files while building after native-comp merge

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Eli Zaretskii <eliz@gnu.org>,  stefan@marxist.se,  48079@debbugs.gnu.org
> Date: Sun, 02 Jan 2022 18:38:41 -0500
> 
> Hmm... IIUC the situation is the following:
> 
> In the plain old byte-compiler, these files are *very* short lived
> because they're not created during the compilation itself but only at
> the very end when we save the result to a file (and we just do it by
> first saving to `foo.elcNNMMPP` and then renaming that to `foo.elc`).
> 
> Now with `batch-byte-native-compile-for-bootstrap` apparently we "suspend
> the byte-compiler" right in the middle of this small time window, i.e. after
> writing to `foo.elcNNMMPP` but before its renamed.  Then we call the
> native compiler and only once the native compiler is done, we resume the
> byte compilation which just renames the file and exits.
> 
> If that understanding is correct, then I think we may be able to fix the
> problem by just changing the moment at which we suspend the byte-compiler:
> suspend it *before* it writes to `foo.elcNNMMPP`.

Are you sure your description above is accurate?  We have a backtrace
in bug#48978 that shows when we create the .elcXXX temporary file.  My
reading of that backtrace is that it's the other way around:
native-compilation invokes byte-compile-file, which compiles the Lisp
into bytecode, creates the file with make-temp-file, and writes out
the bytecode.  It is true that we then defer renaming of the temporary
file in this case, but it looks like the "suspend later" idea might
not work?

bug#48079: Temporary files while building after native-comp merge

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Stefan Monnier <monnier@iro.umontreal.ca>
>> Cc: Eli Zaretskii <eliz@gnu.org>,  stefan@marxist.se,  48079@debbugs.gnu.org
>> Date: Sun, 02 Jan 2022 18:38:41 -0500
>> 
>> Hmm... IIUC the situation is the following:
>> 
>> In the plain old byte-compiler, these files are *very* short lived
>> because they're not created during the compilation itself but only at
>> the very end when we save the result to a file (and we just do it by
>> first saving to `foo.elcNNMMPP` and then renaming that to `foo.elc`).
>> 
>> Now with `batch-byte-native-compile-for-bootstrap` apparently we "suspend
>> the byte-compiler" right in the middle of this small time window, i.e. after
>> writing to `foo.elcNNMMPP` but before its renamed.  Then we call the
>> native compiler and only once the native compiler is done, we resume the
>> byte compilation which just renames the file and exits.
>> 
>> If that understanding is correct, then I think we may be able to fix the
>> problem by just changing the moment at which we suspend the byte-compiler:
>> suspend it *before* it writes to `foo.elcNNMMPP`.
>
> Are you sure your description above is accurate?  We have a backtrace
> in bug#48978 that shows when we create the .elcXXX temporary file.  My
> reading of that backtrace is that it's the other way around:
> native-compilation invokes byte-compile-file, which compiles the Lisp
> into bytecode, creates the file with make-temp-file, and writes out
> the bytecode.  It is true that we then defer renaming of the temporary
> file in this case

That's correct, and we do that on purpose in order not to produce the
.elc file before the .eln one is produced.  Otherwise in case of
interruption we might mess-up the build as 'make' is looking at the .elc
files only as targets.

BR
  Andrea

bug#48079: Temporary files while building after native-comp merge

[Glenn Morris]

Andrea Corallo wrote:

> Otherwise in case of interruption we might mess-up the build as 'make'
> is looking at the .elc files only as targets.

Did you consider using a pattern rule with two targets in the Makefile
to properly express the relationship between .el and .elc/.eln?

See last example in
https://www.gnu.org/software/make/manual/html_node/Pattern-Examples.html

bug#48079: Temporary files while building after native-comp merge

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Cc: Eli Zaretskii <eliz@gnu.org>,  48079@debbugs.gnu.org,  stefan@marxist.se,  Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Thu, 06 Jan 2022 11:30:26 -0500
> 
> Did you consider using a pattern rule with two targets in the Makefile
> to properly express the relationship between .el and .elc/.eln?

The problem here is that the *.eln files end up in a subdirectory of
native-lisp/ whose precise name is only known to Emacs, and so cannot
be easily written.  You need the Emacs binary to be built (which
already introduces some "interesting" dependencies), and you need to
ask Emacs about the name of that subdirectory.

And if that's not enough, there's another complication: some of the
*.eln files need to be in that unknown directory, and some (most) need
to be in its preloaded/ subdirectory.  So we need two different rules
for 2 groups of *.el files, and we need to keep those groups
up-to-date at all times.

bug#48079: Temporary files while building after native-comp merge

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Andrea Corallo [2022-01-06 16:10:37] wrote:
> Eli Zaretskii <eliz@gnu.org> writes:
>>> In the plain old byte-compiler, these files are *very* short lived
>>> because they're not created during the compilation itself but only at
>>> the very end when we save the result to a file (and we just do it by
>>> first saving to `foo.elcNNMMPP` and then renaming that to `foo.elc`).
>>> 
>>> Now with `batch-byte-native-compile-for-bootstrap` apparently we "suspend
>>> the byte-compiler" right in the middle of this small time window, i.e. after
>>> writing to `foo.elcNNMMPP` but before its renamed.  Then we call the
>>> native compiler and only once the native compiler is done, we resume the
>>> byte compilation which just renames the file and exits.
>>> 
>>> If that understanding is correct, then I think we may be able to fix the
>>> problem by just changing the moment at which we suspend the byte-compiler:
>>> suspend it *before* it writes to `foo.elcNNMMPP`.
>>
>> Are you sure your description above is accurate?

No, but I haven't heard any indication to the contrary either.

>> We have a backtrace in bug#48978 that shows when we create the
>> .elcXXX temporary file.  My reading of that backtrace is that it's
>> the other way around: native-compilation invokes byte-compile-file,
>> which compiles the Lisp into bytecode, creates the file with
>> make-temp-file, and writes out the bytecode.  It is true that we then
>> defer renaming of the temporary file in this case

I don't see why you say "the other way around".  Maybe you're putting
more emphasis on some detail of what I wrote than what I intended.
The core os what I wrote is that we should change the code so that the
native compilation takes place before the `.elcNNMMPP` file is written
rather than after.

AFAICT the native compiler does not read/need that file because it gets
the same information in a more convenient format straight from
`bytecomp.el`.

> That's correct,

I don't know what "That" refers to.


        Stefan

bug#48079: Temporary files while building after native-comp merge

[Andrea Corallo]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Andrea Corallo [2022-01-06 16:10:37] wrote:
>> Eli Zaretskii <eliz@gnu.org> writes:
>>>> In the plain old byte-compiler, these files are *very* short lived
>>>> because they're not created during the compilation itself but only at
>>>> the very end when we save the result to a file (and we just do it by
>>>> first saving to `foo.elcNNMMPP` and then renaming that to `foo.elc`).
>>>> 
>>>> Now with `batch-byte-native-compile-for-bootstrap` apparently we "suspend
>>>> the byte-compiler" right in the middle of this small time window, i.e. after
>>>> writing to `foo.elcNNMMPP` but before its renamed.  Then we call the
>>>> native compiler and only once the native compiler is done, we resume the
>>>> byte compilation which just renames the file and exits.
>>>> 
>>>> If that understanding is correct, then I think we may be able to fix the
>>>> problem by just changing the moment at which we suspend the byte-compiler:
>>>> suspend it *before* it writes to `foo.elcNNMMPP`.
>>>
>>> Are you sure your description above is accurate?
>
> No, but I haven't heard any indication to the contrary either.
>
>>> We have a backtrace in bug#48978 that shows when we create the
>>> .elcXXX temporary file.  My reading of that backtrace is that it's
>>> the other way around: native-compilation invokes byte-compile-file,
>>> which compiles the Lisp into bytecode, creates the file with
>>> make-temp-file, and writes out the bytecode.  It is true that we then
>>> defer renaming of the temporary file in this case
>
> I don't see why you say "the other way around".  Maybe you're putting
> more emphasis on some detail of what I wrote than what I intended.
> The core os what I wrote is that we should change the code so that the
> native compilation takes place before the `.elcNNMMPP` file is written
> rather than after.
>
> AFAICT the native compiler does not read/need that file because it gets
> the same information in a more convenient format straight from
> `bytecomp.el`.
>
>> That's correct,
>
> I don't know what "That" refers to.

Eli's description of how it works.

BR

  Andrea

bug#48079: Temporary files while building after native-comp merge

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Andrea Corallo [2022-01-07 16:30:05] wrote:
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>> Andrea Corallo [2022-01-06 16:10:37] wrote:
>>> Eli Zaretskii <eliz@gnu.org> writes:
>>>>> In the plain old byte-compiler, these files are *very* short lived
>>>>> because they're not created during the compilation itself but only at
>>>>> the very end when we save the result to a file (and we just do it by
>>>>> first saving to `foo.elcNNMMPP` and then renaming that to `foo.elc`).
>>>>> 
>>>>> Now with `batch-byte-native-compile-for-bootstrap` apparently we "suspend
>>>>> the byte-compiler" right in the middle of this small time window, i.e. after
>>>>> writing to `foo.elcNNMMPP` but before its renamed.  Then we call the
>>>>> native compiler and only once the native compiler is done, we resume the
>>>>> byte compilation which just renames the file and exits.
>>>>> 
>>>>> If that understanding is correct, then I think we may be able to fix the
>>>>> problem by just changing the moment at which we suspend the byte-compiler:
>>>>> suspend it *before* it writes to `foo.elcNNMMPP`.
>>>>
>>>> Are you sure your description above is accurate?
>>
>> No, but I haven't heard any indication to the contrary either.
>>
>>>> We have a backtrace in bug#48978 that shows when we create the
>>>> .elcXXX temporary file.  My reading of that backtrace is that it's
>>>> the other way around: native-compilation invokes byte-compile-file,
>>>> which compiles the Lisp into bytecode, creates the file with
>>>> make-temp-file, and writes out the bytecode.  It is true that we then
>>>> defer renaming of the temporary file in this case
>>
>> I don't see why you say "the other way around".  Maybe you're putting
>> more emphasis on some detail of what I wrote than what I intended.
>> The core os what I wrote is that we should change the code so that the
>> native compilation takes place before the `.elcNNMMPP` file is written
>> rather than after.
>>
>> AFAICT the native compiler does not read/need that file because it gets
>> the same information in a more convenient format straight from
>> `bytecomp.el`.
>>
>>> That's correct,
>>
>> I don't know what "That" refers to.
>
> Eli's description of how it works.

But your response seems to say that what I propose can't work, whereas
I can't see in which sense what Eli says contradicts what I say.


        Stefan

bug#48079: Temporary files while building after native-comp merge

[Lars Ingebrigtsen]

Stefan Kangas <stefan@marxist.se> writes:

> After the native-comp merge, I see temporary files like this while
> building:
>
>     lisp/emacs-lisp/cl-generic.elcsEFkYM
>     lisp/emacs-lisp/lisp-mode.elcghtoKr
>     lisp/emacs-lisp/macroexp.elco2VM6n
>     lisp/emacs-lisp/nadvice.elcCtU6Tw
>     lisp/emacs-lisp/syntax.elcM42AoI
>     lisp/emacs-lisp/tabulated-list.elc0hWK4o
>
> These files show up in "git status" (in my case, more specifically, in
> my `magit-status' buffer) if you run that command at the wrong time.
>
> It would be nice if these files could be added to .gitignore or
> something so that one doesn't have to see them, even briefly.

Andrea fixed this a while ago in Emacs 29, so I'm closing this bug report.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no