Reproducible work with natively compiled Emacs

Reproducible work with natively compiled Emacs

[Pedro A. Aranda]

[-- Attachment #1: Type: text/plain, Size: 514 bytes --]

Hi,

I have been experiencing some inconsistencies with org-mode, which I 
have been able to trace own to using native compilation and not knowing 
where the files from main were.

Attached is a patch in four parts to make natively compiled orgmode more 
consistent.

This is a first step and I would like reactions to it, before submitting 
a unified patch.

I hope that splitting it up helps understanding the different parts of 
my "solution". Maybe some parts can be cherry picked better this way...

best /PA

[-- Attachment #2: 0001-Add-native-option-to-make-help.patch --]
[-- Type: text/x-patch, Size: 875 bytes --]

From e47289a60f225a0fa6d05efec9d38119d886f4a3 Mon Sep 17 00:00:00 2001
From: "Pedro A. Aranda" <paaguti@gmail.com>
Date: Sun, 10 Mar 2024 11:23:09 +0100
Subject: [PATCH 1/4] Add 'native' option to 'make help'

---
 Makefile | 1 +
 1 file changed, 1 insertion(+)

diff --git a/Makefile b/Makefile
index f476a3ea7..35250eed0 100644
--- a/Makefile
+++ b/Makefile
@@ -27,6 +27,7 @@ help helpall::
 	$(info make all            - ditto)
 	$(info make compile        - build Org ELisp files)
 	$(info make single         - build Org ELisp files, single Emacs per source)
+	$(info make native         - build Org natively compiled ELisp)
 	$(info make autoloads      - create org-loaddefs.el to load Org in-place)
 	$(info make test           - build Org ELisp files and run test suite)
 	$(info make vanilla        - run Emacs with this Org-mode and no personal config)
-- 
2.34.1


[-- Attachment #3: 0002-Store-.eln-files-alongside-the-lisp-files.patch --]
[-- Type: text/x-patch, Size: 646 bytes --]

From 2d3c6b45d435a6356f198c421d71743d9b09efca Mon Sep 17 00:00:00 2001
From: "Pedro A. Aranda" <paaguti@gmail.com>
Date: Sun, 10 Mar 2024 11:26:46 +0100
Subject: [PATCH 2/4] Store .eln files alongside the lisp files

---
 mk/default.mk | 1 +
 1 file changed, 1 insertion(+)

diff --git a/mk/default.mk b/mk/default.mk
index b75aac345..312dbc6aa 100644
--- a/mk/default.mk
+++ b/mk/default.mk
@@ -158,6 +158,7 @@ ELC	= $(BATCHL) \
 
 # How to native-compile a single file
 ELN	= $(BATCHL) \
+	  --eval "(startup-redirect-eln-cache \"$(PWD)/lisp\")" \
 	  --eval '(batch-native-compile)'
 
 # How to make a pdf file from a texinfo file
-- 
2.34.1


[-- Attachment #4: 0003-A-short-doc-for-reproducible-native-compilations-whe.patch --]
[-- Type: text/x-patch, Size: 1792 bytes --]

From 34787b58add09decbf31e0e583fd315a207ce88d Mon Sep 17 00:00:00 2001
From: "Pedro A. Aranda" <paaguti@gmail.com>
Date: Sun, 10 Mar 2024 11:44:38 +0100
Subject: [PATCH 3/4] A short doc for reproducible native compilations when
 developing

---
 doc/native-devel.org | 33 +++++++++++++++++++++++++++++++++
 1 file changed, 33 insertions(+)
 create mode 100644 doc/native-devel.org

diff --git a/doc/native-devel.org b/doc/native-devel.org
new file mode 100644
index 000000000..0cc18a6fe
--- /dev/null
+++ b/doc/native-devel.org
@@ -0,0 +1,33 @@
+#+title: A reproducible natively compiled org-mode
+#+subtitle:  Release {{{version}}}
+#+author:    The Org Mode Developers
+#+language:  en
+
+* Introduction
+
+In order to develop and test ~org-mode~ when your Emacs editor
+supports native compilation, there are two things to take into
+account:
+
+1. you will need to =make native= to generate the =.eln= files. These
+   will be stored under the ~lisp/~ directory of your development
+   ~org-mode~ directory.
+2. Add the following code in your Emacs ~early-init.el~ file:
+
+#+BEGIN_SRC emacs-lisp
+(let ((org-devel-path (expand-file-name "<org-mode development dir>/lisp")))
+  (when (file-directory-p org-devel-path)
+    (when (native-comp-available-p)
+      (let ((first (pop native-comp-eln-load-path)))
+        ;; put in second place
+        (add-to-list 'native-comp-eln-load-path org-devel-path t)
+        (add-to-list 'native-comp-eln-load-path first t)))
+    (add-to-list 'load-path  org-devel-path)))
+#+END_SRC
+
+putting the correct path for your org-mode development directory. In
+my case, it is =~/Devel/org-mode=, so the first =let= looks like this:
+
+#+BEGIN_SRC emacs-lisp
+(let ((org-devel-path (expand-file-name "~/Devel/org-mode/lisp")))
+#+END_SRC
-- 
2.34.1


[-- Attachment #5: 0004-Add-cleaning-native-compilation.patch --]
[-- Type: text/x-patch, Size: 1905 bytes --]

From 0f10451eef8d826ffb31f70f8a6c2db36ebc16a5 Mon Sep 17 00:00:00 2001
From: "Pedro A. Aranda" <paaguti@gmail.com>
Date: Sun, 10 Mar 2024 12:06:28 +0100
Subject: [PATCH 4/4] Add cleaning native compilation

---
 lisp/Makefile | 15 +++++++++++++--
 1 file changed, 13 insertions(+), 2 deletions(-)

diff --git a/lisp/Makefile b/lisp/Makefile
index c570d9cfa..1f03539c2 100644
--- a/lisp/Makefile
+++ b/lisp/Makefile
@@ -10,13 +10,16 @@ LISPB 	:= $(LISPA:%el=%elc) org-install.elc
 LISPF 	:= $(filter-out $(LISPA),$(sort $(wildcard *.el)))
 LISPC 	:= $(filter-out $(LISPB) $(LISPN:%el=%elc),$(LISPF:%el=%elc))
 LISPN 	:= $(filter-out $(LISPB) $(LISPN:%el=%eln),$(LISPF:%el=%eln))
+# Should not be empty if lisp/<version>/org-lint<hash>.eln exists
+# like after a make native
+ELNDIR  := $(shell find . -name org-lint\*.eln -exec dirname {} \;)
 _ORGCM_ := dirall single native source slint1 slint2
 -include local.mk
 
 .PHONY:	all compile compile-dirty \
 	$(_ORGCM_) $(_ORGCM_:%=compile-%) \
 	autoloads \
-	install clean cleanauto cleanall cleanelc clean-install
+	install clean cleanauto cleanall cleanelc cleaneln clean-install
 
 # do not clean here, done in toplevel make
 all compile compile-dirty::	 | autoloads
@@ -41,7 +44,7 @@ single:
 	@$(info ==================== $@ ====================)
 native:
 	@$(info ==================== $@ ====================)
-source: cleanelc
+source: cleanelc cleaneln
 	@$(info ==================== $@ ====================)
 	@$(foreach elc,$(LISPC),$(MAKE) $(elc) && $(RM) $(elc);)
 slint1:
@@ -78,6 +81,14 @@ cleanauto clean cleanall::
 	$(RM) $(LISPA) $(LISPB)
 clean cleanall cleanelc::
 	$(RM) *.elc
+ifeq ($(ELNDIR),)
+cleaneln::
+	@-echo "No previous native compilation "
+else
+clean cleanall cleaneln::
+	@-echo "Removing previous native compilation"
+	@$(RM) -rf $(ELNDIR)
+endif
 
 clean-install:
 	if [ -d $(DESTDIR)$(lispdir) ] ; then \
-- 
2.34.1

Re: Reproducible work with natively compiled Emacs

[Pedro Andres Aranda Gutierrez]

[-- Attachment #1.1: Type: text/plain, Size: 951 bytes --]

HI,

nothing like sleeping over a problem. This solution seems to be easier...

/PA

On Sun, 10 Mar 2024 at 12:49, Pedro A. Aranda <paaguti@gmail.com> wrote:

> Hi,
>
> I have been experiencing some inconsistencies with org-mode, which I
> have been able to trace own to using native compilation and not knowing
> where the files from main were.
>
> Attached is a patch in four parts to make natively compiled orgmode more
> consistent.
>
> This is a first step and I would like reactions to it, before submitting
> a unified patch.
>
> I hope that splitting it up helps understanding the different parts of
> my "solution". Maybe some parts can be cherry picked better this way...
>
> best /PA
>


-- 
Fragen sind nicht da, um beantwortet zu werden,
Fragen sind da um gestellt zu werden
Georg Kreisler

Headaches with a Juju log:
unit-basic-16: 09:17:36 WARNING juju.worker.uniter.operation we should run
a leader-deposed hook here, but we can't yet

[-- Attachment #1.2: Type: text/html, Size: 1589 bytes --]

[-- Attachment #2: 0002-Clean-.eln-files-from-user-emacs-configuration.patch --]
[-- Type: text/x-patch, Size: 1468 bytes --]

From 9476898539bd9ad72a5b45bc1625fa398eada879 Mon Sep 17 00:00:00 2001
From: "Pedro A. Aranda" <paaguti@gmail.com>
Date: Mon, 11 Mar 2024 09:48:10 +0100
Subject: [PATCH 2/3] Clean .eln files from user emacs configuration

* lisp/Makefile: Get the directory where emacs stores the compiled files
  in the user storage and use that to locate the .eln files generated by
  'make native' and delete them

---
 lisp/Makefile | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/lisp/Makefile b/lisp/Makefile
index c570d9cfa..506aa75e1 100644
--- a/lisp/Makefile
+++ b/lisp/Makefile
@@ -11,6 +11,9 @@ LISPF 	:= $(filter-out $(LISPA),$(sort $(wildcard *.el)))
 LISPC 	:= $(filter-out $(LISPB) $(LISPN:%el=%elc),$(LISPF:%el=%elc))
 LISPN 	:= $(filter-out $(LISPB) $(LISPN:%el=%eln),$(LISPF:%el=%eln))
 _ORGCM_ := dirall single native source slint1 slint2
+# This is where Emacs stores the .eln files
+ELNDIR        = $(shell emacs --batch --eval '(princ (car native-comp-eln-load-path))')
+
 -include local.mk

 .PHONY:	all compile compile-dirty \
@@ -79,6 +82,11 @@ cleanauto clean cleanall::
 clean cleanall cleanelc::
 	$(RM) *.elc

+cleaneln::
+	for elnf in $(patsubst %.eln, %-*.eln, $(LISPN)); do \
+	  find $(ELNDIR) -name $$elnf -exec $(RM) -v {} \; ;\
+	done
+
 clean-install:
 	if [ -d $(DESTDIR)$(lispdir) ] ; then \
 	  $(RM) $(DESTDIR)$(lispdir)/org*.el* $(DESTDIR)$(lispdir)/ob*.el* $(DESTDIR)$(lispdir)/ol*.el* $(DESTDIR)$(lispdir)/ox*.el* ; \
--
2.34.1

[-- Attachment #3: 0001-Add-cleaneln-target.patch --]
[-- Type: text/x-patch, Size: 643 bytes --]

From 6d98dfb3d72171e0231823260165b23e8b9963ee Mon Sep 17 00:00:00 2001
From: "Pedro A. Aranda" <paaguti@gmail.com>
Date: Mon, 11 Mar 2024 09:47:12 +0100
Subject: [PATCH 1/3] Add cleaneln target

* mk/targets.mk: add the 'cleaneln' target

---
 mk/targets.mk | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/mk/targets.mk b/mk/targets.mk
index de849c4fb..ab9ec2a1f 100644
--- a/mk/targets.mk
+++ b/mk/targets.mk
@@ -143,6 +143,9 @@ cleanall: cleandirs cleantest
 $(CLEANDIRS:%=clean%):
 	-$(FIND) $(@:clean%=%) \( -name \*~ -o -name \*.elc \) -exec $(RM) {} +

+cleaneln:
+	$(MAKE) -C lisp $@
+
 cleanelc:
 	$(MAKE) -C lisp $@

--
2.34.1

[-- Attachment #4: 0003-Announce-cleaneln.patch --]
[-- Type: text/x-patch, Size: 1335 bytes --]

From 0b5d791d32f26698e53299e639d7ef73581a6310 Mon Sep 17 00:00:00 2001
From: "Pedro A. Aranda" <paaguti@gmail.com>
Date: Mon, 11 Mar 2024 09:56:05 +0100
Subject: [PATCH 3/3] Announce cleaneln

---
 etc/ORG-NEWS | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/etc/ORG-NEWS b/etc/ORG-NEWS
index abe62daaf..325fd412b 100644
--- a/etc/ORG-NEWS
+++ b/etc/ORG-NEWS
@@ -11,7 +11,8 @@ See the end of the file for license conditions.
 
 Please send Org bug reports to mailto:emacs-orgmode@gnu.org.
 
-* Version 9.7 (not released yet)
+* V
+ersion 9.7 (not released yet)
 ** Important announcements and breaking changes
 *** When ~org-link-file-path-type~ is a function, its argument is now a filename as it is read by ~org-insert-link~; not an absolute path
 
@@ -1263,6 +1264,11 @@ optional argument =NEW-HEADING-CONTAINER= specifies where in the
 buffer it will be added.  If not specified, new headings are created
 at level 1 at the end of the accessible part of the buffer, as before.
 
+*** ~make cleaneln~ will remove the .eln files from the user emacs configuration
+
+Natively compiled Emacs lisp files generated with ~make native~ are
+now correctly located and deleted with ~make cleaneln~.
+
 ** Miscellaneous
 *** =org-crypt.el= now applies initial visibility settings to decrypted entries
 
-- 
2.34.1

Re: Reproducible work with natively compiled Emacs

[Ihor Radchenko]

Pedro Andres Aranda Gutierrez <paaguti@gmail.com> writes:

>> I have been experiencing some inconsistencies with org-mode, which I
>> have been able to trace own to using native compilation and not knowing
>> where the files from main were.

Thanks for the patch!
See my comments inline.

> +# This is where Emacs stores the .eln files
> +ELNDIR        = $(shell emacs --batch --eval '(princ (car native-comp-eln-load-path))')

Please use $(EMACS).

> +cleaneln::
> +	for elnf in $(patsubst %.eln, %-*.eln, $(LISPN)); do \
> +	  find $(ELNDIR) -name $$elnf -exec $(RM) -v {} \; ;\
> +	done

$(FIND).
Also, avoid using bash as much as possible in favor or GNU make
constructs. You can use $(foreach ...) here.

>  clean-install:
>  	if [ -d $(DESTDIR)$(lispdir) ] ; then \
>  	  $(RM) $(DESTDIR)$(lispdir)/org*.el* $(DESTDIR)$(lispdir)/ob*.el* $(DESTDIR)$(lispdir)/ol*.el* $(DESTDIR)$(lispdir)/ox*.el* ; \
> --
> 2.34.1
> From 6d98dfb3d72171e0231823260165b23e8b9963ee Mon Sep 17 00:00:00 2001
> From: "Pedro A. Aranda" <paaguti@gmail.com>
> Date: Mon, 11 Mar 2024 09:47:12 +0100
> Subject: [PATCH 1/3] Add cleaneln target
>
> * mk/targets.mk: add the 'cleaneln' target
>
> ---
>  mk/targets.mk | 3 +++
>  1 file changed, 3 insertions(+)
>
> diff --git a/mk/targets.mk b/mk/targets.mk
> index de849c4fb..ab9ec2a1f 100644
> --- a/mk/targets.mk
> +++ b/mk/targets.mk
> @@ -143,6 +143,9 @@ cleanall: cleandirs cleantest
>  $(CLEANDIRS:%=clean%):
>  	-$(FIND) $(@:clean%=%) \( -name \*~ -o -name \*.elc \) -exec $(RM) {} +
>
> +cleaneln:
> +	$(MAKE) -C lisp $@
> +

Should be in .PHONY

> --- a/etc/ORG-NEWS
> +++ b/etc/ORG-NEWS
> @@ -11,7 +11,8 @@ See the end of the file for license conditions.
>  
>  Please send Org bug reports to mailto:emacs-orgmode@gnu.org.
>  
> -* Version 9.7 (not released yet)
> +* V
> +ersion 9.7 (not released yet)

This is a stray newline.

> +*** ~make cleaneln~ will remove the .eln files from the user emacs configuration
> +
> +Natively compiled Emacs lisp files generated with ~make native~ are
> +now correctly located and deleted with ~make cleaneln~.
> +
>  ** Miscellaneous
>  *** =org-crypt.el= now applies initial visibility settings to decrypted entries

"Add 'native' option to 'make help'" from the previous version of the
patch still makes sense.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Reproducible work with natively compiled Emacs

[Pedro Andres Aranda Gutierrez]

[-- Attachment #1.1: Type: text/plain, Size: 3202 bytes --]

Hi Ihor,

Thanks for the feedback! I'm attaching a unified patch.

TODO: check that the emacs installed supports native compilation. This is
more for
make native.

Best, /PA


On Wed, 13 Mar 2024 at 13:46, Ihor Radchenko <yantar92@posteo.net> wrote:

> Pedro Andres Aranda Gutierrez <paaguti@gmail.com> writes:
>
> >> I have been experiencing some inconsistencies with org-mode, which I
> >> have been able to trace own to using native compilation and not knowing
> >> where the files from main were.
>
> Thanks for the patch!
> See my comments inline.
>
> > +# This is where Emacs stores the .eln files
> > +ELNDIR        = $(shell emacs --batch --eval '(princ (car
> native-comp-eln-load-path))')
>
> Please use $(EMACS).
>
> > +cleaneln::
> > +     for elnf in $(patsubst %.eln, %-*.eln, $(LISPN)); do \
> > +       find $(ELNDIR) -name $$elnf -exec $(RM) -v {} \; ;\
> > +     done
>
> $(FIND).
> Also, avoid using bash as much as possible in favor or GNU make
> constructs. You can use $(foreach ...) here.
>
> >  clean-install:
> >       if [ -d $(DESTDIR)$(lispdir) ] ; then \
> >         $(RM) $(DESTDIR)$(lispdir)/org*.el* $(DESTDIR)$(lispdir)/ob*.el*
> $(DESTDIR)$(lispdir)/ol*.el* $(DESTDIR)$(lispdir)/ox*.el* ; \
> > --
> > 2.34.1
> > From 6d98dfb3d72171e0231823260165b23e8b9963ee Mon Sep 17 00:00:00 2001
> > From: "Pedro A. Aranda" <paaguti@gmail.com>
> > Date: Mon, 11 Mar 2024 09:47:12 +0100
> > Subject: [PATCH 1/3] Add cleaneln target
> >
> > * mk/targets.mk: add the 'cleaneln' target
> >
> > ---
> >  mk/targets.mk | 3 +++
> >  1 file changed, 3 insertions(+)
> >
> > diff --git a/mk/targets.mk b/mk/targets.mk
> > index de849c4fb..ab9ec2a1f 100644
> > --- a/mk/targets.mk
> > +++ b/mk/targets.mk
> > @@ -143,6 +143,9 @@ cleanall: cleandirs cleantest
> >  $(CLEANDIRS:%=clean%):
> >       -$(FIND) $(@:clean%=%) \( -name \*~ -o -name \*.elc \) -exec $(RM)
> {} +
> >
> > +cleaneln:
> > +     $(MAKE) -C lisp $@
> > +
>
> Should be in .PHONY
>
> > --- a/etc/ORG-NEWS
> > +++ b/etc/ORG-NEWS
> > @@ -11,7 +11,8 @@ See the end of the file for license conditions.
> >
> >  Please send Org bug reports to mailto:emacs-orgmode@gnu.org.
> >
> > -* Version 9.7 (not released yet)
> > +* V
> > +ersion 9.7 (not released yet)
>
> This is a stray newline.
>
> > +*** ~make cleaneln~ will remove the .eln files from the user emacs
> configuration
> > +
> > +Natively compiled Emacs lisp files generated with ~make native~ are
> > +now correctly located and deleted with ~make cleaneln~.
> > +
> >  ** Miscellaneous
> >  *** =org-crypt.el= now applies initial visibility settings to decrypted
> entries
>
> "Add 'native' option to 'make help'" from the previous version of the
> patch still makes sense.
>
> --
> Ihor Radchenko // yantar92,
> Org mode contributor,
> Learn more about Org mode at <https://orgmode.org/>.
> Support Org development at <https://liberapay.com/org-mode>,
> or support my work at <https://liberapay.com/yantar92>
>


-- 
Fragen sind nicht da, um beantwortet zu werden,
Fragen sind da um gestellt zu werden
Georg Kreisler

Headaches with a Juju log:
unit-basic-16: 09:17:36 WARNING juju.worker.uniter.operation we should run
a leader-deposed hook here, but we can't yet

[-- Attachment #1.2: Type: text/html, Size: 5153 bytes --]

[-- Attachment #2: 0001-Clean-eln-files-from-the-user-emacs-eln-directory.patch --]
[-- Type: text/x-patch, Size: 3889 bytes --]

From 52bcbc71c62717e84dd0504ccf37d32668ed3cac Mon Sep 17 00:00:00 2001
From: "Pedro A. Aranda" <paaguti@gmail.com>
Date: Wed, 13 Mar 2024 17:41:56 +0100
Subject: [PATCH] Clean eln files from the user emacs eln directory

---
lisp/Makefile: clean .eln files from the directory where the user's natively
compiled files are stored

* Makefile: Add help for =make native=
* lisp/Makefile: Implement =make cleaneln=
* mk/targets.mk: Add =cleaneln= target


 Makefile      | 1 +
 etc/ORG-NEWS  | 7 +++++++
 lisp/Makefile | 9 ++++++++-
 mk/targets.mk | 5 ++++-
 4 files changed, 20 insertions(+), 2 deletions(-)

diff --git a/Makefile b/Makefile
index f476a3ea7..b3a167ecb 100644
--- a/Makefile
+++ b/Makefile
@@ -27,6 +27,7 @@ help helpall::
 	$(info make all            - ditto)
 	$(info make compile        - build Org ELisp files)
 	$(info make single         - build Org ELisp files, single Emacs per source)
+	$(info make native         - build Org natively compiled Elisp files)
 	$(info make autoloads      - create org-loaddefs.el to load Org in-place)
 	$(info make test           - build Org ELisp files and run test suite)
 	$(info make vanilla        - run Emacs with this Org-mode and no personal config)
diff --git a/etc/ORG-NEWS b/etc/ORG-NEWS
index ca73f06e7..cb8ed9b71 100644
--- a/etc/ORG-NEWS
+++ b/etc/ORG-NEWS
@@ -1281,6 +1281,13 @@ optional argument =NEW-HEADING-CONTAINER= specifies where in the
 buffer it will be added.  If not specified, new headings are created
 at level 1 at the end of the accessible part of the buffer, as before.

+*** ~make cleaneln~ will remove the .eln files from the user emacs configuration
+
+Natively compiled Emacs lisp files generated with ~make native~ are
+now correctly located and deleted with ~make cleaneln~.
+
+*** Add text for =make native= to ~make help~
+
 ** Miscellaneous
 *** =org-crypt.el= now applies initial visibility settings to decrypted entries

diff --git a/lisp/Makefile b/lisp/Makefile
index c570d9cfa..0186b4246 100644
--- a/lisp/Makefile
+++ b/lisp/Makefile
@@ -11,12 +11,15 @@ LISPF 	:= $(filter-out $(LISPA),$(sort $(wildcard *.el)))
 LISPC 	:= $(filter-out $(LISPB) $(LISPN:%el=%elc),$(LISPF:%el=%elc))
 LISPN 	:= $(filter-out $(LISPB) $(LISPN:%el=%eln),$(LISPF:%el=%eln))
 _ORGCM_ := dirall single native source slint1 slint2
+# This is where Emacs stores the .eln files
+ELNDIR        = $(shell $(EMACS) --batch --eval '(princ (car native-comp-eln-load-path))')
+
 -include local.mk

 .PHONY:	all compile compile-dirty \
 	$(_ORGCM_) $(_ORGCM_:%=compile-%) \
 	autoloads \
-	install clean cleanauto cleanall cleanelc clean-install
+	install clean cleanauto cleanall cleanelc cleaneln clean-install

 # do not clean here, done in toplevel make
 all compile compile-dirty::	 | autoloads
@@ -79,6 +82,10 @@ cleanauto clean cleanall::
 clean cleanall cleanelc::
 	$(RM) *.elc

+cleaneln::
+	@$(foreach elnf,$(patsubst %.eln, %-*.eln, $(LISPN)), \
+	  $(FIND) $(ELNDIR) -name $(elnf) -exec $(RM) -v {} \; ;)
+
 clean-install:
 	if [ -d $(DESTDIR)$(lispdir) ] ; then \
 	  $(RM) $(DESTDIR)$(lispdir)/org*.el* $(DESTDIR)$(lispdir)/ob*.el* $(DESTDIR)$(lispdir)/ol*.el* $(DESTDIR)$(lispdir)/ox*.el* ; \
diff --git a/mk/targets.mk b/mk/targets.mk
index de849c4fb..4ab5fa13c 100644
--- a/mk/targets.mk
+++ b/mk/targets.mk
@@ -31,7 +31,7 @@ endif
 	info html pdf card refcard doc docs \
 	autoloads cleanall clean $(CLEANDIRS:%=clean%) \
 	clean-install cleanelc cleandirs \
-	cleanlisp cleandoc cleandocs cleantest \
+	cleanlisp cleandoc cleandocs cleantest cleaneln \
 	compile compile-dirty uncompiled \
 	config config-test config-exe config-all config-eol config-version \
 	vanilla repro
@@ -143,6 +143,9 @@ cleanall: cleandirs cleantest
 $(CLEANDIRS:%=clean%):
 	-$(FIND) $(@:clean%=%) \( -name \*~ -o -name \*.elc \) -exec $(RM) {} +

+cleaneln:
+	$(MAKE) -C lisp $@
+
 cleanelc:
 	$(MAKE) -C lisp $@

--
2.34.1

Re: Reproducible work with natively compiled Emacs

[Ihor Radchenko]

Pedro Andres Aranda Gutierrez <paaguti@gmail.com> writes:

> Thanks for the feedback! I'm attaching a unified patch.

Thanks!
Some more comments.

> +# This is where Emacs stores the .eln files
> +ELNDIR        = $(shell $(EMACS) --batch --eval '(princ (car native-comp-eln-load-path))')

Maybe := is better here. Otherwise, we risk invoking emacs multiple times.

> +cleaneln::

Why ::?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Reproducible work with natively compiled Emacs

[Pedro Andres Aranda Gutierrez]

[-- Attachment #1.1: Type: text/plain, Size: 1797 bytes --]

Hi Ihor,

attached is the patch in what I would call the final version. The main
enhancement is that
make native and make cleaneln with exit gracefully on systems where native
compilation is not enabled.

Best, /PA

On Thu, 14 Mar 2024 at 07:49, Pedro Andres Aranda Gutierrez <
paaguti@gmail.com> wrote:

> HI,
>
> On Wed, 13 Mar 2024 at 18:13, Ihor Radchenko <yantar92@posteo.net> wrote:
>
>> Pedro Andres Aranda Gutierrez <paaguti@gmail.com> writes:
>>
>> > Thanks for the feedback! I'm attaching a unified patch.
>>
>> Thanks!
>> Some more comments.
>>
>> > +# This is where Emacs stores the .eln files
>> > +ELNDIR        = $(shell $(EMACS) --batch --eval '(princ (car
>> native-comp-eln-load-path))')
>>
>> Maybe := is better here. Otherwise, we risk invoking emacs multiple times.
>>
> You are right
>
>> > +cleaneln::
>>
>> Why ::?
>>
> You are absolutely right, it's an isolated rule and it doesn't need ::
>
>> --
>> Ihor Radchenko // yantar92,
>> Org mode contributor,
>> Learn more about Org mode at <https://orgmode.org/>.
>> Support Org development at <https://liberapay.com/org-mode>,
>> or support my work at <https://liberapay.com/yantar92>
>>
>
> Patch will follow. I'm testing it against an Emacs WITHOUT native-comp to
> check it there too.
>
> Best, /PA
> --
> Fragen sind nicht da, um beantwortet zu werden,
> Fragen sind da um gestellt zu werden
> Georg Kreisler
>
> Headaches with a Juju log:
> unit-basic-16: 09:17:36 WARNING juju.worker.uniter.operation we should run
> a leader-deposed hook here, but we can't yet
>
>

-- 
Fragen sind nicht da, um beantwortet zu werden,
Fragen sind da um gestellt zu werden
Georg Kreisler

Headaches with a Juju log:
unit-basic-16: 09:17:36 WARNING juju.worker.uniter.operation we should run
a leader-deposed hook here, but we can't yet

[-- Attachment #1.2: Type: text/html, Size: 3589 bytes --]

[-- Attachment #2: 0001-Make-cleaneln-work-execute-cleanly-when-native-comp-.patch --]
[-- Type: text/x-patch, Size: 4437 bytes --]

From 797a63581bac53f8eb9cba268a975807ec6b841e Mon Sep 17 00:00:00 2001
From: "Pedro A. Aranda" <paaguti@gmail.com>
Date: Thu, 14 Mar 2024 08:17:11 +0100
Subject: [PATCH] Make cleaneln work, execute cleanly when native comp not
 available.

---
lisp/Makefile: clean .eln files from the directory where the user's natively
compiled files are stored. Detect when native compilation is not available
and exit gracefully in that case

* Makefile: Add help for =make native=
* lisp/Makefile: Implement =make cleaneln=
* mk/targets.mk: Add =cleaneln= target


Makefile      |  1 +
 etc/ORG-NEWS  |  7 +++++++
 lisp/Makefile | 20 +++++++++++++++++++-
 mk/targets.mk |  5 ++++-
 4 files changed, 31 insertions(+), 2 deletions(-)

diff --git a/Makefile b/Makefile
index f476a3ea7..b3a167ecb 100644
--- a/Makefile
+++ b/Makefile
@@ -27,6 +27,7 @@ help helpall::
 	$(info make all            - ditto)
 	$(info make compile        - build Org ELisp files)
 	$(info make single         - build Org ELisp files, single Emacs per source)
+	$(info make native         - build Org natively compiled Elisp files)
 	$(info make autoloads      - create org-loaddefs.el to load Org in-place)
 	$(info make test           - build Org ELisp files and run test suite)
 	$(info make vanilla        - run Emacs with this Org-mode and no personal config)
diff --git a/etc/ORG-NEWS b/etc/ORG-NEWS
index ca73f06e7..cb8ed9b71 100644
--- a/etc/ORG-NEWS
+++ b/etc/ORG-NEWS
@@ -1281,6 +1281,13 @@ optional argument =NEW-HEADING-CONTAINER= specifies where in the
 buffer it will be added.  If not specified, new headings are created
 at level 1 at the end of the accessible part of the buffer, as before.

+*** ~make cleaneln~ will remove the .eln files from the user emacs configuration
+
+Natively compiled Emacs lisp files generated with ~make native~ are
+now correctly located and deleted with ~make cleaneln~.
+
+*** Add text for =make native= to ~make help~
+
 ** Miscellaneous
 *** =org-crypt.el= now applies initial visibility settings to decrypted entries

diff --git a/lisp/Makefile b/lisp/Makefile
index c570d9cfa..ff49dccff 100644
--- a/lisp/Makefile
+++ b/lisp/Makefile
@@ -11,12 +11,15 @@ LISPF 	:= $(filter-out $(LISPA),$(sort $(wildcard *.el)))
 LISPC 	:= $(filter-out $(LISPB) $(LISPN:%el=%elc),$(LISPF:%el=%elc))
 LISPN 	:= $(filter-out $(LISPB) $(LISPN:%el=%eln),$(LISPF:%el=%eln))
 _ORGCM_ := dirall single native source slint1 slint2
+# This is where Emacs stores the .eln files
+ELNDIR  := $(shell $(EMACS) --batch --eval '(when (native-comp-available-p) (princ (car native-comp-eln-load-path)))')
+
 -include local.mk

 .PHONY:	all compile compile-dirty \
 	$(_ORGCM_) $(_ORGCM_:%=compile-%) \
 	autoloads \
-	install clean cleanauto cleanall cleanelc clean-install
+	install clean cleanauto cleanall cleanelc cleaneln clean-install

 # do not clean here, done in toplevel make
 all compile compile-dirty::	 | autoloads
@@ -52,9 +55,15 @@ slint1:
 	@$(info Compiling single $(abspath $<)...)
 	-@$(ELC) $<

+ifeq (,$(ELNDIR))
+%.eln: %.el
+	@$(info Native compilation not enabled!)
+	@false
+else
 %.eln: %.el
 	@$(info Native compiling single $(abspath $<)...)
 	-@$(ELN) $<
+endif

 autoloads:	cleanauto $(LISPI) $(LISPV)

@@ -79,6 +88,15 @@ cleanauto clean cleanall::
 clean cleanall cleanelc::
 	$(RM) *.elc

+ifeq (,$(ELNDIR))
+cleaneln:
+	@$(info Native compilation not enabled. Nothing to do.)
+else
+cleaneln:
+	@$(foreach elnf,$(patsubst %.eln, %-*.eln, $(LISPN)), \
+	  $(FIND) $(ELNDIR) -name $(elnf) -exec $(RM) -v {} \; ;)
+endif
+
 clean-install:
 	if [ -d $(DESTDIR)$(lispdir) ] ; then \
 	  $(RM) $(DESTDIR)$(lispdir)/org*.el* $(DESTDIR)$(lispdir)/ob*.el* $(DESTDIR)$(lispdir)/ol*.el* $(DESTDIR)$(lispdir)/ox*.el* ; \
diff --git a/mk/targets.mk b/mk/targets.mk
index de849c4fb..4ab5fa13c 100644
--- a/mk/targets.mk
+++ b/mk/targets.mk
@@ -31,7 +31,7 @@ endif
 	info html pdf card refcard doc docs \
 	autoloads cleanall clean $(CLEANDIRS:%=clean%) \
 	clean-install cleanelc cleandirs \
-	cleanlisp cleandoc cleandocs cleantest \
+	cleanlisp cleandoc cleandocs cleantest cleaneln \
 	compile compile-dirty uncompiled \
 	config config-test config-exe config-all config-eol config-version \
 	vanilla repro
@@ -143,6 +143,9 @@ cleanall: cleandirs cleantest
 $(CLEANDIRS:%=clean%):
 	-$(FIND) $(@:clean%=%) \( -name \*~ -o -name \*.elc \) -exec $(RM) {} +

+cleaneln:
+	$(MAKE) -C lisp $@
+
 cleanelc:
 	$(MAKE) -C lisp $@

--
2.34.1

Re: Reproducible work with natively compiled Emacs

[Ihor Radchenko]

Pedro Andres Aranda Gutierrez <paaguti@gmail.com> writes:

> +# This is where Emacs stores the .eln files
> +ELNDIR  := $(shell $(EMACS) --batch --eval '(when (native-comp-available-p) (princ (car native-comp-eln-load-path)))')
> ...
> +cleaneln:
> +	@$(foreach elnf,$(patsubst %.eln, %-*.eln, $(LISPN)), \
> +	  $(FIND) $(ELNDIR) -name $(elnf) -exec $(RM) -v {} \; ;)

I think that it will delete too much - (car native-comp-eln-load-path)
contains multiple directories if several Emacs versions are used in the
system. So, make cleaneln will not only clean .eln files created by make
native, but also all .eln files created by other Emacs processes.

For reference, here is what my eln-cache directory contains:

[yantar92:~/.emacs.d/eln-cache] $ 
> ls
28.2-288c4fd1  30.0.50-394f34f6  30.0.50-464b3753  30.0.50-4f12c581  30.0.50-8437a6f7  30.0.50-86834af4

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Reproducible work with natively compiled Emacs

[Pedro Andres Aranda Gutierrez]

[-- Attachment #1: Type: text/plain, Size: 1948 bytes --]

HI Igor
having more than one 30.0.50 means that you are trying emacs master as
often as me. Only that I tend to cleanup more often :-D

I think I could make it delete the current version only.
It will not delete the org .eln files shipped with a package you install:
that's what the first line in the foreach and (car ...) take care off,
so it will be only anything created at some point by a make native on an
org-mode cloned from git. Anyhow, let me check...

Best, /PA

/PA

On Fri, 15 Mar 2024 at 12:09, Ihor Radchenko <yantar92@posteo.net> wrote:

> Pedro Andres Aranda Gutierrez <paaguti@gmail.com> writes:
>
> > +# This is where Emacs stores the .eln files
> > +ELNDIR  := $(shell $(EMACS) --batch --eval '(when
> (native-comp-available-p) (princ (car native-comp-eln-load-path)))')
> > ...
> > +cleaneln:
> > +     @$(foreach elnf,$(patsubst %.eln, %-*.eln, $(LISPN)), \
> > +       $(FIND) $(ELNDIR) -name $(elnf) -exec $(RM) -v {} \; ;)
>
> I think that it will delete too much - (car native-comp-eln-load-path)
> contains multiple directories if several Emacs versions are used in the
> system. So, make cleaneln will not only clean .eln files created by make
> native, but also all .eln files created by other Emacs processes.
>
> For reference, here is what my eln-cache directory contains:
>
> [yantar92:~/.emacs.d/eln-cache] $
> > ls
> 28.2-288c4fd1  30.0.50-394f34f6  30.0.50-464b3753  30.0.50-4f12c581
> 30.0.50-8437a6f7  30.0.50-86834af4
>
> --
> Ihor Radchenko // yantar92,
> Org mode contributor,
> Learn more about Org mode at <https://orgmode.org/>.
> Support Org development at <https://liberapay.com/org-mode>,
> or support my work at <https://liberapay.com/yantar92>
>


-- 
Fragen sind nicht da, um beantwortet zu werden,
Fragen sind da um gestellt zu werden
Georg Kreisler

Headaches with a Juju log:
unit-basic-16: 09:17:36 WARNING juju.worker.uniter.operation we should run
a leader-deposed hook here, but we can't yet

[-- Attachment #2: Type: text/html, Size: 3011 bytes --]

Re: Reproducible work with natively compiled Emacs

[Ihor Radchenko]

Pedro Andres Aranda Gutierrez <paaguti@gmail.com> writes:

> I have added the eln version to the patch...
>
> Best, /PA
> PS: Just as an example, I recompiled master today and the version number
> for eln changed, so the 'old' files from yesterday were not removed ;-)

Yeah. Not ideal.
The whole system with emacs -Q putting things into .emacs.d is not ideal.

Sometimes, when running things like make test we do not even want to
litter .emacs.d - .eln files generated during make test might correspond
to the broken versions of Org mode that were being tested. Then, running
actual working Emacs session might stumble upon these incorrect versions
of .eln files.

Do I understand correctly that the reason you implemented cleaneln make
target is working around issues with make test/make repro?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Reproducible work with natively compiled Emacs

[Pedro Andres Aranda Gutierrez]

[-- Attachment #1: Type: text/plain, Size: 1854 bytes --]

Hi Ihor,

Answer inline...

On Fri, 15 Mar 2024 at 18:08, Ihor Radchenko <yantar92@posteo.net> wrote:

> Pedro Andres Aranda Gutierrez <paaguti@gmail.com> writes:
>
> > I have added the eln version to the patch...
> >
> > Best, /PA
> > PS: Just as an example, I recompiled master today and the version number
> > for eln changed, so the 'old' files from yesterday were not removed ;-)
>
> Yeah. Not ideal.
> The whole system with emacs -Q putting things into .emacs.d is not ideal.
>
> Sometimes, when running things like make test we do not even want to
> litter .emacs.d - .eln files generated during make test might correspond
> to the broken versions of Org mode that were being tested. Then, running
> actual working Emacs session might stumble upon these incorrect versions
> of .eln files.
>
> Do I understand correctly that the reason you implemented cleaneln make
> target is working around issues with make test/make repro?
>

Yes, that's one of the reasons. And, also because when I set
native.comp-eln-cache-path,
anything that is not part of the Emacs distribution gets compiled into that
directory. For example,
the clone of org-mode main as well as the packages from elpa/melpa/nungnu.

When refreshing my local copy of the org-mode repo, I start with a make
cleaneln before pulling and then
make native. Thus I get a predictable build.


> --
> Ihor Radchenko // yantar92,
> Org mode contributor,
> Learn more about Org mode at <https://orgmode.org/>.
> Support Org development at <https://liberapay.com/org-mode>,
> or support my work at <https://liberapay.com/yantar92>
>

Best, /PA

-- 
Fragen sind nicht da, um beantwortet zu werden,
Fragen sind da um gestellt zu werden
Georg Kreisler

Headaches with a Juju log:
unit-basic-16: 09:17:36 WARNING juju.worker.uniter.operation we should run
a leader-deposed hook here, but we can't yet

[-- Attachment #2: Type: text/html, Size: 3066 bytes --]

Re: Reproducible work with natively compiled Emacs

[Ihor Radchenko]

Pedro Andres Aranda Gutierrez <paaguti@gmail.com> writes:

>> Do I understand correctly that the reason you implemented cleaneln make
>> target is working around issues with make test/make repro?
>>
> Yes, that's one of the reasons. And, also because when I set
> native.comp-eln-cache-path,
> anything that is not part of the Emacs distribution gets compiled into that
> directory. For example,
> the clone of org-mode main as well as the packages from elpa/melpa/nungnu.

Sorry, but I do not fully understand.
May you please explain in more details what kind of problems you
encountered in practice?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Reproducible work with natively compiled Emacs

[Pedro Andres Aranda Gutierrez]

[-- Attachment #1: Type: text/plain, Size: 1638 bytes --]

Hi Ihor.

In practice, I was not able to delete the .eln files from a make native.

In order to have a more controlled environment, I delete them _before_
I refresh my local org-mode/main directory, and then do a make native
after refreshing my local copy.

Same happened when testing modifications. When testing a modification
I always make cleaneln; make native to test it

Maybe I'm a bit too 'meticulous' but that's me ;-)

Best, /PA

On Sat, 16 Mar 2024 at 11:20, Ihor Radchenko <yantar92@posteo.net> wrote:

> Pedro Andres Aranda Gutierrez <paaguti@gmail.com> writes:
>
> >> Do I understand correctly that the reason you implemented cleaneln make
> >> target is working around issues with make test/make repro?
> >>
> > Yes, that's one of the reasons. And, also because when I set
> > native.comp-eln-cache-path,
> > anything that is not part of the Emacs distribution gets compiled into
> that
> > directory. For example,
> > the clone of org-mode main as well as the packages from
> elpa/melpa/nungnu.
>
> Sorry, but I do not fully understand.
> May you please explain in more details what kind of problems you
> encountered in practice?
>
> --
> Ihor Radchenko // yantar92,
> Org mode contributor,
> Learn more about Org mode at <https://orgmode.org/>.
> Support Org development at <https://liberapay.com/org-mode>,
> or support my work at <https://liberapay.com/yantar92>
>


-- 
Fragen sind nicht da, um beantwortet zu werden,
Fragen sind da um gestellt zu werden
Georg Kreisler

Headaches with a Juju log:
unit-basic-16: 09:17:36 WARNING juju.worker.uniter.operation we should run
a leader-deposed hook here, but we can't yet

[-- Attachment #2: Type: text/html, Size: 2739 bytes --]

Re: Reproducible work with natively compiled Emacs

[Max Nikulin]

On 17/03/2024 13:13, Pedro Andres Aranda Gutierrez wrote:
> 
> In practice, I was not able to delete the .eln files from a make native.
> 
> In order to have a more controlled environment, I delete them _before_
> I refresh my local org-mode/main directory, and then do a make native
> after refreshing my local copy.
> 
> Same happened when testing modifications. When testing a modification
> I always make cleaneln; make native to test it

In the past I read a couple of threads on native compilation on 
emacs-devel and maybe a couple of bug reports. My impression that the 
position of developers in response to requests to give more control on 
native compilation is "it should just work and users should not bother".

Do you know a reproducible way leading to errors when .eln files are not 
removed?

Re: Reproducible work with natively compiled Emacs

[Ihor Radchenko]

Pedro Andres Aranda Gutierrez <paaguti@gmail.com> writes:

> In practice, I was not able to delete the .eln files from a make native.

I am wondering why you wanted to run make native.
When I added that target, it was mostly to test inconsistencies between
make single and make native. However, AFAIU, there should be no
inconsistencies in practice. So, maybe we can instead just delete make
native target? Or is there any value in ahead of time native-compilation
when working with Org git repo?

> In order to have a more controlled environment, I delete them _before_
> I refresh my local org-mode/main directory, and then do a make native
> after refreshing my local copy.
>
> Same happened when testing modifications. When testing a modification
> I always make cleaneln; make native to test it
>
> Maybe I'm a bit too 'meticulous' but that's me ;-)

"more controlled environment" does not sound like a real concern caused
by something breaking. I am joining Max's question on whether you
encountered any real issue with native compilation.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Reproducible work with natively compiled Emacs

[Pedro Andres Aranda Gutierrez]

Hi

Bluntly speaking, yes. There is this instance not too long ago where we had the slow down and I was trying to isolate the source. One of my possible suspects was that I might not have the right version of the eln file, because the creation timestamp was seeing with ls-l really made me doubt and I needed a state I could understand.

Best,/PA

Enviado desde mi iPhone

> El 17 mar 2024, a las 11:16, Ihor Radchenko <yantar92@posteo.net> escribió:
> 
> Pedro Andres Aranda Gutierrez <paaguti@gmail.com> writes:
> 
>> In practice, I was not able to delete the .eln files from a make native.
> 
> I am wondering why you wanted to run make native.
> When I added that target, it was mostly to test inconsistencies between
> make single and make native. However, AFAIU, there should be no
> inconsistencies in practice. So, maybe we can instead just delete make
> native target? Or is there any value in ahead of time native-compilation
> when working with Org git repo?
> 
>> In order to have a more controlled environment, I delete them _before_
>> I refresh my local org-mode/main directory, and then do a make native
>> after refreshing my local copy.
>> 
>> Same happened when testing modifications. When testing a modification
>> I always make cleaneln; make native to test it
>> 
>> Maybe I'm a bit too 'meticulous' but that's me ;-)
> 
> "more controlled environment" does not sound like a real concern caused
> by something breaking. I am joining Max's question on whether you
> encountered any real issue with native compilation.
> 
> --
> Ihor Radchenko // yantar92,
> Org mode contributor,
> Learn more about Org mode at <https://orgmode.org/>.
> Support Org development at <https://liberapay.com/org-mode>,
> or support my work at <https://liberapay.com/yantar92>

Re: Reproducible work with natively compiled Emacs

[Ihor Radchenko]

Pedro Andres Aranda Gutierrez <paaguti@gmail.com> writes:

> Bluntly speaking, yes. There is this instance not too long ago where we had the slow down and I was trying to isolate the source. One of my possible suspects was that I might not have the right version of the eln file, because the creation timestamp was seeing with ls-l really made me doubt and I needed a state I could understand.

Many things could cause it. For example, I sometimes get very bad
slowdowns unless I do make bootstrap on my Emacs master build.

I conclude that cleaning .eln files, especially outside the Org git repo
folder, is not something we need. (until there is an evidence that we do
need such a cleanup)

I think that we can still keep make native exclusively for the purposes
of testing native compilation in case if it behaves funnily.

Canceled.
I only committed the changes to make help output.
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=67a8117ca

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>