[bug#67260] [PATCH v6 1/7] gnu: emacs: Wrap EMACSNATIVELOADPATH.

[bug#67260] [PATCH v6 1/7] gnu: emacs: Wrap EMACSNATIVELOADPATH.

[Liliana Marie Prikler]

Am Sonntag, dem 28.01.2024 um 16:17 +0000 schrieb Suhail:
> I'm assuming you're able to reproduce the issue on your end as well
> with
> other features like prog-mode, backquote etc.
> 
> Liliana Marie Prikler <liliana.prikler@gmail.com> writes:
> 
> > I instrumented lread.c to print the relative and absolute file
> > names tried, and it appears that these aren't loaded at all after
> > install while EMACSNATIVELOADPATH is ignored when dumping.
> 
> Interesting.  I did a grep of the Emacs source (branch emacs-29) and
> line 1770 of src/lread.c (in maybe_swap_for_eln) seemed relevant:
> 
> >   /* Look also in preloaded subfolder of the last entry in
> >      `comp-eln-load-path'.  */
> 
> What surprised me was that not only was the "preloaded" directory
> special-cased, but so too was the last entry of "comp-eln-load-path"
> (which I imagine is referring to native-comp-eln-load-path).  Judging
> by your message, I'm guessing you're well aware of this.  I,
> unfortunately, have no additional insights or suggestions.
> 
> However, given that ensuring Emacs is able to find the natively-
> compiled version of files seems non-trivial, it might help for a
> future patch series to be accompanied with tests.  Thoughts?
On upstream, this would add ../native-lisp/preloaded to the search
(since everything is dropped in one directory, this probably works as
expected), but with our patch, we do multiple directories *and* drop
../native-lisp from that early.  We could try expanding
EMACSNATIVELOADPATH earlier, but we can't yet use Fgetenv_internal.

But yeah, we're looking at the same source file right now.

Cheers

[bug#67260] [PATCH v6 1/7] gnu: emacs: Wrap EMACSNATIVELOADPATH.

[Liliana Marie Prikler]

Am Sonntag, dem 28.01.2024 um 00:13 +0000 schrieb Suhail:
> Liliana Marie Prikler <liliana.prikler@gmail.com> writes:
> [...]
> > but ‘uniquify’ remains byte-compiled for no explainable reason.
> 
> And too that as of v8 patch, uniquify still isn't natively-compiled.
> However, if you unload it and then reload it, it becomes
> natively-compiled.
> 
> #+begin_src bash
>   emacs --batch --eval "(message \"%s\" (progn (unload-feature
> 'uniquify) (require 'uniquify) (take 1 (split-string (substring-no-
> properties (describe-function 'uniquify-item-p)) \"\\n\" t))))"
> #+end_src
> 
> #+RESULTS:
> : Type q in help window to delete it
> : (uniquify-item-p is a native-compiled Lisp function in
> `uniquify.el'.)
> 
> 
> I think the issue is because uniquify (along with some others such as
> prog-mode, backquote etc that I tested) are preloaded in Emacs.  I
> believe that these preloaded packages need to be treated specially. 
> In a non-Guix system, these preloaded packages have their .eln files
> stored under something like
> <native-comp-eln-load-path-entry>/29.x-<hash>/preloaded/ , whereas in
> the v8 series there is no "preloaded" directory.  Instead .eln files
> for packages such as uniquify, prog-mode, backquote etc. are in
> various locations:
> 
> - uniquify :: <n-c-e-l-p-entry>/29.2-<hash>/uniquify.eln
> - prog-mode :: <n-c-e-l-p-entry>/29.2-<hash>/progmodes/prog-mode.eln
> - backquote :: <n-c-e-l-p-entry>/29.2-<hash>/emacs-lisp/backquote.eln
> 
> I suspect that the code which does the preloading may need to be
> patched and/or the preloaded packages may need to have their .eln
> files generated under a preloaded sub-directory (similar to what
> happens in non-Guix systems).
From my experiments, the installing of these files is a red herring.  I
instrumented lread.c to print the relative and absolute file names
tried, and it appears that these aren't loaded at all after install
while EMACSNATIVELOADPATH is ignored when dumping.

Cheers

[bug#67260] [PATCH v6 1/7] gnu: emacs: Wrap EMACSNATIVELOADPATH.

[Liliana Marie Prikler]

Am Samstag, dem 27.01.2024 um 19:39 +0000 schrieb Suhail:
> Suhail via Guix-patches via <guix-patches@gnu.org> writes:
> 
> > However, the 2 issues pointed out in the previous email remain:
> 
> Correction, there's only 1 issue.  The first issue was a false
> positive.  Apologies for the noise.
> 
> > 1. failure to load natively-compiled version of non-builtin
> > libraries like emacs-htmlize possibly due to symlinks, and
> 
> This was a false positive.  Turns out, if the function in question is
> only autoloaded describe-function won't correctly identify the
> function as being natively-compiled.  This is consistent with what
> happens on Emacs 29.1 on a non-Guix system.  Not an issue.
Interesting, but probably not avoidable.  There's only so much
information you can put into autoloads.

> > 2. failure to load natively-compiled version of some builtin
> > libraries like uniquify due to incorrect naming of .eln files
> 
> This issue remains as of v7 of the patch series.
I fixed this one locally, but ‘uniquify’ remains byte-compiled for no
explainable reason.  Will send v8 soon.

Cheer

[bug#67260] [PATCH v6 1/7] gnu: emacs: Wrap EMACSNATIVELOADPATH.

[Liliana Marie Prikler]

Am Samstag, dem 27.01.2024 um 17:15 +0000 schrieb Suhail:
> Liliana Marie Prikler <liliana.prikler@gmail.com> writes:
> 
> > Would you care to debug this a bit more and report your findings?
> 
> The locations in question are in the store and hence write-protected.
> What's a "safe" way to debug this?
> 
> > > Packages such as ox-html are available from two locations in the
> > > native-comp-eln-load-path.  In one location they are symlinked,
> > > but (presumably because org is a builtin package) it is also
> > > available from another location in the native-comp-eln-load-path
> > > and in the latter location it's not symlinked in.  I believe this
> > > difference is why for some packages natively-compiled versions
> > > are loaded (e.g. org, ox-html, etc) whereas for others it's not
> > > the case.
> > Well, as pointed out in the deleted code, dlopen has a "one shared
> > library per file name" limitation.  I don't think we're hit by that
> > thanks to unique hashes in the store directory, but I might be
> > wrong about that.  Maybe we have to do java-style FQDNs instead.
> 
> Perhaps.  I'd wondered, more simply, if symlinked .eln files were
> ever being loaded.  In the limited testing I did, I didn't find such
> an instance.  For what it's worth, only one of the entries in
> native-comp-eln-load-path seems to have symlinked entries (namely,
> ~/.guix-profile/lib/emacs/native-site-lisp).
You could try stepping through loading a natively-compiled, but
symlinked library, either with the emacs debugger or gdb.  I sadly
don't have more hints to give; you probably know more about this bug
than I do.

Alternatively, you could try union-building emacs + your libraries and
resolving those symlinks as you do.  This should give us a clear hint
that it's the symlinks and not something else that goes wrong.

In addition, you might want to verify that comp-el-to-*-filename
matches the files you want to load.  We could try resolving symlinks in
there with realpath as well.

> > > In addition, I believe there's another issue.  Some packages'
> > > names are getting  truncated.  For instance, instead of
> > > uniquify.eln, I observe niquify.eln.  In this case, the .eln
> > > isn't symlinked (presumably because it's a builtin), but due to
> > > the name being messed up (perhaps too aggressive a truncation of
> > > hashes?) the natively compiled version is not available:
> > I am not truncating any hashes, I'm not even computing them in the
> > first place.  The functions I'm modifying are publicly callable,
> > namely comp-el-to-eln-rel-filename for the relative file names, and
> > comp-el-to-eln-filename for the absolute ones.
> > 
> > There could be an off-by-one error hidden in the stripping of the
> > BOGUS_DIRS, however.  Let's investigate that.
> 
> Were you thinking out loud, or are there specific steps you'd like me
> to help with?
That last sentence is me thinking loud for a solution that I'll try for
v8.

Cheers

[bug#67260] [PATCH v6 1/7] gnu: emacs: Wrap EMACSNATIVELOADPATH.

[Liliana Marie Prikler]

Am Samstag, dem 27.01.2024 um 15:36 +0000 schrieb Suhail:
> Liliana Marie Prikler <liliana.prikler@gmail.com> writes:
> […]
> For emacs-htmlize, I believe the issue may stem from the fact that
> some .eln files are available from entries in native-comp-eln-load-
> path only via symlinks and the .eln loader may not be able to load
> symlinks.  
Would you care to debug this a bit more and report your findings?

> Packages such as ox-html are available from two locations in the
> native-comp-eln-load-path.  In one location they are symlinked, but
> (presumably because org is a builtin package) it is also available
> from another location in the native-comp-eln-load-path and in the
> latter location it's not symlinked in.  I believe this difference is
> why for some packages natively-compiled versions are loaded (e.g.
> org, ox-html, etc) whereas for others it's not the case.
Well, as pointed out in the deleted code, dlopen has a "one shared
library per file name" limitation.  I don't think we're hit by that
thanks to unique hashes in the store directory, but I might be wrong
about that.  Maybe we have to do java-style FQDNs instead.

> In addition, I believe there's another issue.  Some packages' names
> are getting  truncated.  For instance, instead of uniquify.eln, I
> observe niquify.eln.  In this case, the .eln isn't symlinked
> (presumably because it's a builtin), but due to the name being messed
> up (perhaps too aggressive a truncation of hashes?) the natively
> compiled version is not available:
I am not truncating any hashes, I'm not even computing them in the
first place.  The functions I'm modifying are publicly callable, namely
comp-el-to-eln-rel-filename for the relative file names, and
comp-el-to-eln-filename for the absolute ones.

There could be an off-by-one error hidden in the stripping of the
BOGUS_DIRS, however.  Let's investigate that.

> > uniquify-item-p is a byte-compiled Lisp function in `uniquify.el'.
> 
> The manifest file that was used is below.
> 
> #+begin_src scheme
>   (use-modules (guix transformations)
>                (gnu packages))
> 
>   (define transform1
>     (options->transformation
>      '((with-input . "emacs-minimal=emacs")
>        (without-tests . "emacs-clojure-mode"))))
> 
>   (packages->manifest
>     (list (transform1 (specification->package "bash"))
>           (transform1 (specification->package "coreutils"))
>           (transform1 (specification->package "git"))
>           (transform1 (specification->package "make"))
>           (transform1 (specification->package "emacs"))
>           (transform1 (specification->package "emacs-org"))
>           (transform1
>             (specification->package "emacs-org-contrib"))
>           (transform1
>             (specification->package "emacs-citeproc-el"))
>           (transform1
>             (specification->package "emacs-engrave-faces"))
>           (transform1
>             (specification->package "emacs-haskell-mode"))
>           (transform1
>             (specification->package "emacs-htmlize"))
>           (transform1
>             (specification->package "emacs-markdown-mode"))
>           (transform1
>             (specification->package
>               "emacs-org-pandoc-import"))
>           (transform1 (specification->package "pandoc"))
>           ))
> #+end_src
Note, that you can map transform1, saving some typing overhead, and
since you are transforming all of your packages you could compose that
with specification->package.

Cheers

[bug#67260] [PATCH v6 1/7] gnu: emacs: Wrap EMACSNATIVELOADPATH.

[Liliana Marie Prikler]

Am Freitag, dem 26.01.2024 um 22:45 +0000 schrieb Suhail:
> Liliana Marie Prikler <liliana.prikler@gmail.com> writes:
> 
> > > Btw, does every package require something like [PATCH v6 5/7] or
> > > [PATCH v6 6/7] in order for it to be natively compiled?
> > No, these patches are only for natively compiled packages that
> > don't use emacs-build-system.
> 
> I.e., any and all packages using the emacs-build-system should be
> natively-compiled?  If so, I believe I've found a counter-example.
No, but packages using emacs-build-system should not need any changes
to accomodate the currently existing hack of adding the option 
  --with-input=emacs-minimal=emacs
to ‘guix build’, ‘guix package’, ‘guix shell’, etc.

> After installing v6 series and building emacs-htmlize my expectation
> is that functions provided by it, such as htmlize-buffer, would be
> natively-compiled.  However, I get:
> 
> > htmlize-buffer is an autoloaded interactive Lisp function in
> > `htmlize.el'.
> 
> Note the lack of "native-compiled" in the above.
If you compile with emacs-minimal, that's the expected results. 
Different builds of emacs are still incompatible w.r.t. native code and
produce directories with different hashes.  We'll have to see as to
whether this bugs grafts, but I'm hoping it doesn't.

Cheers

[bug#67260] [PATCH v6 1/7] gnu: emacs: Wrap EMACSNATIVELOADPATH.

[Liliana Marie Prikler]

Am Freitag, dem 26.01.2024 um 08:20 +0000 schrieb Suhail:
> Suhail via Guix-patches via <guix-patches@gnu.org> writes:
> 
> > hopefully the patch series can soon be merged into master for
> > general availability.
> 
> Btw, does every package require something like [PATCH v6 5/7] or
> [PATCH v6 6/7] in order for it to be natively compiled?
No, these patches are only for natively compiled packages that don't
use emacs-build-system.  That being said, this is likely an incomplete
list.

Cheers

[bug#67260] [PATCH v6 1/7] gnu: emacs: Wrap EMACSNATIVELOADPATH.

[Liliana Marie Prikler]

Hi,

Am Donnerstag, dem 25.01.2024 um 23:53 +0000 schrieb Suhail:
> Liliana Marie Prikler <liliana.prikler@gmail.com> writes:
> 
> > > Suhail via Guix-patches via <guix-patches@gnu.org> writes:
> > > 
> > > > I don't believe this patch is sufficient.  It seems that the
> > > > variable 'native-comp-eln-load-path may also need to be
> > > > tweaked.
> > > > The path to the "native-lisp" directory needs to be specified
> > > > absolutely to make it not be dependent on the invocation-
> > > > directory.
> > > 
> > ... I've managed to resolve it
> 
> I can confirm that Emacs built by installing v6 patches on emacs-team
> branch and built with --no-grafts is now able to locate
> natively-compiled versions of features like help-fns etc. (which it
> couldn't with v5 patches).
> 
> #+begin_src bash :results replace
>   emacs --batch --eval "(message \"%s\" (car (split-string (describe-
> function 'describe-function) \"\n\")))"
> #+end_src
> 
> #+RESULTS:
> : Type q in help window to delete it
> : describe-function is an autoloaded interactive native-compiled Lisp
> 
> 
> However, there may be a spurious entry in native-comp-eln-load-path
> that should probably be removed.
> 
> #+begin_src bash :results replace
>   emacs --batch --eval "(message \"%s\" native-comp-eln-load-path)"
> #+end_src
> 
> #+RESULTS:
> : (/home/user/.emacs.d/eln-cache/ /home/user/.guix-
> profile/lib/emacs/native-site-lisp
> /gnu/store/4jvap9wxifizm56p6pmxc6rk0zyvm5zc-emacs-
> 29.2/lib/emacs/29.2/native-lisp ../native-lisp/)
> 
> 
> The "../native-lisp/" entry above, seems out of place.  Based on the
> documentation of native-comp-eln-load-path:
> 
> > If the name of a directory in this list is not absolute, it is
> > assumed to be relative to invocation-directory.
> 
> Contrasting the above with an Emacs 29.1 installation on a non-Guix
> system, I notice that the latter doesn't have any non-absolute paths
> in native-comp-eln-load-path.  Thoughts?
I have noticed this myself, but I don't think that it's related to this
series.  Can you try 29.2 without the patches?  It might be related to
that bump.

Cheers

[bug#67260] [PATCH emacs-team 0/2] Think ahead when compiling

[Liliana Marie Prikler]

Hi Guix,

this series (hopefully) makes it so that everything we need to be natively
compiled in Emacs a) is natively compiled, and b) is found in the right
location.  Please check that you no longer get gratuitous writes to your
local eln-cache when trying this out.

Cheers

Fixes: emacs-build-system … mismatching hashes <https://bugs.gnu.org/66864>

Liliana Marie Prikler (2):
  gnu: emacs: Build trampolines.
  gnu: emacs: Don't hash file names in native compilation.

 gnu/local.mk                                  |  1 +
 gnu/packages/emacs.scm                        |  6 +-
 .../emacs-native-comp-fix-filenames.patch     | 93 +++++++++++++++++++
 3 files changed, 99 insertions(+), 1 deletion(-)
 create mode 100644 gnu/packages/patches/emacs-native-comp-fix-filenames.patch


base-commit: 60c97924e9519361494aaf0686e28eb831a42315
-- 
2.41.0

[bug#67260] [PATCH v6 1/7] gnu: emacs: Wrap EMACSNATIVELOADPATH.

[Liliana Marie Prikler]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 2759 bytes --]

* gnu/packages/emacs.scm (emacs-minimal)[wrap-emacs-paths]: Also wrap
EMACSNATIVELOADPATH.
---
Am Montag, dem 22.01.2024 um 04:36 +0000 schrieb Suhail:
> Suhail via Guix-patches via <guix-patches@gnu.org> writes:
> 
> > I don't believe this patch is sufficient.  It seems that the
> > variable 'native-comp-eln-load-path may also need to be tweaked.
> > The path to the "native-lisp" directory needs to be specified
> > absolutely to make it not be dependent on the invocation-directory.
> 
> Or it's possible that a patch that ensures that
> 'native-comp-eln-load-path contains the absolute value of the
> "native-lisp" directory obviates this one (6/6) in its entirety.
This was the most cryptic hint towards the actual issue, but I've
managed to resolve it anyhow.  We still shouldn't enable jit compilation
however; I've updated our reason as to why.

 gnu/packages/emacs.scm | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)

diff --git a/gnu/packages/emacs.scm b/gnu/packages/emacs.scm
index 5f27c551e0..2cdc9b8bca 100644
--- a/gnu/packages/emacs.scm
+++ b/gnu/packages/emacs.scm
@@ -273,7 +273,11 @@ (define-public emacs-minimal
               (let* ((out (assoc-ref outputs "out"))
                      (lisp-dirs (find-files (string-append out "/share/emacs")
                                             "^lisp$"
-                                            #:directories? #t)))
+                                            #:directories? #t))
+                     (native-lisp-dirs (find-files
+                                        (string-append out "/lib/emacs")
+                                        "^native-lisp$"
+                                        #:directories? #t)))
                 (for-each
                  (lambda (prog)
                    (wrap-program prog
@@ -285,7 +289,11 @@ (define-public emacs-minimal
                              (list (search-input-file inputs "/bin/gzip")
                                    ;; for coreutils
                                    (search-input-file inputs "/bin/yes"))))
-                     `("EMACSLOADPATH" suffix ,lisp-dirs)))
+                     `("EMACSLOADPATH" suffix ,lisp-dirs)
+                     ;; Note: the interpretation order of EMACSNATIVELOADPATH
+                     ;; is reversed, so 'prefix functions just like 'suffix
+                     ;; for EMACSLOADPATH.
+                     `("EMACSNATIVELOADPATH" prefix ,native-lisp-dirs)))
                  (find-files (string-append out "/bin")
                              ;; Matches versioned and unversioned emacs binaries.
                              ;; We don't patch emacsclient, because it takes its

base-commit: aae61f54ff6acf5cc0e0355dc85babf29f625660
-- 
2.41.0

[bug#67260] [PATCH v6 1/7] gnu: emacs: Wrap EMACSNATIVELOADPATH.

[Suhail via Guix-patches via]

Liliana Marie Prikler <liliana.prikler@gmail.com> writes:

>> Suhail via Guix-patches via <guix-patches@gnu.org> writes:
>> 
>> > I don't believe this patch is sufficient.  It seems that the
>> > variable 'native-comp-eln-load-path may also need to be tweaked.
>> > The path to the "native-lisp" directory needs to be specified
>> > absolutely to make it not be dependent on the invocation-directory.
>> 
> ... I've managed to resolve it

I can confirm that Emacs built by installing v6 patches on emacs-team
branch and built with --no-grafts is now able to locate
natively-compiled versions of features like help-fns etc. (which it
couldn't with v5 patches).

#+begin_src bash :results replace
  emacs --batch --eval "(message \"%s\" (car (split-string (describe-function 'describe-function) \"\n\")))"
#+end_src

#+RESULTS:
: Type q in help window to delete it
: describe-function is an autoloaded interactive native-compiled Lisp


However, there may be a spurious entry in native-comp-eln-load-path that
should probably be removed.

#+begin_src bash :results replace
  emacs --batch --eval "(message \"%s\" native-comp-eln-load-path)"
#+end_src

#+RESULTS:
: (/home/user/.emacs.d/eln-cache/ /home/user/.guix-profile/lib/emacs/native-site-lisp /gnu/store/4jvap9wxifizm56p6pmxc6rk0zyvm5zc-emacs-29.2/lib/emacs/29.2/native-lisp ../native-lisp/)


The "../native-lisp/" entry above, seems out of place.  Based on the
documentation of native-comp-eln-load-path:

> If the name of a directory in this list is not absolute, it is assumed
> to be relative to invocation-directory.

Contrasting the above with an Emacs 29.1 installation on a non-Guix
system, I notice that the latter doesn't have any non-absolute paths in
native-comp-eln-load-path.  Thoughts?

-- 
Suhail

[bug#67260] [PATCH v6 1/7] gnu: emacs: Wrap EMACSNATIVELOADPATH.

[Suhail via Guix-patches via]

Liliana Marie Prikler <liliana.prikler@gmail.com> writes:

>> Contrasting the above with an Emacs 29.1 installation on a non-Guix
>> system, I notice that the latter doesn't have any non-absolute paths
>> in native-comp-eln-load-path.  Thoughts?
> I have noticed this myself, but I don't think that it's related to this
> series.

It seems to be.  It's possible that it may also be related to other
things already in the emacs-team branch (e.g. if the patch series is
only responsible for uncovering a pre-existing defect as opposed to
causing it), but the relative path only shows up during the series.

> Can you try 29.2 without the patches?

emacs-team branch at aae61f54ff doesn't have the relative path in
native-comp-eln-load-path.  The relative path shows up only after
[PATCH v6 3/7] is applied.  Hope this helps and hopefully the patch
series can soon be merged into master for general availability.


[PATCH v6 3/7] gnu: emacs: Don't hash file names in native compilation.

-- 
Suhail

[bug#67260] [PATCH v6 1/7] gnu: emacs: Wrap EMACSNATIVELOADPATH.

[Suhail via Guix-patches via]

Suhail via Guix-patches via <guix-patches@gnu.org> writes:

> hopefully the patch series can soon be merged into master for general
> availability.

Btw, does every package require something like [PATCH v6 5/7] or
[PATCH v6 6/7] in order for it to be natively compiled?

-- 
Suhail

[bug#67260] [PATCH v6 1/7] gnu: emacs: Wrap EMACSNATIVELOADPATH.

[Suhail via Guix-patches via]

Liliana Marie Prikler <liliana.prikler@gmail.com> writes:

>> Btw, does every package require something like [PATCH v6 5/7] or
>> [PATCH v6 6/7] in order for it to be natively compiled?
> No, these patches are only for natively compiled packages that don't
> use emacs-build-system.

I.e., any and all packages using the emacs-build-system should be
natively-compiled?  If so, I believe I've found a counter-example.

After installing v6 series and building emacs-htmlize my expectation is
that functions provided by it, such as htmlize-buffer, would be
natively-compiled.  However, I get:

> htmlize-buffer is an autoloaded interactive Lisp function in
> `htmlize.el'.

Note the lack of "native-compiled" in the above.

-- 
Suhail

[bug#67260] [PATCH v6 1/7] gnu: emacs: Wrap EMACSNATIVELOADPATH.

[Suhail via Guix-patches via]

Liliana Marie Prikler <liliana.prikler@gmail.com> writes:

> If you compile with emacs-minimal, that's the expected results. 

It was compiled with emacs-minimal=emacs.  For reference, I have inlined
the manifest file that was used (with the --no-grafts CLI option to
guix-package) towards the end of this email.  I'm sharing the one that
was actually used, but it could be minimized.

I believe there are two issues (not including the spurious entry in
native-comp-eln-load-path which may already have been resolved in v7 of
the series).  All my observations below are based on v6 patch series.

For emacs-htmlize, I believe the issue may stem from the fact that some
.eln files are available from entries in native-comp-eln-load-path only
via symlinks and the .eln loader may not be able to load symlinks.
Packages such as ox-html are available from two locations in the
native-comp-eln-load-path.  In one location they are symlinked, but
(presumably because org is a builtin package) it is also available from
another location in the native-comp-eln-load-path and in the latter
location it's not symlinked in.  I believe this difference is why for
some packages natively-compiled versions are loaded (e.g. org, ox-html
etc) whereas for others it's not the case.

In addition, I believe there's another issue.  Some packages' names are
getting  truncated.  For instance, instead of uniquify.eln, I observe
niquify.eln.  In this case, the .eln isn't symlinked (presumably because
it's a builtin), but due to the name being messed up (perhaps too
aggressive a truncation of hashes?) the natively compiled version is not
available:

> uniquify-item-p is a byte-compiled Lisp function in `uniquify.el'.

The manifest file that was used is below.

#+begin_src scheme
  (use-modules (guix transformations)
               (gnu packages))

  (define transform1
    (options->transformation
     '((with-input . "emacs-minimal=emacs")
       (without-tests . "emacs-clojure-mode"))))

  (packages->manifest
    (list (transform1 (specification->package "bash"))
          (transform1 (specification->package "coreutils"))
          (transform1 (specification->package "git"))
          (transform1 (specification->package "make"))
          (transform1 (specification->package "emacs"))
          (transform1 (specification->package "emacs-org"))
          (transform1
            (specification->package "emacs-org-contrib"))
          (transform1
            (specification->package "emacs-citeproc-el"))
          (transform1
            (specification->package "emacs-engrave-faces"))
          (transform1
            (specification->package "emacs-haskell-mode"))
          (transform1
            (specification->package "emacs-htmlize"))
          (transform1
            (specification->package "emacs-markdown-mode"))
          (transform1
            (specification->package
              "emacs-org-pandoc-import"))
          (transform1 (specification->package "pandoc"))
          ))
#+end_src

Thank you for your continued work on this!

-- 
Suhail

This email is not an offer capable of acceptance, does not evidence an
intention to enter into an agreement, has no operative effect until a
definitive agreement is signed in writing by both parties, and that no
party should act in reliance on the email or any representations of the
sender until a definitive agreement is signed in writing by both
parties.

[bug#67260] [PATCH v6 1/7] gnu: emacs: Wrap EMACSNATIVELOADPATH.

[Suhail via Guix-patches via]

Suhail via Guix-patches via <guix-patches@gnu.org> writes:

> I believe there are two issues (not including the spurious entry in
> native-comp-eln-load-path which may already have been resolved in v7 of
> the series).  All my observations below are based on v6 patch series.

I can confirm that the v7 series fixes the spurious entry in
native-comp-eln-load-path.

However, the 2 issues pointed out in the previous email remain:

1. failure to load natively-compiled version of non-builtin libraries
   like emacs-htmlize possibly due to symlinks, and

2. failure to load natively-compiled version of some builtin libraries
   like uniquify due to incorrect naming of .eln files

-- 
Suhail

[bug#67260] [PATCH v6 1/7] gnu: emacs: Wrap EMACSNATIVELOADPATH.

[Suhail via Guix-patches via]

Liliana Marie Prikler <liliana.prikler@gmail.com> writes:

> Would you care to debug this a bit more and report your findings?

The locations in question are in the store and hence write-protected.
What's a "safe" way to debug this?

>> Packages such as ox-html are available from two locations in the
>> native-comp-eln-load-path.  In one location they are symlinked, but
>> (presumably because org is a builtin package) it is also available
>> from another location in the native-comp-eln-load-path and in the
>> latter location it's not symlinked in.  I believe this difference is
>> why for some packages natively-compiled versions are loaded (e.g.
>> org, ox-html, etc) whereas for others it's not the case.
> Well, as pointed out in the deleted code, dlopen has a "one shared
> library per file name" limitation.  I don't think we're hit by that
> thanks to unique hashes in the store directory, but I might be wrong
> about that.  Maybe we have to do java-style FQDNs instead.

Perhaps.  I'd wondered, more simply, if symlinked .eln files were ever
being loaded.  In the limited testing I did, I didn't find such an
instance.  For what it's worth, only one of the entries in
native-comp-eln-load-path seems to have symlinked entries (namely,
~/.guix-profile/lib/emacs/native-site-lisp).

>> In addition, I believe there's another issue.  Some packages' names
>> are getting  truncated.  For instance, instead of uniquify.eln, I
>> observe niquify.eln.  In this case, the .eln isn't symlinked
>> (presumably because it's a builtin), but due to the name being messed
>> up (perhaps too aggressive a truncation of hashes?) the natively
>> compiled version is not available:
> I am not truncating any hashes, I'm not even computing them in the
> first place.  The functions I'm modifying are publicly callable, namely
> comp-el-to-eln-rel-filename for the relative file names, and
> comp-el-to-eln-filename for the absolute ones.
>
> There could be an off-by-one error hidden in the stripping of the
> BOGUS_DIRS, however.  Let's investigate that.

Were you thinking out loud, or are there specific steps you'd like me to
help with?

> Note, that you can map transform1, saving some typing overhead, and
> since you are transforming all of your packages you could compose that
> with specification->package.

Appreciate the tips, but I should clarify that the manifest file in
question was generated by way of guix shell --export-manifest.  I'd
omitted the header.

-- 
Suhail

This email is not an offer capable of acceptance, does not evidence an
intention to enter into an agreement, has no operative effect until a
definitive agreement is signed in writing by both parties, and that no
party should act in reliance on the email or any representations of the
sender until a definitive agreement is signed in writing by both
parties.

[bug#67260] [PATCH v6 1/7] gnu: emacs: Wrap EMACSNATIVELOADPATH.

[Suhail via Guix-patches via]

Suhail via Guix-patches via <guix-patches@gnu.org> writes:

> However, the 2 issues pointed out in the previous email remain:

Correction, there's only 1 issue.  The first issue was a false positive.
Apologies for the noise.

> 1. failure to load natively-compiled version of non-builtin libraries
>    like emacs-htmlize possibly due to symlinks, and

This was a false positive.  Turns out, if the function in question is
only autoloaded describe-function won't correctly identify the function
as being natively-compiled.  This is consistent with what happens on
Emacs 29.1 on a non-Guix system.  Not an issue.

> 2. failure to load natively-compiled version of some builtin libraries
>    like uniquify due to incorrect naming of .eln files

This issue remains as of v7 of the patch series.

-- 
Suhail

[bug#67260] [PATCH v6 1/7] gnu: emacs: Wrap EMACSNATIVELOADPATH.

[Suhail via Guix-patches via]

Liliana Marie Prikler <liliana.prikler@gmail.com> writes:

>> > 2. failure to load natively-compiled version of some builtin
>> > libraries like uniquify due to incorrect naming of .eln files
>> 
>> This issue remains as of v7 of the patch series.
> I fixed this one locally,

I can confirm that the .eln filenames are no longer truncated (as far as
I can tell).

> but ‘uniquify’ remains byte-compiled for no explainable reason.

And too that as of v8 patch, uniquify still isn't natively-compiled.
However, if you unload it and then reload it, it becomes
natively-compiled.

#+begin_src bash
  emacs --batch --eval "(message \"%s\" (progn (unload-feature 'uniquify) (require 'uniquify) (take 1 (split-string (substring-no-properties (describe-function 'uniquify-item-p)) \"\\n\" t))))"
#+end_src

#+RESULTS:
: Type q in help window to delete it
: (uniquify-item-p is a native-compiled Lisp function in `uniquify.el'.)


I think the issue is because uniquify (along with some others such as
prog-mode, backquote etc that I tested) are preloaded in Emacs.  I
believe that these preloaded packages need to be treated specially.  In
a non-Guix system, these preloaded packages have their .eln files stored
under something like
<native-comp-eln-load-path-entry>/29.x-<hash>/preloaded/ , whereas in
the v8 series there is no "preloaded" directory.  Instead .eln files for
packages such as uniquify, prog-mode, backquote etc. are in various
locations:

- uniquify :: <n-c-e-l-p-entry>/29.2-<hash>/uniquify.eln
- prog-mode :: <n-c-e-l-p-entry>/29.2-<hash>/progmodes/prog-mode.eln
- backquote :: <n-c-e-l-p-entry>/29.2-<hash>/emacs-lisp/backquote.eln

I suspect that the code which does the preloading may need to be patched
and/or the preloaded packages may need to have their .eln files
generated under a preloaded sub-directory (similar to what happens in
non-Guix systems).

Below is a list of .eln files that exist in the preloaded sub-directory
in Emacs 29.1 on a non-Guix system.  It's possible that the set of
preloaded packages on Emacs 29.2 is different.  I suspect that all of
these packages are similarly affected.

#+begin_example
  abbrev.eln
  backquote.eln
  bindings.eln
  buff-menu.eln
  burmese.eln
  button.eln
  byte-run.eln
  case-table.eln
  cconv.eln
  cham.eln
  characters.eln
  chinese.eln
  cl-generic.eln
  cl-preloaded.eln
  common-win.eln
  composite.eln
  cp51932.eln
  cus-face.eln
  cus-start.eln
  custom.eln
  cyrillic.eln
  czech.eln
  debug-early.eln
  disp-table.eln
  dnd.eln
  dos-fns.eln
  dos-vars.eln
  dos-w32.eln
  dynamic-setting.eln
  easymenu.eln
  ediff-hook.eln
  eldoc.eln
  electric.eln
  elisp-mode.eln
  english.eln
  env.eln
  epa-hook.eln
  ethiopic.eln
  eucjp-ms.eln
  european.eln
  faces.eln
  files.eln
  fill.eln
  float-sup.eln
  font-core.eln
  font-lock.eln
  fontset.eln
  format.eln
  frame.eln
  fringe.eln
  georgian.eln
  greek.eln
  haiku-win.eln
  hebrew.eln
  help.eln
  image.eln
  indent.eln
  indian.eln
  indonesian.eln
  internal.eln
  isearch.eln
  iso-transl.eln
  japanese.eln
  jit-lock.eln
  jka-cmpr-hook.eln
  keymap.eln
  khmer.eln
  korean.eln
  lao.eln
  lisp.eln
  lisp-mode.eln
  ls-lisp.eln
  macroexp.eln
  map-ynp.eln
  menu-bar.eln
  minibuffer.eln
  misc-lang.eln
  mouse.eln
  mule.eln
  mule-cmds.eln
  mule-conf.eln
  mule-util.eln
  mwheel.eln
  nadvice.eln
  newcomment.eln
  ns-win.eln
  obarray.eln
  oclosure.eln
  page.eln
  paragraphs.eln
  paren.eln
  pc-win.eln
  pgtk-dnd.eln
  pgtk-win.eln
  philippine.eln
  prog-mode.eln
  regexp-opt.eln
  register.eln
  replace.eln
  rfn-eshadow.eln
  rmc.eln
  romanian.eln
  scroll-bar.eln
  select.eln
  seq.eln
  shorthands.eln
  simple.eln
  sinhala.eln
  slovak.eln
  startup.eln
  subr.eln
  syntax.eln
  tab-bar.eln
  tabulated-list.eln
  tai-viet.eln
  text-mode.eln
  thai.eln
  tibetan.eln
  timer.eln
  tool-bar.eln
  tooltip.eln
  tty-colors.eln
  ucs-normalize.eln
  uniquify.eln
  utf-8-lang.eln
  vc-hooks.eln
  version.eln
  vietnamese.eln
  w32-fns.eln
  w32-vars.eln
  w32-win.eln
  widget.eln
  window.eln
  x-dnd.eln
  x-win.eln
#+end_example


-- 
Suhail

This email is not an offer capable of acceptance, does not evidence an
intention to enter into an agreement, has no operative effect until a
definitive agreement is signed in writing by both parties, and that no
party should act in reliance on the email or any representations of the
sender until a definitive agreement is signed in writing by both
parties.

[bug#67260] [PATCH v6 1/7] gnu: emacs: Wrap EMACSNATIVELOADPATH.

[Suhail via Guix-patches via]

Suhail via Guix-patches via <guix-patches@gnu.org> writes:

> Below is a list of .eln files that exist in the preloaded
> sub-directory in Emacs 29.1 on a non-Guix system.

Minor correction.  The list of files I posted was /adapted/ from the
list of files in the preloaded sub-directory on a non-Guix system (with
Emacs 29.1).  Specifically, the list I shared had the hash suffix
removed to make comparison with the .eln file on Guix easier.

-- 
Suhail

[bug#67260] [PATCH v6 1/7] gnu: emacs: Wrap EMACSNATIVELOADPATH.

[Suhail via Guix-patches via]

I'm assuming you're able to reproduce the issue on your end as well with
other features like prog-mode, backquote etc.

Liliana Marie Prikler <liliana.prikler@gmail.com> writes:

> I instrumented lread.c to print the relative and absolute file names
> tried, and it appears that these aren't loaded at all after install
> while EMACSNATIVELOADPATH is ignored when dumping.

Interesting.  I did a grep of the Emacs source (branch emacs-29) and
line 1770 of src/lread.c (in maybe_swap_for_eln) seemed relevant:

>   /* Look also in preloaded subfolder of the last entry in
>      `comp-eln-load-path'.  */

What surprised me was that not only was the "preloaded" directory
special-cased, but so too was the last entry of "comp-eln-load-path"
(which I imagine is referring to native-comp-eln-load-path).  Judging by
your message, I'm guessing you're well aware of this.  I, unfortunately,
have no additional insights or suggestions.

However, given that ensuring Emacs is able to find the natively-compiled
version of files seems non-trivial, it might help for a future patch
series to be accompanied with tests.  Thoughts?

-- 
Suhail