bug#47987: 28.0.50; [native-comp] Can't compile ELN if EL and ELC reside in different directories

bug#47987: 28.0.50; [native-comp] Can't compile ELN if EL and ELC reside in different directories

[Tassilo Horn]

I use mu with its mu4e Emacs package for reading my mail.  I have it
checked out as

  ~/Repos/el/mu/

and the *.el files are in

  ~/Repos/el/mu/mu4e/.

Mu can be build with the meson build tool which places the build
artifacts in a separate build/ subdirectory, so the executables are in

  ~/Repos/el/mu/build/mu/

and the *.elc files are in

  ~/Repos/el/mu/build/mu4e/.

I've added both ~/Repos/el/mu/mu4e/ and ~/Repos/el/mu/build/mu4e/ to
`load-path', so `C-h f mu4e' says

  mu4e is an interactive compiled Lisp function in ‘mu4e.el’.

i.e., it has loaded the compiled file and the help links to the source
file (probably because I've set `load-prefer-newer').  Perfect!

However, the native compiler complains that it cannot find the source
files in a *Warnings* buffer:

--8<---------------cut here---------------start------------->8---
Warning (comp): Cannot look-up eln file as no source file was found for /home/horn/Repos/el/mu/build/mu4e/mu4e.elc Disable showing Disable logging
Warning (comp): Cannot look-up eln file as no source file was found for /home/horn/Repos/el/mu/build/mu4e/mu4e-vars.elc Disable showing Disable logging
Warning (comp): Cannot look-up eln file as no source file was found for /home/horn/Repos/el/mu/build/mu4e/mu4e-headers.elc Disable showing Disable logging
Warning (comp): Cannot look-up eln file as no source file was found for /home/horn/Repos/el/mu/build/mu4e/mu4e-utils.elc Disable showing Disable logging
Warning (comp): Cannot look-up eln file as no source file was found for /home/horn/Repos/el/mu/build/mu4e/mu4e-message.elc Disable showing Disable logging
Warning (comp): Cannot look-up eln file as no source file was found for /home/horn/Repos/el/mu/build/mu4e/mu4e-lists.elc Disable showing Disable logging
Warning (comp): Cannot look-up eln file as no source file was found for /home/horn/Repos/el/mu/build/mu4e/mu4e-proc.elc Disable showing Disable logging
Warning (comp): Cannot look-up eln file as no source file was found for /home/horn/Repos/el/mu/build/mu4e/mu4e-mark.elc Disable showing Disable logging
Warning (comp): Cannot look-up eln file as no source file was found for /home/horn/Repos/el/mu/build/mu4e/mu4e-compose.elc Disable showing Disable logging
Warning (comp): Cannot look-up eln file as no source file was found for /home/horn/Repos/el/mu/build/mu4e/mu4e-actions.elc Disable showing Disable logging
Warning (comp): Cannot look-up eln file as no source file was found for /home/horn/Repos/el/mu/build/mu4e/mu4e-draft.elc Disable showing Disable logging
Warning (comp): Cannot look-up eln file as no source file was found for /home/horn/Repos/el/mu/build/mu4e/mu4e-context.elc Disable showing Disable logging
Warning (comp): Cannot look-up eln file as no source file was found for /home/horn/Repos/el/mu/build/mu4e/mu4e-view.elc Disable showing Disable logging
Warning (comp): Cannot look-up eln file as no source file was found for /home/horn/Repos/el/mu/build/mu4e/mu4e-view-common.elc Disable showing Disable logging
Warning (comp): Cannot look-up eln file as no source file was found for /home/horn/Repos/el/mu/build/mu4e/mu4e-view-gnus.elc Disable showing Disable logging
Warning (comp): Cannot look-up eln file as no source file was found for /home/horn/Repos/el/mu/build/mu4e/mu4e-main.elc Disable showing Disable logging
Warning (comp): Cannot look-up eln file as no source file was found for /home/horn/Repos/el/mu/build/mu4e/mu4e-org.elc Disable showing Disable logging
--8<---------------cut here---------------end--------------->8---

As the no-native-comp emacs has no problem locating byte-compiled and
source file, the native-comp emacs shouldn't have that problem, too.

In the information below, see that mu4e's byte-compiled files are listed
as load-path-shadows, shadowing the source files.


In GNU Emacs 28.0.50 (build 58, x86_64-pc-linux-gnu, GTK+ Version 3.24.28, cairo version 1.17.4)
 of 2021-04-24 built on thinkpad-t440p
Repository revision: 72ac4461676c49b7c580c40c2df10e02411fd320
Repository branch: master
Windowing system distributor 'The X.Org Foundation', version 11.0.12101001
System Description: Arch Linux

Configured using:
 'configure --with-nativecomp'

Configured features:
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GPM GSETTINGS HARFBUZZ JPEG
JSON LCMS2 LIBOTF LIBSYSTEMD LIBXML2 M17N_FLT MODULES NOTIFY INOTIFY
PDUMPER PNG RSVG SECCOMP SOUND THREADS TIFF TOOLKIT_SCROLL_BARS X11 XDBE
XIM XPM GTK3 ZLIB

Important settings:
  value of $LC_MONETARY: de_DE.utf8
  value of $LC_NUMERIC: de_DE.utf8
  value of $LC_TIME: de_DE.utf8
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: mu4e:main

Minor modes in effect:
  global-aggressive-indent-mode: t
  dynamic-completion-mode: t
  which-key-mode: t
  company-posframe-mode: t
  global-company-mode: t
  yas-global-mode: t
  yas-minor-mode: t
  global-git-commit-mode: t
  magit-auto-revert-mode: t
  override-global-mode: t
  vertico-mode: t
  aggressive-completion-mode: t
  minibuffer-depth-indicate-mode: t
  recentf-mode: t
  pixel-scroll-mode: t
  save-place-mode: t
  savehist-mode: t
  show-paren-mode: t
  shell-dirtrack-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  buffer-read-only: t
  column-number-mode: t
  line-number-mode: t
  transient-mark-mode: t
  overwrite-mode: overwrite-mode-binary

Load-path shadows:
~/Repos/el/mu/build/mu4e/mu4e hides ~/Repos/el/mu/mu4e/mu4e
~/Repos/el/mu/build/mu4e/mu4e-main hides ~/Repos/el/mu/mu4e/mu4e-main
~/Repos/el/mu/build/mu4e/mu4e-view hides ~/Repos/el/mu/mu4e/mu4e-view
~/Repos/el/mu/build/mu4e/mu4e-org hides ~/Repos/el/mu/mu4e/mu4e-org
~/Repos/el/mu/build/mu4e/mu4e-lists hides ~/Repos/el/mu/mu4e/mu4e-lists
~/Repos/el/mu/build/mu4e/mu4e-actions hides ~/Repos/el/mu/mu4e/mu4e-actions
~/Repos/el/mu/build/mu4e/mu4e-utils hides ~/Repos/el/mu/mu4e/mu4e-utils
~/Repos/el/mu/build/mu4e/mu4e-context hides ~/Repos/el/mu/mu4e/mu4e-context
~/Repos/el/mu/build/mu4e/mu4e-draft hides ~/Repos/el/mu/mu4e/mu4e-draft
~/Repos/el/mu/build/mu4e/mu4e-message hides ~/Repos/el/mu/mu4e/mu4e-message
~/Repos/el/mu/build/mu4e/mu4e-compose hides ~/Repos/el/mu/mu4e/mu4e-compose
~/Repos/el/mu/build/mu4e/mu4e-view-common hides ~/Repos/el/mu/mu4e/mu4e-view-common
~/Repos/el/mu/build/mu4e/mu4e-view-old hides ~/Repos/el/mu/mu4e/mu4e-view-old
~/Repos/el/mu/build/mu4e/mu4e-view-gnus hides ~/Repos/el/mu/mu4e/mu4e-view-gnus
~/Repos/el/mu/build/mu4e/mu4e-headers hides ~/Repos/el/mu/mu4e/mu4e-headers
~/Repos/el/mu/build/mu4e/mu4e-mark hides ~/Repos/el/mu/mu4e/mu4e-mark
~/Repos/el/mu/build/mu4e/mu4e-icalendar hides ~/Repos/el/mu/mu4e/mu4e-icalendar
~/Repos/el/mu/build/mu4e/mu4e-speedbar hides ~/Repos/el/mu/mu4e/mu4e-speedbar
~/Repos/el/mu/build/mu4e/mu4e-contrib hides ~/Repos/el/mu/mu4e/mu4e-contrib
~/Repos/el/mu/build/mu4e/mu4e-proc hides ~/Repos/el/mu/mu4e/mu4e-proc
~/Repos/el/mu/build/mu4e/mu4e-meta hides ~/Repos/el/mu/mu4e/mu4e-meta
~/Repos/el/mu/build/mu4e/mu4e-vars hides ~/Repos/el/mu/mu4e/mu4e-vars
/home/horn/.emacs.d/elpa/transient-20210420.1635/transient hides /home/horn/Repos/el/emacs/lisp/transient

Features:
(shadow face-remap emacsbug eieio-opt speedbar ezimage dframe shortdoc
help-fns radix-tree company-oddmuse company-keywords company-etags etags
fileloop generator xref project company-gtags company-dabbrev-code
company-dabbrev company-files company-clang company-capf company-cmake
company-semantic company-template company-bbdb paredit vc-mtn vc-hg
vc-bzr vc-src vc-sccs vc-svn vc-cvs vc-rcs display-fill-column-indicator
auto-package-update finder-inf generic yaml-mode fish-mode cargo
cargo-process rust-utils rust-mode rust-rustfmt rust-playpen
rust-compile compile rust-cargo web-mode disp-table preview-latex
auto-loads tex-site deadgrep spinner s hl-todo aggressive-indent
rainbow-mode vc-git vc-dir ewoc vc vc-dispatcher epa-file dired-x mu4e
mu4e-org mu4e-main mu4e-view mu4e-view-gnus mu4e-view-common
mu4e-headers mu4e-compose mu4e-context mu4e-draft mu4e-actions ido
rfc2368 mu4e-mark mu4e-proc mu4e-utils doc-view jka-compr image-mode
exif mu4e-lists mu4e-message flow-fill org ob ob-tangle ob-ref ob-lob
ob-table ob-exp org-macro org-footnote org-src ob-comint org-pcomplete
org-list org-faces org-entities org-version ob-emacs-lisp ob-core
ob-eval org-table ol org-keys org-compat org-macs org-loaddefs find-func
cal-menu calendar cal-loaddefs mule-util hl-line mu4e-vars mu4e-meta
smtpmail-multi smtpmail sendmail ecomplete completion auto-dictionary
flyspell ispell tramp-smb which-key highlight-parentheses
company-restclient know-your-http-well http-status-codes http-relations
http-methods http-headers restclient company-posframe posframe company
pcase yasnippet forge-list forge-commands forge-semi forge-bitbucket
buck forge-gogs gogs forge-gitea gtea forge-gitlab glab forge-github
ghub-graphql treepy gsexp ghub let-alist gnutls forge-notify
forge-revnote forge-pullreq forge-issue forge-topic forge-post
markdown-mode color thingatpt noutline outline forge-repo forge
forge-core forge-db closql emacsql-sqlite advice emacsql
emacsql-compiler magit-submodule magit-obsolete magit-blame magit-stash
magit-reflog magit-bisect magit-push magit-pull magit-fetch magit-clone
magit-remote magit-commit magit-sequence magit-notes magit-worktree
magit-tag magit-merge magit-branch magit-reset magit-files magit-refs
magit-status magit magit-repos magit-apply magit-wip magit-log
which-func imenu magit-diff smerge-mode diff diff-mode git-commit
log-edit pcvs-util add-log magit-core magit-autorevert autorevert
filenotify magit-margin magit-transient magit-process with-editor server
magit-mode transient magit-git magit-section magit-utils crm dash
visual-filename-abbrev debbugs soap-client url-http url-auth url-gw nsm
warnings rng-xsd rng-dt rng-util xsd-regexp bug-reference
use-package-bind-key bind-key easy-mmode vertico aggressive-completion
icomplete mb-depth use-package-diminish windmove rx tramp-cache tramp-sh
recentf tree-widget pixel-scroll saveplace savehist paren smiley
gnus-art mm-uu mml2015 mm-view mml-smime smime dig gnus-sum shr kinsoku
svg dom gnus-group gnus-undo gnus-start gnus-dbus dbus xml gnus-cloud
nnimap nnmail mail-source utf7 netrc nnoo gnus-spec gnus-int gnus-range
message rmc puny dired dired-loaddefs rfc822 mml mml-sec epa derived epg
epg-config mm-decode mm-bodies mm-encode mail-parse rfc2231 mailabbrev
gmm-utils mailheader gnus-win gnus wid-edit nnheader gnus-util rmail
rmail-loaddefs rfc2047 rfc2045 ietf-drums text-property-search mm-util
mail-prsvr mail-utils edmacro kmacro dracula-theme diminish cl-extra
help-mode use-package-ensure use-package-core tramp tramp-loaddefs
trampver tramp-integration files-x tramp-compat shell pcomplete comint
ansi-color ring parse-time iso8601 time-date ls-lisp format-spec info
package browse-url url url-proxy url-privacy url-expand url-methods
url-history url-cookie url-domsuf url-util mailcap url-handlers
url-parse auth-source cl-seq eieio eieio-core cl-macs eieio-loaddefs
password-cache json subr-x map url-vars seq byte-opt gv bytecomp
byte-compile cconv cl-loaddefs cl-lib iso-transl tooltip eldoc electric
uniquify ediff-hook vc-hooks lisp-float-type mwheel term/x-win x-win
term/common-win x-dnd tool-bar dnd fontset image regexp-opt fringe
tabulated-list replace newcomment text-mode elisp-mode lisp-mode
prog-mode register page tab-bar menu-bar rfn-eshadow isearch easymenu
timer select scroll-bar mouse jit-lock font-lock syntax font-core
term/tty-colors frame minibuffer cl-generic cham georgian utf-8-lang
misc-lang vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms
cp51932 hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese composite charscript charprop case-table epa-hook
jka-cmpr-hook help simple abbrev obarray cl-preloaded nadvice button
loaddefs faces cus-face macroexp files window text-properties overlay
sha1 md5 base64 format env code-pages mule custom widget
hashtable-print-readable backquote threads dbusbind inotify lcms2
dynamic-setting system-font-setting font-render-setting cairo
move-toolbar gtk x-toolkit x multi-tty make-network-process emacs)

Memory information:
((conses 16 532892 73350)
 (symbols 48 42498 5)
 (strings 32 195128 7829)
 (string-bytes 1 5835947)
 (vectors 16 74806)
 (vector-slots 8 886755 20954)
 (floats 8 564 336)
 (intervals 56 1103 27)
 (buffers 992 13))

bug#47987: 28.0.50; [native-comp] Can't compile ELN if EL and ELC reside in different directories

[Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Tassilo Horn <tsdh@gnu.org> writes:

> I use mu with its mu4e Emacs package for reading my mail.  I have it
> checked out as
>
>   ~/Repos/el/mu/
>
> and the *.el files are in
>
>   ~/Repos/el/mu/mu4e/.
>
> Mu can be build with the meson build tool which places the build
> artifacts in a separate build/ subdirectory, so the executables are in
>
>   ~/Repos/el/mu/build/mu/
>
> and the *.elc files are in
>
>   ~/Repos/el/mu/build/mu4e/.
>
> I've added both ~/Repos/el/mu/mu4e/ and ~/Repos/el/mu/build/mu4e/ to
> `load-path', so `C-h f mu4e' says
>
>   mu4e is an interactive compiled Lisp function in ‘mu4e.el’.
>
> i.e., it has loaded the compiled file and the help links to the source
> file (probably because I've set `load-prefer-newer').  Perfect!
>
> However, the native compiler complains that it cannot find the source
> files in a *Warnings* buffer:
>
> Warning (comp): Cannot look-up eln file as no source file was found for /home/horn/Repos/el/mu/build/mu4e/mu4e.elc Disable showing Disable logging

[...]

> As the no-native-comp emacs has no problem locating byte-compiled and
> source file, the native-comp emacs shouldn't have that problem, too.
>
> In the information below, see that mu4e's byte-compiled files are listed
> as load-path-shadows, shadowing the source files.

Hi Tassilo,

The trouble is that given the bytecode being loaded Emacs needs to find
the source file, both to native compile it (if necessary) both to hash
it in order to the locate the correct .eln file to load.

Now given the .elc file removing the final 'c' was the simple algo
applied for this scope :)  But is this is not sufficient we have to
either provide a more sophisticate mechanism so that a package can
register a new mapping or either disable eln load for some package.

Thanks

  Andrea

bug#47987: 28.0.50; [native-comp] Can't compile ELN if EL and ELC reside in different directories

[Eli Zaretskii]

> Cc: 47987@debbugs.gnu.org
> Date: Tue, 27 Apr 2021 17:00:28 +0000
> From:  Andrea Corallo via "Bug reports for GNU Emacs,
>  the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>
> 
> The trouble is that given the bytecode being loaded Emacs needs to find
> the source file, both to native compile it (if necessary) both to hash
> it in order to the locate the correct .eln file to load.
> 
> Now given the .elc file removing the final 'c' was the simple algo
> applied for this scope :)  But is this is not sufficient we have to
> either provide a more sophisticate mechanism so that a package can
> register a new mapping or either disable eln load for some package.

We can either (a) look in all the directories in load-path, if the .el
file is not found in the same directory as the .elc; or (b) add a new
variable a-la load-path that is only searched if the current look up
fails.

I prefer the former, unless it has some downsides.

bug#47987: 28.0.50; [native-comp] Can't compile ELN if EL and ELC reside in different directories

[Tassilo Horn]

Eli Zaretskii <eliz@gnu.org> writes:

Hi all,

> We can either (a) look in all the directories in load-path, if the .el
> file is not found in the same directory as the .elc; or (b) add a new
> variable a-la load-path that is only searched if the current look up
> fails.
>
> I prefer the former, unless it has some downsides.

I just saw that the specific issue with mu4e has been fixed on their
side. The meson/ninja build now copies the el files over to build/mu4e/,
too, so now el & elc reside in the same directory.

Bye,
Tassilo

bug#47987: 28.0.50; [native-comp] Can't compile ELN if EL and ELC reside in different directories

[Eli Zaretskii]

> From: Tassilo Horn <tsdh@gnu.org>
> Date: Sun, 06 Nov 2022 10:33:51 +0100
> 
> I just saw that the specific issue with mu4e has been fixed on their
> side. The meson/ninja build now copies the el files over to build/mu4e/,
> too, so now el & elc reside in the same directory.

Thanks, so I guess this bug can be closed now.