bug#52376: 28.0.90; libdir is missing from native-comp-eln-load-path with GTK3 build

bug#52376: 28.0.90; libdir is missing from native-comp-eln-load-path with GTK3 build

[Bhavin Gandhi]

I have been trying to build a RPM package for pretest 28.0.90. This
process builds three variants, GTK3, Lucid, and No X.

When I install and start GTK3 based Emacs with emacs -Q, the value of
native-comp-eln-load-path does not have
/usr/lib64/emacs/28.0.90/native-lisp/ in it.

--8<---------------cut here---------------start------------->8---
native-comp-eln-load-path is a variable defined in ‘C source code’.

Its value is
("/home/bhavin/.emacs.d/eln-cache/" "/home/bhavin/src/emacs/native-lisp/")
--8<---------------cut here---------------end--------------->8---

$ ls -ll /usr/lib64/emacs/28.0.90/native-lisp/28.0.90-619a407c/
total 3080
-rw-r--r--. 1 root root  81216 Dec  8 15:14 autoload-f3599901-fca77eea.eln
…

But in case of other two variants Lucid and the No X, the libdir is there.

To isolate the issue I tried following commands with newly extracted
pretest tar:


--8<---------------cut here---------------start------------->8---
$ ./configure --build=x86_64-redhat-linux-gnu
--host=x86_64-redhat-linux-gnu --program-prefix=
--disable-dependency-tracking --prefix=/usr --exec-prefix=/usr
--bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc
--datadir=/usr/share --includedir=/usr/include --libdir=/usr/lib64
--libexecdir=/usr/libexec --localstatedir=/var
--sharedstatedir=/var/lib --mandir=/usr/share/man
--infodir=/usr/share/info --with-dbus --with-gif --with-jpeg
--with-png --with-rsvg --with-tiff --with-xft --with-xpm
--with-x-toolkit=gtk3 --with-gpm=no --with-xwidgets --with-modules
--with-harfbuzz --with-cairo --with-json --with-native-compilation
$ make -j8 bootstrap
$ make -j8
$ sudo make -j8 install
--8<---------------cut here---------------end--------------->8---

With this installation, the libdir is detected and loaded as well.

I'm not sure where to look and how to debug this issue, I tried
looking at the source code to find places where this variable is set,
but wasn't able to find any code related to libdir. This might not be
a bug, but with my very little knowledge on how Emacs starts and loads
things, I have already ran out of things to try out for this problem.

This behaviour is consistently reproducible, so I should be able to
run commands, give more details, and try any instructions to debug
this. All the build steps, above commands are run in separate and new
containers.



In GNU Emacs 28.0.90 (build 1, x86_64-redhat-linux-gnu, GTK+ Version
3.24.30, cairo version 1.17.4)
 of 2021-12-08 built on toolbox
Repository revision: c4a21caf591131351dab4f26adac00541614fb9f
Repository branch: master
Windowing system distributor 'The X.Org Foundation', version 11.0.12101003
System Description: Fedora Linux 35 (Container Image)

Configured using:
 'configure --build=x86_64-redhat-linux-gnu
 --host=x86_64-redhat-linux-gnu --program-prefix=
 --disable-dependency-tracking --prefix=/usr --exec-prefix=/usr
 --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc
 --datadir=/usr/share --includedir=/usr/include --libdir=/usr/lib64
 --libexecdir=/usr/libexec --localstatedir=/var
 --sharedstatedir=/var/lib --mandir=/usr/share/man
 --infodir=/usr/share/info --with-dbus --with-gif --with-jpeg --with-png
 --with-rsvg --with-tiff --with-xft --with-xpm --with-x-toolkit=gtk3
 --with-gpm=no --with-xwidgets --with-modules --with-harfbuzz
 --with-cairo --with-json --with-native-compilation
 build_alias=x86_64-redhat-linux-gnu host_alias=x86_64-redhat-linux-gnu
 CC=gcc 'CFLAGS=-DMAIL_USE_LOCKF -O2 -flto=auto -ffat-lto-objects
 -fexceptions -g -grecord-gcc-switches -pipe -Wall
 -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2
 -Wp,-D_GLIBCXX_ASSERTIONS
 -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong
 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic
 -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection'
 LDFLAGS=-Wl,-z,relro
 PKG_CONFIG_PATH=:/usr/lib64/pkgconfig:/usr/share/pkgconfig'

Configured features:
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GSETTINGS HARFBUZZ JPEG JSON
LIBOTF LIBSELINUX LIBSYSTEMD LIBXML2 M17N_FLT MODULES NATIVE_COMP NOTIFY
INOTIFY PDUMPER PNG RSVG SECCOMP SOUND THREADS TIFF TOOLKIT_SCROLL_BARS
X11 XDBE XIM XPM XWIDGETS GTK3 ZLIB

Important settings:
  value of $LANG: C.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Lisp Interaction

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  indent-tabs-mode: t
  transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message rmc puny dired dired-loaddefs
rfc822 mml mml-sec epa derived epg rfc6068 epg-config gnus-util rmail
rmail-loaddefs auth-source eieio eieio-core eieio-loaddefs
password-cache json map text-property-search time-date mm-decode
mm-bodies mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader
sendmail rfc2047 rfc2045 ietf-drums mm-util mail-prsvr mail-utils comp
comp-cstr warnings subr-x rx cl-seq cl-macs cl-extra help-mode seq
byte-opt gv cl-loaddefs cl-lib bytecomp byte-compile cconv iso-transl
tooltip eldoc paren electric uniquify ediff-hook vc-hooks
lisp-float-type elisp-mode mwheel term/x-win x-win term/common-win x-dnd
tool-bar dnd fontset image regexp-opt fringe tabulated-list replace
newcomment text-mode lisp-mode prog-mode register page tab-bar menu-bar
rfn-eshadow isearch easymenu timer select scroll-bar mouse jit-lock
font-lock syntax font-core term/tty-colors frame minibuffer cl-generic
cham georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao
korean japanese eucjp-ms cp51932 hebrew greek romanian slovak czech
european ethiopic indian cyrillic chinese composite emoji-zwj charscript
charprop case-table epa-hook jka-cmpr-hook help simple abbrev obarray
cl-preloaded nadvice button loaddefs faces cus-face macroexp files
window text-properties overlay sha1 md5 base64 format env code-pages
mule custom widget hashtable-print-readable backquote threads
xwidget-internal dbusbind inotify dynamic-setting system-font-setting
font-render-setting cairo move-toolbar gtk x-toolkit x multi-tty
make-network-process native-compile emacs)

Memory information:
((conses 16 74321 8216)
 (symbols 48 7850 1)
 (strings 32 21907 3978)
 (string-bytes 1 764195)
 (vectors 16 17728)
 (vector-slots 8 256054 16638)
 (floats 8 27 37)
 (intervals 56 304 0)
 (buffers 992 12))

bug#52376: 28.0.90; libdir is missing from native-comp-eln-load-path with GTK3 build

[Eli Zaretskii]

> From: Bhavin Gandhi <bhavin7392@gmail.com>
> Date: Wed, 8 Dec 2021 22:51:14 +0530
> 
> I have been trying to build a RPM package for pretest 28.0.90. This
> process builds three variants, GTK3, Lucid, and No X.

Please describe in more detail how you build the 3 versions.  Are you
using the same source tree, but different build directories, for
example?  Do you clean up the tree between different builds?

> When I install and start GTK3 based Emacs with emacs -Q, the value of
> native-comp-eln-load-path does not have
> /usr/lib64/emacs/28.0.90/native-lisp/ in it.
> 
> --8<---------------cut here---------------start------------->8---
> native-comp-eln-load-path is a variable defined in ‘C source code’.
> 
> Its value is
> ("/home/bhavin/.emacs.d/eln-cache/" "/home/bhavin/src/emacs/native-lisp/")
> --8<---------------cut here---------------end--------------->8---

This is normal if you start Emacs from the build tree.  is that what
you did?

> $ ls -ll /usr/lib64/emacs/28.0.90/native-lisp/28.0.90-619a407c/
> total 3080
> -rw-r--r--. 1 root root  81216 Dec  8 15:14 autoload-f3599901-fca77eea.eln
> …

Was this directory produced by "make install" from the same build
that built the GTK3 version?  The *.eln files are specific to the
binary with which they were produced, so the fact that you have the
*.eln files there doesn't yet mean that a specific Emacs binary will
accept them as matching the binary.

> To isolate the issue I tried following commands with newly extracted
> pretest tar:
> 
> 
> --8<---------------cut here---------------start------------->8---
> $ ./configure --build=x86_64-redhat-linux-gnu
> --host=x86_64-redhat-linux-gnu --program-prefix=
> --disable-dependency-tracking --prefix=/usr --exec-prefix=/usr
> --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc
> --datadir=/usr/share --includedir=/usr/include --libdir=/usr/lib64
> --libexecdir=/usr/libexec --localstatedir=/var
> --sharedstatedir=/var/lib --mandir=/usr/share/man
> --infodir=/usr/share/info --with-dbus --with-gif --with-jpeg
> --with-png --with-rsvg --with-tiff --with-xft --with-xpm
> --with-x-toolkit=gtk3 --with-gpm=no --with-xwidgets --with-modules
> --with-harfbuzz --with-cairo --with-json --with-native-compilation
> $ make -j8 bootstrap
> $ make -j8
> $ sudo make -j8 install

How is this different from the commands you used to produce the build
which didn't find the *.eln files?

bug#52376: 28.0.90; libdir is missing from native-comp-eln-load-path with GTK3 build

[Bhavin Gandhi]

On Wed, 8 Dec 2021 at 23:04, Eli Zaretskii <eliz@gnu.org> wrote:
> Please describe in more detail how you build the 3 versions.  Are you
> using the same source tree, but different build directories, for
> example?  Do you clean up the tree between different builds?

Same source tree is used like this (removing templating used by RPM spec
file):

mkdir build-gtk && cd build-gtk
ln -s ../configure .
./configure[1]
make bootstrap
make

cd ..
mkdir build-lucid && cd build-lucid
./configure [flags]
make bootstrap
make

cd ..
mkdir build-nox && cd build-nox
ln -s ../configure .
./configure [flags]
# there is no bootstrap here, not sure why that's the case.
make

No clean-up is done between the two builds.

> This is normal if you start Emacs from the build tree.  is that what
> you did?

I'm starting it from home and not from the build tree. The container
which build the binaries (and the package), and the one which is running
it are different.

> > $ ls -ll /usr/lib64/emacs/28.0.90/native-lisp/28.0.90-619a407c/
> > total 3080
> > -rw-r--r--. 1 root root  81216 Dec  8 15:14 autoload-f3599901-fca77eea.eln
> > …
>
> Was this directory produced by "make install" from the same build
> that built the GTK3 version?  The *.eln files are specific to the
> binary with which they were produced, so the fact that you have the
> *.eln files there doesn't yet mean that a specific Emacs binary will
> accept them as matching the binary.

Yes, these files are produced by the same build which created the GTK3
version. The comp-native-version-dir variable's value matches the above
one. Is there any other way to verify that the *.eln files are the correct ones?

> > […]
> > $ make -j8 bootstrap
> > $ make -j8
> > $ sudo make -j8 install
>
> How is this different from the commands you used to produce the build
> which didn't find the *.eln files?

One difference is having a different build directory (like build-gtk),
and the RPM build process adds a bunch of CFLAGS, and few more
variables.

[1] Here are all the extra flags added when building via RPM packaging
script (spec file), but these are the same for all 3 builds in this
case.


+ CFLAGS='-DMAIL_USE_LOCKF -O2 -flto=auto -ffat-lto-objects
-fexceptions -g -grecord-gcc-switches -pipe -Wall
-Werror=format-security -Wp,-D_FORTIFY_SOURCE=2
-Wp,-D_GLIBCXX_ASSERTIONS
-specs=/usr/lib/rpm/redhat/redhat-hardened-cc1
-fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1
 -m64  -mtune=generic -fasynchronous-unwind-tables
-fstack-clash-protection -fcf-protection'
+ export CFLAGS
+ CXXFLAGS='-O2 -flto=auto -ffat-lto-objects -fexceptions -g
-grecord-gcc-switches -pipe -Wall -Werror=format-security
-Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS
-specs=/usr/lib/rpm/redhat/redhat-hardened-cc1
-fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1
 -m64  -mtune=generic -fasynchronous-unwind-tables
-fstack-clash-protection -fcf-protection'
+ export CXXFLAGS
+ FFLAGS='-O2 -flto=auto -ffat-lto-objects -fexceptions -g
-grecord-gcc-switches -pipe -Wall -Werror=format-security
-Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS
-specs=/usr/lib/rpm/redhat/redhat-hardened-cc1
-fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1
 -m64  -mtune=generic -fasynchronous-unwind-tables
-fstack-clash-protection -fcf-protection
-I/usr/lib64/gfortran/modules'
+ export FFLAGS
+ FCFLAGS='-O2 -flto=auto -ffat-lto-objects -fexceptions -g
-grecord-gcc-switches -pipe -Wall -Werror=format-security
-Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS
-specs=/usr/lib/rpm/redhat/redhat-hardened-cc1
-fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1
 -m64  -mtune=generic -fasynchronous-unwind-tables
-fstack-clash-protection -fcf-protection
-I/usr/lib64/gfortran/modules'

bug#52376: 28.0.90; libdir is missing from native-comp-eln-load-path with GTK3 build

[Eli Zaretskii]

> From: Bhavin Gandhi <bhavin7392@gmail.com>
> Date: Wed, 8 Dec 2021 23:35:40 +0530
> Cc: 52376@debbugs.gnu.org
> 
> mkdir build-gtk && cd build-gtk
> ln -s ../configure .
> ./configure[1]
> make bootstrap
> make
> 
> cd ..
> mkdir build-lucid && cd build-lucid
> ./configure [flags]
> make bootstrap
> make
> 
> cd ..
> mkdir build-nox && cd build-nox
> ln -s ../configure .
> ./configure [flags]
> # there is no bootstrap here, not sure why that's the case.
> make
> 
> No clean-up is done between the two builds.

Maybe you should do the cleanup.

Also you didn't show the "make install" commands -- are they
installing into different places?  Where are the corresponding
emacs.pdmp files?

> > > […]
> > > $ make -j8 bootstrap
> > > $ make -j8
> > > $ sudo make -j8 install
> >
> > How is this different from the commands you used to produce the build
> > which didn't find the *.eln files?
> 
> One difference is having a different build directory (like build-gtk),
> and the RPM build process adds a bunch of CFLAGS, and few more
> variables.
> 
> [1] Here are all the extra flags added when building via RPM packaging
> script (spec file), but these are the same for all 3 builds in this
> case.

I don't think compiler options could cause this.  It's somehow related
to the *.eln files and to the location of the emacs.pdmp file.

Did the final dump process show that it loads "native-compiled lisp"
files for all the preloaded files?

bug#52376: 28.0.90; libdir is missing from native-comp-eln-load-path with GTK3 build

[Bhavin Gandhi]

On Wed, 8 Dec 2021 at 23:56, Eli Zaretskii <eliz@gnu.org> wrote:
> >
> > No clean-up is done between the two builds.
>
> Maybe you should do the cleanup.

How can I do that? By extracting the source tar again?

> Also you didn't show the "make install" commands -- are they
> installing into different places?  Where are the corresponding
> emacs.pdmp files?

The way this is being done right now is:

After all three are built, make install is run from build-gtk directory
only. And for other two, only relevant files are copied over to the
required locations as:

/usr/bin/emacs => /etc/alternatives/emacs => /usr/bin/emacs-28.0.90
(actual binary)
/usr/bin/emacs-28.0.90.pdmp
/usr/bin/emacs-lucid => /etc/alternatives/emacs-lucid =>
/usr/bin/emacs-28.0.90-lucid
/usr/bin/emacs-28.0.90-lucid.pdmp

The corresponding directories from native-lisp for each of them are
copied over as well to /usr/lib64/emacs/28.0.90/native-lisp/

This means the *.el and *.elc files from GTK3 build are used by the
other two.

These are the steps which happen after building all three:
%{buildroot}: the final destination from which the RPM package is
created, it is a chroot.
%{version}: 28.0.90 here
%{native_lisp}: /usr/lib64/emacs/28.0.90/native-lisp

# Remove versioned file so that we end up with .1 suffix and only one DOC file
rm build-{gtk,lucid,nox}/src/emacs-%{version}.*

cd build-gtk
%make_install
cd ..

# Let alternatives manage the symlink
rm %{buildroot}%{_bindir}/emacs
touch %{buildroot}%{_bindir}/emacs

# Remove emacs.pdmp from common (we copy it over in later steps)
rm %{buildroot}%{emacs_libexecdir}/emacs.pdmp

# Install emacs.pdmp of the emacs with GTK+
install -p -m 0644 build-gtk/src/emacs.pdmp
%{buildroot}%{_bindir}/emacs-%{version}.pdmp

# Install the emacs with LUCID toolkit
install -p -m 0755 build-lucid/src/emacs
%{buildroot}%{_bindir}/emacs-%{version}-lucid
install -p -m 0644 build-lucid/src/emacs.pdmp
%{buildroot}%{_bindir}/emacs-%{version}-lucid.pdmp

# Install the emacs without X
install -p -m 0755 build-nox/src/emacs
%{buildroot}%{_bindir}/emacs-%{version}-nox
install -p -m 0644 build-nox/src/emacs.pdmp
%{buildroot}%{_bindir}/emacs-%{version}-nox.pdmp

lucid_comp_native_ver=$(ls -1 build-lucid/native-lisp)
nox_comp_native_ver=$(ls -1 build-nox/native-lisp)
cp -ar build-lucid/native-lisp/${lucid_comp_native_ver}
%{buildroot}%{native_lisp}
cp -ar build-nox/native-lisp/${nox_comp_native_ver} %{buildroot}%{native_lisp}


> Did the final dump process show that it loads "native-compiled lisp"
> files for all the preloaded files?

In the native-lisp/*/preloaded directory there are 124 files, and the
following log has 113 lines which say (native compiled elisp). It is
same for all the 3 builds. This is what you were referring to, right?

Loading loadup.el (source)...
Dump mode: pdump
Using load-path
(/home/bhavin/src/emacs/emacs-pretest-rpm/emacs-28.0.90/build-gtk/../lisp)
Loading emacs-lisp/byte-run (native compiled elisp)...

bug#52376: 28.0.90; libdir is missing from native-comp-eln-load-path with GTK3 build

[Eli Zaretskii]

> From: Bhavin Gandhi <bhavin7392@gmail.com>
> Date: Thu, 9 Dec 2021 00:49:36 +0530
> Cc: 52376@debbugs.gnu.org
> 
> On Wed, 8 Dec 2021 at 23:56, Eli Zaretskii <eliz@gnu.org> wrote:
> > >
> > > No clean-up is done between the two builds.
> >
> > Maybe you should do the cleanup.
> 
> How can I do that? By extracting the source tar again?

That's one way; another is to say "make extraclean".

> > Also you didn't show the "make install" commands -- are they
> > installing into different places?  Where are the corresponding
> > emacs.pdmp files?
> 
> The way this is being done right now is:
> 
> After all three are built, make install is run from build-gtk directory
> only. And for other two, only relevant files are copied over to the
> required locations as:
> 
> /usr/bin/emacs => /etc/alternatives/emacs => /usr/bin/emacs-28.0.90
> (actual binary)
> /usr/bin/emacs-28.0.90.pdmp
> /usr/bin/emacs-lucid => /etc/alternatives/emacs-lucid =>
> /usr/bin/emacs-28.0.90-lucid
> /usr/bin/emacs-28.0.90-lucid.pdmp
> 
> The corresponding directories from native-lisp for each of them are
> copied over as well to /usr/lib64/emacs/28.0.90/native-lisp/
> 
> This means the *.el and *.elc files from GTK3 build are used by the
> other two.

But what about the *.eln files? are they also reused?  That cannot
work, I think.

Also, I think the games you play with the *.pdmp files is the reason
for the problem.  Emacs decides where to look for the *.eln files by
the place where it finds the .pdmp file, and you rename those and use
symlinks for them.  I'm not sure the startup code supports what you
expect it to support.  I suggest to step in a debugger through the
code in load_pdump, and see which of the *.pdmp files it finds in the
problematic case; my guess is that it finds the one in the build
directory, not in the installation directory.

> These are the steps which happen after building all three:
> %{buildroot}: the final destination from which the RPM package is
> created, it is a chroot.
> %{version}: 28.0.90 here
> %{native_lisp}: /usr/lib64/emacs/28.0.90/native-lisp
> 
> # Remove versioned file so that we end up with .1 suffix and only one DOC file
> rm build-{gtk,lucid,nox}/src/emacs-%{version}.*
> 
> cd build-gtk
> %make_install
> cd ..
> 
> # Let alternatives manage the symlink
> rm %{buildroot}%{_bindir}/emacs
> touch %{buildroot}%{_bindir}/emacs
> 
> # Remove emacs.pdmp from common (we copy it over in later steps)
> rm %{buildroot}%{emacs_libexecdir}/emacs.pdmp
> 
> # Install emacs.pdmp of the emacs with GTK+
> install -p -m 0644 build-gtk/src/emacs.pdmp
> %{buildroot}%{_bindir}/emacs-%{version}.pdmp
> 
> # Install the emacs with LUCID toolkit
> install -p -m 0755 build-lucid/src/emacs
> %{buildroot}%{_bindir}/emacs-%{version}-lucid
> install -p -m 0644 build-lucid/src/emacs.pdmp
> %{buildroot}%{_bindir}/emacs-%{version}-lucid.pdmp
> 
> # Install the emacs without X
> install -p -m 0755 build-nox/src/emacs
> %{buildroot}%{_bindir}/emacs-%{version}-nox
> install -p -m 0644 build-nox/src/emacs.pdmp
> %{buildroot}%{_bindir}/emacs-%{version}-nox.pdmp
> 
> lucid_comp_native_ver=$(ls -1 build-lucid/native-lisp)
> nox_comp_native_ver=$(ls -1 build-nox/native-lisp)
> cp -ar build-lucid/native-lisp/${lucid_comp_native_ver}
> %{buildroot}%{native_lisp}
> cp -ar build-nox/native-lisp/${nox_comp_native_ver} %{buildroot}%{native_lisp}

I see you copy the *.eln files for the Lucid and no-X build, but not
for the GTK3 build?

> > Did the final dump process show that it loads "native-compiled lisp"
> > files for all the preloaded files?
> 
> In the native-lisp/*/preloaded directory there are 124 files, and the
> following log has 113 lines which say (native compiled elisp). It is
> same for all the 3 builds. This is what you were referring to, right?

Yes.

bug#52376: 28.0.90; libdir is missing from native-comp-eln-load-path with GTK3 build

[Bhavin Gandhi]

On Thu, 9 Dec 2021 at 01:41, Eli Zaretskii <eliz@gnu.org> wrote:
> > This means the *.el and *.elc files from GTK3 build are used by the
> > other two.
>
> But what about the *.eln files? are they also reused?  That cannot
> work, I think.

They are not. The *.eln files are different for each binary, at the end
of installation this is what I get (one directory per variant).

$ ls /usr/lib64/emacs/28.0.90/native-lisp/
28.0.90-619a407c  28.0.90-a325c617  28.0.90-f21cc02e

> Also, I think the games you play with the *.pdmp files is the reason
> for the problem.  Emacs decides where to look for the *.eln files by
> the place where it finds the .pdmp file, and you rename those and use
> symlinks for them.  I'm not sure the startup code supports what you
> expect it to support.

Understood. The symlinks are for the binaries and not for the .pdmp
files though.

The result is same if I start /usr/bin/emacs-28.0.90 which has a
/usr/bin/emacs-28.0.90.pdmp along with it.

> I suggest to step in a debugger through the
> code in load_pdump, and see which of the *.pdmp files it finds in the
> problematic case; my guess is that it finds the one in the build
> directory, not in the installation directory.

Okay. I will try to do that and see what it shows.

> I see you copy the *.eln files for the Lucid and no-X build, but not
> for the GTK3 build?

Those are installed by make install command, initially I was copying
those like Lucid and no-X, but that didn't make any difference.

Thank you for the help so far. I will experiment and post results here.

bug#52376: 28.0.90; libdir is missing from native-comp-eln-load-path with GTK3 build

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Bhavin Gandhi <bhavin7392@gmail.com>
>> Date: Thu, 9 Dec 2021 00:49:36 +0530
>> Cc: 52376@debbugs.gnu.org
>> 
>> On Wed, 8 Dec 2021 at 23:56, Eli Zaretskii <eliz@gnu.org> wrote:
>> > >
>> > > No clean-up is done between the two builds.
>> >
>> > Maybe you should do the cleanup.
>> 
>> How can I do that? By extracting the source tar again?
>
> That's one way; another is to say "make extraclean".
>
>> > Also you didn't show the "make install" commands -- are they
>> > installing into different places?  Where are the corresponding
>> > emacs.pdmp files?
>> 
>> The way this is being done right now is:
>> 
>> After all three are built, make install is run from build-gtk directory
>> only. And for other two, only relevant files are copied over to the
>> required locations as:
>> 
>> /usr/bin/emacs => /etc/alternatives/emacs => /usr/bin/emacs-28.0.90
>> (actual binary)
>> /usr/bin/emacs-28.0.90.pdmp
>> /usr/bin/emacs-lucid => /etc/alternatives/emacs-lucid =>
>> /usr/bin/emacs-28.0.90-lucid
>> /usr/bin/emacs-28.0.90-lucid.pdmp
>> 
>> The corresponding directories from native-lisp for each of them are
>> copied over as well to /usr/lib64/emacs/28.0.90/native-lisp/
>> 
>> This means the *.el and *.elc files from GTK3 build are used by the
>> other two.
>
> But what about the *.eln files? are they also reused?  That cannot
> work, I think.
>
> Also, I think the games you play with the *.pdmp files is the reason
> for the problem.  Emacs decides where to look for the *.eln files by
> the place where it finds the .pdmp file, and you rename those and use
> symlinks for them.  I'm not sure the startup code supports what you
> expect it to support.  I suggest to step in a debugger through the
> code in load_pdump, and see which of the *.pdmp files it finds in the
> problematic case; my guess is that it finds the one in the build
> directory, not in the installation directory.
>
>> These are the steps which happen after building all three:
>> %{buildroot}: the final destination from which the RPM package is
>> created, it is a chroot.
>> %{version}: 28.0.90 here
>> %{native_lisp}: /usr/lib64/emacs/28.0.90/native-lisp
>> 
>> # Remove versioned file so that we end up with .1 suffix and only one DOC file
>> rm build-{gtk,lucid,nox}/src/emacs-%{version}.*
>> 
>> cd build-gtk
>> %make_install
>> cd ..
>> 
>> # Let alternatives manage the symlink
>> rm %{buildroot}%{_bindir}/emacs
>> touch %{buildroot}%{_bindir}/emacs
>> 
>> # Remove emacs.pdmp from common (we copy it over in later steps)
>> rm %{buildroot}%{emacs_libexecdir}/emacs.pdmp
>> 
>> # Install emacs.pdmp of the emacs with GTK+
>> install -p -m 0644 build-gtk/src/emacs.pdmp
>> %{buildroot}%{_bindir}/emacs-%{version}.pdmp
>> 
>> # Install the emacs with LUCID toolkit
>> install -p -m 0755 build-lucid/src/emacs
>> %{buildroot}%{_bindir}/emacs-%{version}-lucid
>> install -p -m 0644 build-lucid/src/emacs.pdmp
>> %{buildroot}%{_bindir}/emacs-%{version}-lucid.pdmp
>> 
>> # Install the emacs without X
>> install -p -m 0755 build-nox/src/emacs
>> %{buildroot}%{_bindir}/emacs-%{version}-nox
>> install -p -m 0644 build-nox/src/emacs.pdmp
>> %{buildroot}%{_bindir}/emacs-%{version}-nox.pdmp
>> 
>> lucid_comp_native_ver=$(ls -1 build-lucid/native-lisp)
>> nox_comp_native_ver=$(ls -1 build-nox/native-lisp)
>> cp -ar build-lucid/native-lisp/${lucid_comp_native_ver}
>> %{buildroot}%{native_lisp}
>> cp -ar build-nox/native-lisp/${nox_comp_native_ver} %{buildroot}%{native_lisp}
>
> I see you copy the *.eln files for the Lucid and no-X build, but not
> for the GTK3 build?

eln files are most likely not sharable between these two builds, copying
them will have no effect as Emacs will decide not use them.

eln files should be produced for each flavour of build independently.

BR

  Andrea

bug#52376: 28.0.90; libdir is missing from native-comp-eln-load-path with GTK3 build

[Bhavin Gandhi]

Hello Eli and Andrea

On Thu, 9 Dec 2021 at 16:36, Andrea Corallo <akrl@sdf.org> wrote:
> eln files are most likely not sharable between these two builds, copying
> them will have no effect as Emacs will decide not use them.
>
> eln files should be produced for each flavour of build independently.

Yes, I'm keeping them for each build independently, the three
directories below are for GTK3, Lucid, and no-X

$ ls /usr/lib64/emacs/28.0.90/native-lisp/
28.0.90-619a407c  28.0.90-a325c617  28.0.90-f21cc02e

On Thu, 9 Dec 2021 at 02:01, Bhavin Gandhi <bhavin7392@gmail.com> wrote:
> > I suggest to step in a debugger through the
> > code in load_pdump, and see which of the *.pdmp files it finds in the
> > problematic case; my guess is that it finds the one in the build
> > directory, not in the installation directory.
>
> Okay. I will try to do that and see what it shows.
> […]
> Thank you for the help so far. I will experiment and post results here.

So, I build the binaries again as suggested by etc/DEBUG, set breakpoint
at load_pdump function. Here is the command log and output values I got.

--8<---------------cut here---------------start------------->8---
bhavin@toolbox ~ $ gdb  --args /usr/bin/emacs -Q
Reading symbols from /usr/bin/emacs...
Reading symbols from
/usr/lib/debug/usr/bin/emacs-28.0.90-28.0.90-1.fc35.x86_64.debug...
(gdb) directory
/home/bhavin/src/emacs/emacs-pretest-rpm/emacs-28.0.90/build-gtk/src
Source directories searched:
/home/bhavin/src/emacs/emacs-pretest-rpm/emacs-28.0.90/build-gtk/src:$cdir:$cwd
(gdb) source /home/bhavin/src/emacs/emacs-pretest-rpm/emacs-28.0.90/build-gtk/src/.gdbinit
Warning: /home/bhavin/../lwlib: No such file or directory.
SIGINT is used by the debugger.
Are you sure you want to change it? (y or n) [answered Y; input not
from terminal]
DISPLAY = :0
TERM = xterm-256color
Breakpoint 1 at 0x5d58c0: file ../../src/emacs.c, line 399.
Breakpoint 2 at 0x592bad: file ../../src/xterm.c, line 10252.

(gdb) break load_pdump
Breakpoint 3 at 0x5d6679: file ../../src/emacs.c, line 829.

(gdb) run
Starting program: /usr/bin/emacs -Q

Breakpoint 3, load_pdump (argc=2, argv=0x7fffffffe1b8) at ../../src/emacs.c:829
829    {

I pressed n a couple of times here.

871      emacs_executable = load_pdump_find_executable (argv[0], &bufsize);
(gdb)
872      exec_bufsize = bufsize;

(gdb) p emacs_executable
$1 = 0xdf0190 "/usr/bin/emacs-28.0.90"

(gdb)
909      result = pdumper_load (dump_file, emacs_executable);
(gdb) p dump_file
$2 = 0xd99900 "/usr/bin/emacs-28.0.90.pdmp"

(gdb) n
910      if (result == PDUMPER_LOAD_SUCCESS)
(gdb) p result
$3 = 0
(gdb) c
Continuing.
--8<---------------cut here---------------start------------->8---

So, I think it is able to find the correct .pdmp file in this case, but
when I continue and inspect native-comp-eln-load-path, the libdir is
still not there.

In case of Lucid, it finds the /usr/bin/emacs-28.0.90-lucid.pdmp, and
when continuing the libdir is there is native-comp-eln-load-path.

bug#52376: 28.0.90; libdir is missing from native-comp-eln-load-path with GTK3 build

[Andrea Corallo]

Bhavin Gandhi <bhavin7392@gmail.com> writes:

> Hello Eli and Andrea
>
> On Thu, 9 Dec 2021 at 16:36, Andrea Corallo <akrl@sdf.org> wrote:
>> eln files are most likely not sharable between these two builds, copying
>> them will have no effect as Emacs will decide not use them.
>>
>> eln files should be produced for each flavour of build independently.
>
> Yes, I'm keeping them for each build independently, the three
> directories below are for GTK3, Lucid, and no-X
>
> $ ls /usr/lib64/emacs/28.0.90/native-lisp/
> 28.0.90-619a407c  28.0.90-a325c617  28.0.90-f21cc02e
>
> On Thu, 9 Dec 2021 at 02:01, Bhavin Gandhi <bhavin7392@gmail.com> wrote:
>> > I suggest to step in a debugger through the
>> > code in load_pdump, and see which of the *.pdmp files it finds in the
>> > problematic case; my guess is that it finds the one in the build
>> > directory, not in the installation directory.
>>
>> Okay. I will try to do that and see what it shows.
>> […]
>> Thank you for the help so far. I will experiment and post results here.
>
> So, I build the binaries again as suggested by etc/DEBUG, set breakpoint
> at load_pdump function. Here is the command log and output values I got.
>
> bhavin@toolbox ~ $ gdb  --args /usr/bin/emacs -Q
> Reading symbols from /usr/bin/emacs...
> Reading symbols from
> /usr/lib/debug/usr/bin/emacs-28.0.90-28.0.90-1.fc35.x86_64.debug...
> (gdb) directory
> /home/bhavin/src/emacs/emacs-pretest-rpm/emacs-28.0.90/build-gtk/src
> Source directories searched:
> /home/bhavin/src/emacs/emacs-pretest-rpm/emacs-28.0.90/build-gtk/src:$cdir:$cwd
> (gdb) source /home/bhavin/src/emacs/emacs-pretest-rpm/emacs-28.0.90/build-gtk/src/.gdbinit
> Warning: /home/bhavin/../lwlib: No such file or directory.
> SIGINT is used by the debugger.
> Are you sure you want to change it? (y or n) [answered Y; input not
> from terminal]
> DISPLAY = :0
> TERM = xterm-256color
> Breakpoint 1 at 0x5d58c0: file ../../src/emacs.c, line 399.
> Breakpoint 2 at 0x592bad: file ../../src/xterm.c, line 10252.
>
> (gdb) break load_pdump
> Breakpoint 3 at 0x5d6679: file ../../src/emacs.c, line 829.
>
> (gdb) run
> Starting program: /usr/bin/emacs -Q
>
> Breakpoint 3, load_pdump (argc=2, argv=0x7fffffffe1b8) at ../../src/emacs.c:829
> 829    {
>
> I pressed n a couple of times here.
>
> 871      emacs_executable = load_pdump_find_executable (argv[0], &bufsize);
> (gdb)
> 872      exec_bufsize = bufsize;
>
> (gdb) p emacs_executable
> $1 = 0xdf0190 "/usr/bin/emacs-28.0.90"
>
> (gdb)
> 909      result = pdumper_load (dump_file, emacs_executable);
> (gdb) p dump_file
> $2 = 0xd99900 "/usr/bin/emacs-28.0.90.pdmp"
>
> (gdb) n
> 910      if (result == PDUMPER_LOAD_SUCCESS)
> (gdb) p result
> $3 = 0
> (gdb) c
> Continuing.
>
> So, I think it is able to find the correct .pdmp file in this case, but
> when I continue and inspect native-comp-eln-load-path, the libdir is
> still not there.
>
> In case of Lucid, it finds the /usr/bin/emacs-28.0.90-lucid.pdmp, and
> when continuing the libdir is there is native-comp-eln-load-path.

Hi Bhavin,

I suggest to set a breakpoint at pdumper.c:5321 and print 'fndata' to
see where we do expect to find the installed eln we don't find.

Thanks

  Andrea

bug#52376: 28.0.90; libdir is missing from native-comp-eln-load-path with GTK3 build

[Bhavin Gandhi]

On Thu, 9 Dec 2021 at 22:53, Andrea Corallo <akrl@sdf.org> wrote:
>
> I suggest to set a breakpoint at pdumper.c:5321 and print 'fndata' to
> see where we do expect to find the installed eln we don't find.

It did not reach that breakpoint, and Emacs just started as usual. Seems
like the RELOC_NATIVE_COMP_UNIT case is not getting matched.

bug#52376: 28.0.90; libdir is missing from native-comp-eln-load-path with GTK3 build

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: Eli Zaretskii <eliz@gnu.org>, 52376@debbugs.gnu.org
> Date: Thu, 09 Dec 2021 17:23:22 +0000
> 
> I suggest to set a breakpoint at pdumper.c:5321 and print 'fndata' to
> see where we do expect to find the installed eln we don't find.

Yes, you should continue stepping through the code to see in which of
the directories does it look for the *.eln files.  It decides on that
according to the place where it found the .pdmp file, see
pdumper_set_emacs_execdir, and then the use of emacs_execdir in
pdumper.c.

bug#52376: 28.0.90; libdir is missing from native-comp-eln-load-path with GTK3 build

[Andrea Corallo]

Bhavin Gandhi <bhavin7392@gmail.com> writes:

> On Thu, 9 Dec 2021 at 22:53, Andrea Corallo <akrl@sdf.org> wrote:
>>
>> I suggest to set a breakpoint at pdumper.c:5321 and print 'fndata' to
>> see where we do expect to find the installed eln we don't find.
>
> It did not reach that breakpoint, and Emacs just started as usual. Seems
> like the RELOC_NATIVE_COMP_UNIT case is not getting matched.

Actually this is very bizarre.  A native compiled Emacs that has been
dumped has to go through there to startup so either:

1- we are debugging temacs and non emacs
2- emacs was compiled with native compilation
3- the breakpoint is not working as expected, maybe because it's an
optimized build.

I guess most likely we are looking at case 3 here.  Could you try in
this case building pdumper.c (or all Emacs) with -O0 -g3?

Thanks

  Andrea

bug#52376: 28.0.90; libdir is missing from native-comp-eln-load-path with GTK3 build

[Andrea Corallo]

Andrea Corallo <akrl@sdf.org> writes:

> Bhavin Gandhi <bhavin7392@gmail.com> writes:
>
>> On Thu, 9 Dec 2021 at 22:53, Andrea Corallo <akrl@sdf.org> wrote:
>>>
>>> I suggest to set a breakpoint at pdumper.c:5321 and print 'fndata' to
>>> see where we do expect to find the installed eln we don't find.
>>
>> It did not reach that breakpoint, and Emacs just started as usual. Seems
>> like the RELOC_NATIVE_COMP_UNIT case is not getting matched.
>
> Actually this is very bizarre.  A native compiled Emacs that has been
> dumped has to go through there to startup so either:
>
> 1- we are debugging temacs and non emacs
> 2- emacs was compiled with native compilation
> 3- the breakpoint is not working as expected, maybe because it's an
> optimized build.
>
> I guess most likely we are looking at case 3 here.  Could you try in
> this case building pdumper.c (or all Emacs) with -O0 -g3?
>
> Thanks
>
>   Andrea

Just to be 100% sure, one mine src/pdumper.c. from emacs28 line 5321 is:

           if (file_access_p (fndata, F_OK))

Thanks

  Andrea

bug#52376: 28.0.90; libdir is missing from native-comp-eln-load-path with GTK3 build

[Bhavin Gandhi]

On Fri, 10 Dec 2021 at 01:44, Andrea Corallo <akrl@sdf.org> wrote:
> >> I suggest to set a breakpoint at pdumper.c:5321 and print 'fndata' to
> >> see where we do expect to find the installed eln we don't find.
> >
> > It did not reach that breakpoint, and Emacs just started as usual. Seems
> > like the RELOC_NATIVE_COMP_UNIT case is not getting matched.
>
> Actually this is very bizarre.  A native compiled Emacs that has been
> dumped has to go through there to startup so either:
>
> 1- we are debugging temacs and non emacs
> 2- emacs was compiled with native compilation
> 3- the breakpoint is not working as expected, maybe because it's an
> optimized build.
>
> I guess most likely we are looking at case 3 here.  Could you try in
> this case building pdumper.c (or all Emacs) with -O0 -g3?

Emacs was indeed compiled with -O0 -g3, but to rule out that possibility,
I compiled it again with these two flags. And still it is showing same
behaviour, it goes ahead without stopping at the breakpoint.

> Just to be 100% sure, one mine src/pdumper.c. from emacs28 line 5321 is:
>
>            if (file_access_p (fndata, F_OK))

In my case that line is at pdumper.c:5319

When I test the same breakpoint with the Lucid build I get the
following, it is able to find the correct paths here.

--8<---------------cut here---------------start------------->8---
$ gdb  --args /usr/bin/emacs-28.0.90-lucid -Q
Reading symbols from /usr/bin/emacs-28.0.90-lucid...
Reading symbols from
/usr/lib/debug/usr/bin/emacs-28.0.90-lucid-28.0.90-1.fc35.x86_64.debug...
(gdb) break pdumper.c:5319
Breakpoint 3 at 0x692487: file ../../src/pdumper.c, line 5319.
(gdb) run
Starting program: /usr/bin/emacs-28.0.90-lucid -Q

Breakpoint 3, dump_do_dump_relocation (dump_base=140737265905664,
reloc=...) at ../../src/pdumper.c:5319
5319            if (file_access_p (fndata, F_OK))
(gdb) p *comp_u
$3 = {
  header = {
    size = 4611686018830053383
  },
  file = XIL(0x7ffff3295623),
  optimize_qualities = XIL(0x7ffff3295593),
  lambda_gc_guard_h = XIL(0xe4d5f5),
  lambda_c_name_idx_h = XIL(0x7ffff3292625),
  data_fdoc_v = XIL(0),
  data_vec = XIL(0x7ffff328dc4d),
  data_impure_vec = XIL(0x7ffff2beb415),
  data_imp_relocs = 0x0,
  loaded_once = false,
  load_ongoing = false,
  handle = 0x0
}
(gdb) p comp_u->file
$4 = XIL(0x7ffff3295623)
(gdb) pr
("../lib64/emacs/28.0.90/native-lisp/28.0.90-be2fc0ab/preloaded/window-0d1b8b93-41a00537.eln"
. "../native-lisp/28.0.90-be2fc0ab/preloaded/window-0d1b8b93-41a00537.eln")
(gdb) p cu_file1
$5 = XIL(0x7ffff3295654)
(gdb) pr
"../lib64/emacs/28.0.90/native-lisp/28.0.90-be2fc0ab/preloaded/window-0d1b8b93-41a00537.eln"
(gdb) p cu_file2
$6 = XIL(0x7ffff3295634)
(gdb) pr
"../native-lisp/28.0.90-be2fc0ab/preloaded/window-0d1b8b93-41a00537.eln"
 --8<---------------cut here---------------end--------------->8---

From the GTK3 build:

Configured using:
 'configure --build=x86_64-redhat-linux-gnu
 --host=x86_64-redhat-linux-gnu --program-prefix=
 --disable-dependency-tracking --prefix=/usr --exec-prefix=/usr
 --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc
 --datadir=/usr/share --includedir=/usr/include --libdir=/usr/lib64
 --libexecdir=/usr/libexec --localstatedir=/var
 --sharedstatedir=/var/lib --mandir=/usr/share/man
 --infodir=/usr/share/info --with-dbus --with-gif --with-jpeg --with-png
 --with-rsvg --with-tiff --with-xft --with-xpm --with-x-toolkit=gtk3
 --with-gpm=no --with-xwidgets --with-modules --with-harfbuzz
 --with-cairo --with-json --with-native-compilation
 --enable-checking=yes,glyphs --enable-check-lisp-object-type
 build_alias=x86_64-redhat-linux-gnu host_alias=x86_64-redhat-linux-gnu
 CC=gcc 'CFLAGS=-DMAIL_USE_LOCKF -flto=auto -ffat-lto-objects
 -fexceptions -grecord-gcc-switches -pipe -Wall -Werror=format-security
 -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS
 -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong
 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic
 -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection
 -O0 -g3' LDFLAGS=-Wl,-z,relro
 PKG_CONFIG_PATH=:/usr/lib64/pkgconfig:/usr/share/pkgconfig'


From the Lucid build:
Configured using:
 'configure --build=x86_64-redhat-linux-gnu
 --host=x86_64-redhat-linux-gnu --program-prefix=
 --disable-dependency-tracking --prefix=/usr --exec-prefix=/usr
 --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc
 --datadir=/usr/share --includedir=/usr/include --libdir=/usr/lib64
 --libexecdir=/usr/libexec --localstatedir=/var
 --sharedstatedir=/var/lib --mandir=/usr/share/man
 --infodir=/usr/share/info --with-dbus --with-gif --with-jpeg --with-png
 --with-rsvg --with-tiff --with-xft --with-xpm --with-x-toolkit=lucid
 --with-gpm=no --with-modules --with-harfbuzz --with-cairo --with-json
 --with-native-compilation build_alias=x86_64-redhat-linux-gnu
 host_alias=x86_64-redhat-linux-gnu CC=gcc 'CFLAGS=-DMAIL_USE_LOCKF -O2
 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches
 -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2
 -Wp,-D_GLIBCXX_ASSERTIONS
 -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong
 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic
 -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection'
 LDFLAGS=-Wl,-z,relro
 PKG_CONFIG_PATH=:/usr/lib64/pkgconfig:/usr/share/pkgconfig'

bug#52376: 28.0.90; libdir is missing from native-comp-eln-load-path with GTK3 build

[Andrea Corallo]

Bhavin Gandhi <bhavin7392@gmail.com> writes:

> On Fri, 10 Dec 2021 at 01:44, Andrea Corallo <akrl@sdf.org> wrote:
>> >> I suggest to set a breakpoint at pdumper.c:5321 and print 'fndata' to
>> >> see where we do expect to find the installed eln we don't find.
>> >
>> > It did not reach that breakpoint, and Emacs just started as usual. Seems
>> > like the RELOC_NATIVE_COMP_UNIT case is not getting matched.
>>
>> Actually this is very bizarre.  A native compiled Emacs that has been
>> dumped has to go through there to startup so either:
>>
>> 1- we are debugging temacs and non emacs
>> 2- emacs was compiled with native compilation
>> 3- the breakpoint is not working as expected, maybe because it's an
>> optimized build.
>>
>> I guess most likely we are looking at case 3 here.  Could you try in
>> this case building pdumper.c (or all Emacs) with -O0 -g3?
>
> Emacs was indeed compiled with -O0 -g3, but to rule out that possibility,
> I compiled it again with these two flags. And still it is showing same
> behaviour, it goes ahead without stopping at the breakpoint.

Okay, that's very bizarre tho.

>> Just to be 100% sure, one mine src/pdumper.c. from emacs28 line 5321 is:
>>
>>            if (file_access_p (fndata, F_OK))
>
> In my case that line is at pdumper.c:5319

Maybe that was the reason?

> When I test the same breakpoint with the Lucid build I get the
> following, it is able to find the correct paths here.
>
> --8<---------------cut here---------------start------------->8---
> $ gdb  --args /usr/bin/emacs-28.0.90-lucid -Q
> Reading symbols from /usr/bin/emacs-28.0.90-lucid...
> Reading symbols from
> /usr/lib/debug/usr/bin/emacs-28.0.90-lucid-28.0.90-1.fc35.x86_64.debug...
> (gdb) break pdumper.c:5319
> Breakpoint 3 at 0x692487: file ../../src/pdumper.c, line 5319.
> (gdb) run
> Starting program: /usr/bin/emacs-28.0.90-lucid -Q
>
> Breakpoint 3, dump_do_dump_relocation (dump_base=140737265905664,
> reloc=...) at ../../src/pdumper.c:5319
> 5319            if (file_access_p (fndata, F_OK))
> (gdb) p *comp_u
> $3 = {
>   header = {
>     size = 4611686018830053383
>   },
>   file = XIL(0x7ffff3295623),
>   optimize_qualities = XIL(0x7ffff3295593),
>   lambda_gc_guard_h = XIL(0xe4d5f5),
>   lambda_c_name_idx_h = XIL(0x7ffff3292625),
>   data_fdoc_v = XIL(0),
>   data_vec = XIL(0x7ffff328dc4d),
>   data_impure_vec = XIL(0x7ffff2beb415),
>   data_imp_relocs = 0x0,
>   loaded_once = false,
>   load_ongoing = false,
>   handle = 0x0
> }
> (gdb) p comp_u->file
> $4 = XIL(0x7ffff3295623)
> (gdb) pr
> ("../lib64/emacs/28.0.90/native-lisp/28.0.90-be2fc0ab/preloaded/window-0d1b8b93-41a00537.eln"
> . "../native-lisp/28.0.90-be2fc0ab/preloaded/window-0d1b8b93-41a00537.eln")
> (gdb) p cu_file1
> $5 = XIL(0x7ffff3295654)
> (gdb) pr
> "../lib64/emacs/28.0.90/native-lisp/28.0.90-be2fc0ab/preloaded/window-0d1b8b93-41a00537.eln"
> (gdb) p cu_file2
> $6 = XIL(0x7ffff3295634)
> (gdb) pr
> "../native-lisp/28.0.90-be2fc0ab/preloaded/window-0d1b8b93-41a00537.eln"
>  --8<---------------cut here---------------end--------------->8---

Good, as mentioned could you 'fndata' now?

Thanks!

  Andrea

bug#52376: 28.0.90; libdir is missing from native-comp-eln-load-path with GTK3 build

[Bhavin Gandhi]

On Fri, 10 Dec 2021 at 15:15, Andrea Corallo <akrl@sdf.org> wrote:
> >> Just to be 100% sure, one mine src/pdumper.c. from emacs28 line 5321 is:
> >>
> >>            if (file_access_p (fndata, F_OK))
> >
> > In my case that line is at pdumper.c:5319
>
> Maybe that was the reason?

Probably not, because the build with which we are having issues i.e. the
GTK3 one, *still has* the same problem. It just starts directly without stopping
at the breakpoint.

> Good, as mentioned could you 'fndata' now?

Sorry for the confusion here, but the working command logs are from
Lucid build (which has been working fine from the beginning)

The working build gives fndata like this, while the GTK3 one
(problematic one), still goes ahead and starts without stopping at the
breakpoint.

(gdb) p fndata
$1 = 0xe5f2f8 "/usr/bin/../lib64/emacs/28.0.90/native-lisp/28.0.90-74229b0e/preloaded/window-0d1b8b93-41a00537.eln"

bug#52376: 28.0.90; libdir is missing from native-comp-eln-load-path with GTK3 build

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Date: Fri, 10 Dec 2021 09:45:20 +0000
> Cc: 52376@debbugs.gnu.org
> 
> >>            if (file_access_p (fndata, F_OK))
> >
> > In my case that line is at pdumper.c:5319
> 
> Maybe that was the reason?

I don't think so: 5319 is correct for the version distributed with the
pretest tarball.  You, Andrea, are probably looking at the current
emacs-28 branch or some other version.

bug#52376: 28.0.90; libdir is missing from native-comp-eln-load-path with GTK3 build

[Eli Zaretskii]

> From: Bhavin Gandhi <bhavin7392@gmail.com>
> Date: Fri, 10 Dec 2021 15:30:06 +0530
> Cc: 52376@debbugs.gnu.org
> 
> > Good, as mentioned could you 'fndata' now?
> 
> Sorry for the confusion here, but the working command logs are from
> Lucid build (which has been working fine from the beginning)
> 
> The working build gives fndata like this, while the GTK3 one
> (problematic one), still goes ahead and starts without stopping at the
> breakpoint.
> 
> (gdb) p fndata
> $1 = 0xe5f2f8 "/usr/bin/../lib64/emacs/28.0.90/native-lisp/28.0.90-74229b0e/preloaded/window-0d1b8b93-41a00537.eln"

Which breakpoint didn't break? the one at line 5321?  That's because
the line number was incorrect, I think.  Please set the breakpoint at
line 5319 instead, and then show us the results from the problematic
build.

bug#52376: 28.0.90; libdir is missing from native-comp-eln-load-path with GTK3 build

[Bhavin Gandhi]

On Fri, 10 Dec 2021 at 17:06, Eli Zaretskii <eliz@gnu.org> wrote:
> Which breakpoint didn't break? the one at line 5321?  That's because
> the line number was incorrect, I think.  Please set the breakpoint at
> line 5319 instead, and then show us the results from the problematic
> build.

The one at 5319 didn't break.

(gdb) break pdumper.c:5319
Breakpoint 3 at 0x69dd1b: file ../../src/pdumper.c, line 5319.
(gdb) run
Starting program: /usr/bin/emacs -Q
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
[New Thread 0x7fffeabbf640 (LWP 2483256)]
[New Thread 0x7fffea345640 (LWP 2483257)]
Gtk-Message: 17:16:59.464: Failed to load module "pk-gtk-module"
Gtk-Message: 17:16:59.465: Failed to load module "pk-gtk-module"
[New Thread 0x7fffe9a21640 (LWP 2483258)]
[New Thread 0x7fffe911a640 (LWP 2483259)]
[Thread 0x7fffe911a640 (LWP 2483259) exited]
[New Thread 0x7fffe911a640 (LWP 2483260)]
[New Thread 0x7fffe8919640 (LWP 2483261)]
[Thread 0x7fffe911a640 (LWP 2483260) exited]
[Thread 0x7fffe8919640 (LWP 2483261) exited]
[Thread 0x7fffe9a21640 (LWP 2483258) exited]
[Thread 0x7fffeabbf640 (LWP 2483256) exited]
[Thread 0x7fffeb9a3400 (LWP 2483252) exited]
# Here I did C-x C-c
[Inferior 1 (process 2483252) exited normally]
(gdb)

bug#52376: 28.0.90; libdir is missing from native-comp-eln-load-path with GTK3 build

[Eli Zaretskii]

> From: Bhavin Gandhi <bhavin7392@gmail.com>
> Date: Fri, 10 Dec 2021 17:21:58 +0530
> Cc: akrl@sdf.org, 52376@debbugs.gnu.org
> 
> On Fri, 10 Dec 2021 at 17:06, Eli Zaretskii <eliz@gnu.org> wrote:
> > Which breakpoint didn't break? the one at line 5321?  That's because
> > the line number was incorrect, I think.  Please set the breakpoint at
> > line 5319 instead, and then show us the results from the problematic
> > build.
> 
> The one at 5319 didn't break.

Does _any_ line of code under RELOC_NATIVE_COMP_UNIT case get
executed?  What if you set the breakpoint here:

  #ifdef HAVE_NATIVE_COMP
      case RELOC_NATIVE_COMP_UNIT:
	{
	  static enum { UNKNOWN, LOCAL_BUILD, INSTALLED } installation_state;
	  struct Lisp_Native_Comp_Unit *comp_u = <<<<<<<<<<<<<<<<<<<<<<<
	    dump_ptr (dump_base, reloc_offset);
	  comp_u->lambda_gc_guard_h = CALLN (Fmake_hash_table, QCtest, Qeq);

(that's line number 5291), or on the next line, does that breakpoint
break?  If it does, please step through the code and tell what gets
executed and why.

bug#52376: 28.0.90; libdir is missing from native-comp-eln-load-path with GTK3 build

[Bhavin Gandhi]

On Fri, 10 Dec 2021 at 17:51, Eli Zaretskii <eliz@gnu.org> wrote:
>
> Does _any_ line of code under RELOC_NATIVE_COMP_UNIT case get
> executed?  What if you set the breakpoint here:
>
>   #ifdef HAVE_NATIVE_COMP
>       case RELOC_NATIVE_COMP_UNIT:
>         {
>           static enum { UNKNOWN, LOCAL_BUILD, INSTALLED } installation_state;
>           struct Lisp_Native_Comp_Unit *comp_u = <<<<<<<<<<<<<<<<<<<<<<<
>             dump_ptr (dump_base, reloc_offset);
>           comp_u->lambda_gc_guard_h = CALLN (Fmake_hash_table, QCtest, Qeq);
>
> (that's line number 5291), or on the next line, does that breakpoint
> break?  If it does, please step through the code and tell what gets
> executed and why.

Nope. It does not break on 5291 (RELOC_NATIVE_COMP_UNIT case) and on
5362 (RELOC_NATIVE_SUBR) as well.

When I set breakpoint on pdumper.c:5264, and print emacs_execdir it
gives "$1 = 0x0". So, HAVE_NATIVE_COMP is indeed defined.

bug#52376: 28.0.90; libdir is missing from native-comp-eln-load-path with GTK3 build

[Eli Zaretskii]

> From: Bhavin Gandhi <bhavin7392@gmail.com>
> Date: Fri, 10 Dec 2021 20:08:07 +0530
> Cc: akrl@sdf.org, 52376@debbugs.gnu.org
> 
> On Fri, 10 Dec 2021 at 17:51, Eli Zaretskii <eliz@gnu.org> wrote:
> >
> > Does _any_ line of code under RELOC_NATIVE_COMP_UNIT case get
> > executed?  What if you set the breakpoint here:
> >
> >   #ifdef HAVE_NATIVE_COMP
> >       case RELOC_NATIVE_COMP_UNIT:
> >         {
> >           static enum { UNKNOWN, LOCAL_BUILD, INSTALLED } installation_state;
> >           struct Lisp_Native_Comp_Unit *comp_u = <<<<<<<<<<<<<<<<<<<<<<<
> >             dump_ptr (dump_base, reloc_offset);
> >           comp_u->lambda_gc_guard_h = CALLN (Fmake_hash_table, QCtest, Qeq);
> >
> > (that's line number 5291), or on the next line, does that breakpoint
> > break?  If it does, please step through the code and tell what gets
> > executed and why.
> 
> Nope. It does not break on 5291 (RELOC_NATIVE_COMP_UNIT case) and on
> 5362 (RELOC_NATIVE_SUBR) as well.
> 
> When I set breakpoint on pdumper.c:5264, and print emacs_execdir it
> gives "$1 = 0x0". So, HAVE_NATIVE_COMP is indeed defined.

Which probably means the .pdmp file is somehow devoid of
natively-compiled code, i.e. it was produced by dumping the *.elc
files, not the *.eln files?

bug#52376: 28.0.90; libdir is missing from native-comp-eln-load-path with GTK3 build

[Eli Zaretskii]

> Date: Fri, 10 Dec 2021 16:50:06 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: 52376@debbugs.gnu.org, akrl@sdf.org
> 
> > Nope. It does not break on 5291 (RELOC_NATIVE_COMP_UNIT case) and on
> > 5362 (RELOC_NATIVE_SUBR) as well.
> > 
> > When I set breakpoint on pdumper.c:5264, and print emacs_execdir it
> > gives "$1 = 0x0". So, HAVE_NATIVE_COMP is indeed defined.
> 
> Which probably means the .pdmp file is somehow devoid of
> natively-compiled code, i.e. it was produced by dumping the *.elc
> files, not the *.eln files?

Does the command below show anything at all?

  $ strings /usr/bin/emacs-28.0.90.pdmp | fgrep .eln

Actually, thinking about it now: why is the emacs.pdmp file in
/usr/bin?  It should be under /usr/libexec/emacs/28.0.90/.

bug#52376: 28.0.90; libdir is missing from native-comp-eln-load-path with GTK3 build

[Bhavin Gandhi]

On Fri, 10 Dec 2021 at 20:38, Eli Zaretskii <eliz@gnu.org> wrote:
> > Which probably means the .pdmp file is somehow devoid of
> > natively-compiled code, i.e. it was produced by dumping the *.elc
> > files, not the *.eln files?
>
> Does the command below show anything at all?
>
>   $ strings /usr/bin/emacs-28.0.90.pdmp | fgrep .eln

Only one string.

$ strings /usr/bin/emacs-28.0.90.pdmp | fgrep .eln
\.eln\'

> Actually, thinking about it now: why is the emacs.pdmp file in
> /usr/bin?  It should be under /usr/libexec/emacs/28.0.90/.

Yeah, I had to do it to support installation of different binaries like
emacs, emacs-lucid, emacs-nox on the same system. The hardcoded case
from emacs.c:917, which uses path_exec, looks only for the emacs.pdmp file
and not the executable name i.e. emacs-28.0.90.pdmp in my case.

  /* Look for "emacs.pdmp" in PATH_EXEC.  We hardcode "emacs" in
     "emacs.pdmp" so that the Emacs binary still works if the user
     copies and renames it.  */

bug#52376: 28.0.90; libdir is missing from native-comp-eln-load-path with GTK3 build

[Eli Zaretskii]

> From: Bhavin Gandhi <bhavin7392@gmail.com>
> Date: Fri, 10 Dec 2021 21:17:38 +0530
> Cc: 52376@debbugs.gnu.org, akrl@sdf.org
> 
> On Fri, 10 Dec 2021 at 20:38, Eli Zaretskii <eliz@gnu.org> wrote:
> > > Which probably means the .pdmp file is somehow devoid of
> > > natively-compiled code, i.e. it was produced by dumping the *.elc
> > > files, not the *.eln files?
> >
> > Does the command below show anything at all?
> >
> >   $ strings /usr/bin/emacs-28.0.90.pdmp | fgrep .eln
> 
> Only one string.
> 
> $ strings /usr/bin/emacs-28.0.90.pdmp | fgrep .eln
> \.eln\'

So we now know this emacs.pdmp file was produced by dumping the *.elc
files, not *.eln files.  I asked you earlier whether you saw Emacs
saying stuff like

   Loading foo (native-compiled lisp)...

when building the GTK build, and you said yes.  But somehow, the
emacs.pdmp file that you installed in /usr/bin isn't the one produced
as above.  Please examine your build procedure to understand why that
happened.

> > Actually, thinking about it now: why is the emacs.pdmp file in
> > /usr/bin?  It should be under /usr/libexec/emacs/28.0.90/.
> 
> Yeah, I had to do it to support installation of different binaries like
> emacs, emacs-lucid, emacs-nox on the same system. The hardcoded case
> from emacs.c:917, which uses path_exec, looks only for the emacs.pdmp file
> and not the executable name i.e. emacs-28.0.90.pdmp in my case.
> 
>   /* Look for "emacs.pdmp" in PATH_EXEC.  We hardcode "emacs" in
>      "emacs.pdmp" so that the Emacs binary still works if the user
>      copies and renames it.  */

Please reconsider.  When Emacs finds the .pdmp file near its
executable, it makes certain assumptions about the location of the
*.eln files that don't fit the installed Emacs.  So placing the .pdmp
files there is not a good idea.

bug#52376: 28.0.90; libdir is missing from native-comp-eln-load-path with GTK3 build

[Ken Brown]

On 12/10/2021 11:56 AM, Eli Zaretskii wrote:
> Please reconsider.  When Emacs finds the .pdmp file near its
> executable, it makes certain assumptions about the location of the
> *.eln files that don't fit the installed Emacs.  So placing the .pdmp
> files there is not a good idea.

Could you elaborate on what problems this causes?  I've been doing something 
similar in my Cygwin builds, and I'm not aware of any problems.  But maybe I 
just haven't noticed.  Here's what I have, with three versions of emacs installed:

$ ls -l /usr/bin/emacs-*
-rwxr-xr-x 1 kbrown-admin None  6221331 2021-12-05 17:50 /usr/bin/emacs-X11.exe
-rw-r--r-- 1 kbrown-admin None 14361888 2021-12-05 17:49 /usr/bin/emacs-X11.pdmp
-rwxr-xr-x 1 kbrown-admin None  5631507 2021-12-05 17:49 /usr/bin/emacs-nox.exe
-rw-r--r-- 1 kbrown-admin None 13627232 2021-12-05 17:49 /usr/bin/emacs-nox.pdmp
-rwxr-xr-x 1 kbrown-admin None  6472211 2021-12-05 17:49 /usr/bin/emacs-w32.exe
-rw-r--r-- 1 kbrown-admin None 14228224 2021-12-05 17:49 /usr/bin/emacs-w32.pdmp

$ ls -ld /usr/lib/emacs/28.0.90/native-lisp/28.0.90-*
drwxr-xr-x+ 1 kbrown-admin None 0 2021-12-05 21:20 
/usr/lib/emacs/28.0.90/native-lisp/28.0.90-18b0dc45/
drwxr-xr-x+ 1 kbrown-admin None 0 2021-12-05 21:20 
/usr/lib/emacs/28.0.90/native-lisp/28.0.90-4b3eff2b/
drwxr-xr-x+ 1 kbrown-admin None 0 2021-12-05 21:20 
/usr/lib/emacs/28.0.90/native-lisp/28.0.90-b73ba39c/

Ken

bug#52376: 28.0.90; libdir is missing from native-comp-eln-load-path with GTK3 build

[Eli Zaretskii]

> Date: Fri, 10 Dec 2021 14:53:30 -0500
> Cc: 52376@debbugs.gnu.org, akrl@sdf.org
> From: Ken Brown <kbrown@cornell.edu>
> 
> On 12/10/2021 11:56 AM, Eli Zaretskii wrote:
> > Please reconsider.  When Emacs finds the .pdmp file near its
> > executable, it makes certain assumptions about the location of the
> > *.eln files that don't fit the installed Emacs.  So placing the .pdmp
> > files there is not a good idea.
> 
> Could you elaborate on what problems this causes?  I've been doing something 
> similar in my Cygwin builds, and I'm not aware of any problems.  But maybe I 
> just haven't noticed.  Here's what I have, with three versions of emacs installed:

Look at the logic in pdumper.c:

	/* Check just once if this is a local build or Emacs was installed.  */
	/* Can't use expand-file-name here, because we are too early
	   in the startup, and we will crash at least on WINDOWSNT.  */
	if (installation_state == UNKNOWN)
	  {
	    eln_fname = make_uninit_string (execdir_len + fn1_len);
	    fndata = SSDATA (eln_fname);
	    memcpy (fndata, emacs_execdir, execdir_len);
	    memcpy (fndata + execdir_len, SSDATA (cu_file1), fn1_len);
	    if (file_access_p (fndata, F_OK))
	      installation_state = INSTALLED;
	    else
	      {
		eln_fname = make_uninit_string (execdir_len + fn2_len);
		fndata = SSDATA (eln_fname);
		memcpy (fndata, emacs_execdir, execdir_len);
		memcpy (fndata + execdir_len, SSDATA (cu_file2), fn2_len);
		installation_state = LOCAL_BUILD;
	      }
	    fixup_eln_load_path (eln_fname);
	  }

It decides whether this is an installed or an uninstalled Emacs
according to whether it can access the .eln file under the first or
the second candidate directory.  If you are careful enough to use real
/usr/bin and /usr/lib directories, this will work.  But if
/usr/bin/emacs is a symlink, and there's no real /usr/lib directory to
go with it, the logic shown about can easily fail.

This logic was generally designed to handle the "normal" case, where
the pdumper file is under /usr/libexec; the case where it is near the
Emacs binary is typically when you run Emacs uninstalled.  It does
attempt to handle several alternative use cases, but not every
possible one of them.  And the logic is tricky enough so I tend to
forget its details all the time, and need to read the code again...

So caveat emptor.

Why don't you configure each Emacs build with a different libexecdir
instead?

bug#52376: 28.0.90; libdir is missing from native-comp-eln-load-path with GTK3 build

[Bhavin Gandhi]

On Fri, 10 Dec 2021 at 22:26, Eli Zaretskii <eliz@gnu.org> wrote:
>
> So we now know this emacs.pdmp file was produced by dumping the *.elc
> files, not *.eln files.  I asked you earlier whether you saw Emacs
> saying stuff like
>
>    Loading foo (native-compiled lisp)...
>
> when building the GTK build, and you said yes.  But somehow, the
> emacs.pdmp file that you installed in /usr/bin isn't the one produced
> as above.  Please examine your build procedure to understand why that
> happened.

You are right. The one I saw with (native-compiled lisp), and the one
being installed are different.

When all three variants are built, the make install is run from
build-gtk directory at the end. And during this install command, it is
creating a emacs.pdmp again, where build says Loading foo only.

Can be reproduced with following commands:

mkdir build-gtk
cd build-gtk/
ln -s ../configure .
./configure <for GTK>
make bootstrap
make
cd ..

mkdir build-nox
cd build-nox/
ln -s ../configure .
./configure <for No-X>
make bootstrap
make

cd ../build-gtk
sudo make install
^^^ This runs configure, make all lib etc again, I can see a new
emacs.pdmp file being created which loads foo.el from source.

Is this type of build and install flow supported?

AFAIK, the RPM build requires the all build commands to be together in
one section, and then all install related commands in a separate
section. That's why this kind of flow is being used it seems.

Two workarounds I can think of are,
1. Build GTK at the end and then do make install from build-gtk
2. Extract the source tar in different directories to keep these builds
   completely separate from each other.

bug#52376: 28.0.90; libdir is missing from native-comp-eln-load-path with GTK3 build

[Ken Brown]

[-- Attachment #1: Type: text/plain, Size: 1613 bytes --]

On 1/1/2022 12:12 PM, Bhavin Gandhi wrote:
> When all three variants are built, the make install is run from
> build-gtk directory at the end. And during this install command, it is
> creating a emacs.pdmp again, where build says Loading foo only.
> 
> Can be reproduced with following commands:
> 
> mkdir build-gtk
> cd build-gtk/
> ln -s ../configure .
> ./configure <for GTK>
> make bootstrap
> make
> cd ..
> 
> mkdir build-nox
> cd build-nox/
> ln -s ../configure .
> ./configure <for No-X>
> make bootstrap
> make
> 
> cd ../build-gtk
> sudo make install
> ^^^ This runs configure, make all lib etc again, I can see a new
> emacs.pdmp file being created which loads foo.el from source.
> 
> Is this type of build and install flow supported?
> 
> AFAIK, the RPM build requires the all build commands to be together in
> one section, and then all install related commands in a separate
> section. That's why this kind of flow is being used it seems.
> 
> Two workarounds I can think of are,
> 1. Build GTK at the end and then do make install from build-gtk
> 2. Extract the source tar in different directories to keep these builds
>     completely separate from each other.

In case it's of any use to you, I can show you how I handle this on Cygwin.  We 
use a tool called cygport, which processes a file emacs.cygport (attached).  You 
can think of this as the analogue of an rpm .spec file.  I realize that you're 
not familiar with cygport, but emacs.cygport is a bash script, and I suspect you 
can decipher most of it.  The crucial things to look at are the src_compile and 
src_install functions.

Ken

[-- Attachment #2: emacs.cygport --]
[-- Type: text/plain, Size: 8534 bytes --]

NAME="emacs"
VERSION="28.0.90"
RELEASE="7"

HOMEPAGE="http://www.gnu.org/software/emacs/"
CATEGORY="Editors Interpreters"
DESCRIPTION="Emacs is a powerful, customizable, self-documenting, modeless text
editor.  Emacs contains special code editing features, a scripting
language (elisp), and the capability to read mail, news, and more
without leaving the editor."

# emacs-basic is alphabetically first among the binary packages and so
# will be selected by setup if the user installs emacs but doesn't
# choose a binary package.
PKG_NAMES="${PN} ${PN}-common ${PN}-basic ${PN}-gtk ${PN}-lucid ${PN}-w32"

emacs_CATEGORY="${CATEGORY} Virtual"
emacs_CONTENTS=""
emacs_REQUIRES="emacs-bin"
emacs_SUMMARY="The extensible, customizable, self-documenting real-time display editor"

emacs_basic_SUMMARY="Emacs binaries with no GUI"
emacs_basic_REQUIRES="emacs-common"
emacs_basic_PROVIDES="emacs-bin"
emacs_basic_DESCRIPTION="${DESCRIPTION}

This package provides binaries for a non-GUI emacs."

EMACS_EXECDIR=usr/libexec/emacs/${VERSION}/${ARCH}-pc-cygwin

emacs_common_SUMMARY="Common files needed for all Emacs binaries"
emacs_common_OBSOLETES="emacs-el"
emacs_common_REQUIRES="terminfo-extra mailutils"
emacs_common_DESCRIPTION="${DESCRIPTION}

This package contains files and documentation needed by the binary
packages emacs-basic, emacs-gtk, emacs-lucid, and emacs-w32."

# /etc/postinstall/emacs.sh is created by cygport because of the
# presence of an icon.
emacs_common_CONTENTS="
	etc/postinstall/emacs.sh
	usr/bin/ebrowse.exe
	usr/bin/emacs.ico
	usr/bin/make-emacs-shortcut
	usr/bin/set-emacs-default.sh
	usr/include/
	usr/lib/systemd/
	${EMACS_EXECDIR}/hexl.exe
	${EMACS_EXECDIR}/rcs2log
	usr/share/
	var/lib/rebase/dynpath.d/emacs"

# Use package lists for the remaining packages so we can add the
# appropriate native-lisp directory at compile time.

emacs_gtk_SUMMARY="Emacs binaries using the X11 GUI with the GTK+ toolkit"
emacs_gtk_OBSOLETES="emacs-X11"
emacs_gtk_REQUIRES="emacs-common bitstream-vera-fonts"
emacs_gtk_PROVIDES="emacs-bin"
emacs_gtk_DESCRIPTION="${DESCRIPTION}

This package provides binaries for an gtk emacs with the GTK+ toolkit.
Install it if you want to run emacs using the X window system and GTK+
for display.  The binaries also can be used in non-gtk mode."

emacs_lucid_SUMMARY="Emacs binaries using the X11 GUI with the Lucid toolkit"
emacs_lucid_REQUIRES="emacs-common bitstream-vera-fonts"
emacs_lucid_PROVIDES="emacs-bin"
emacs_lucid_DESCRIPTION="${DESCRIPTION}

This package provides binaries for an X11 emacs with the Lucid
toolkit.  Install it if you want to run emacs using the X window
system and Lucid for display.  The binaries also can be used in
non-X11 mode."

emacs_w32_SUMMARY="Emacs binaries using the native Windows GUI"
emacs_w32_REQUIRES="emacs-common"
emacs_w32_PROVIDES="emacs-bin"
emacs_w32_DESCRIPTION="${DESCRIPTION}

This package provides emacs binaries that use the native Windows GUI
for display."

# SRC_URI="mirror://gnu/emacs/emacs-${VERSION}.tar.xz"

SRC_URI="https://alpha.gnu.org/gnu/emacs/pretest/emacs-${VERSION}.tar.xz"
# SRC_URI="https://alpha.gnu.org/gnu/emacs/pretest/emacs-${VERSION}-rc2.tar.xz"

# Can generate a source tarball by running './make-dist --snapshot
# --xz --tests' in git repo.  Note: It’s possible to have an
# outdated loaddefs.el in this way.  See admin/make-tarball.txt for
# more careful instructions on making the tarball.  It might suffice
# to run ’make -C lisp autoloads’ first.

# GIT_URI="git://git.savannah.gnu.org/emacs.git"
# GIT_BRANCH="emacs-28"
# GIT_REV=fa0b34b716
# inherit git

SRC_URI+="
	make-emacs-shortcut
	set-emacs-default.sh
	README.Cygwin
"

PATCH_URI="bt.patch"
PATCH_URI+=" no_dialog_on_abort.patch"
PATCH_URI+=" 0001-Avoid-delays-waiting-for-input-on-systems-without-SI.patch"
PATCH_URI+=" 0002-Make-process_pending_signals-useful-on-systems-witho.patch"
PATCH_URI+=" 0001-Make-the-installed-pmdp-file-use-a-fingerprint.patch"
PATCH_URI+=" 0001-Change-fingerprint-to-output-to-stdout.patch"

DOCS="README.Cygwin"

# For emacs-29 we'll also want libfribidi-devel, libwebp-devel, and
# libsqlite3-devel.
BUILD_REQUIRES=" \
	libSM-devel \
	libX11-devel \
	libX11-xcb-devel \
	libXaw3d-devel \
	libXft-devel \
	libXpm-devel \
	libXpm-noX-devel \
	libXrender-devel \
	libdbus1-devel \
	libfontconfig-devel \
	libfreetype-devel \
	libgccjit0 \
	libgif-devel \
	libglib2.0-devel \
	libgnutls-devel \
	libgtk3-devel \
	libharfbuzz-devel \
	libiconv-devel \
	libjansson-devel \
	libjpeg-devel \
	liblcms2-devel \
	libm17n-devel \
	libncurses-devel \
	libotf-devel \
	libpng-devel \
	librsvg2-devel \
	libtiff-devel \
	libxml2-devel"

DIFF_EXCLUDES="loaddefs.el config.in *.elc loaddefs.el~ subdirs.el~ config.in~ emacs.1 *.info"

ACLOCAL_FLAGS='-I m4'

# NATIVE_COMP=yes
NATIVE_COMP=no

if [ ${NATIVE_COMP} = yes ]
then
   CYGCONF_ARGS="--with-native-compilation"
fi

WANT_AUTOCONF=2.7

src_compile() {
	cd ${S}
	cygautoreconf
	# ./autogen.sh
	cd ${B}
	for t in basic w32 gtk lucid
	do
		mkdir ../build-${t}
		cat > ${C}/emacs-${t}.list <<-EOF
			usr/bin/emacs-${t}.exe
			usr/bin/emacsclient-${t}.exe
			usr/bin/set-emacs-default-${t}.sh
			etc/postinstall/emacs-${t}.sh
			etc/preremove/emacs-${t}.sh
			EOF
	done

	cd ${B}/../build-w32
	cygconf --with-w32
	cygmake
	echo "${EMACS_EXECDIR}/emacs-$(src/emacs --fingerprint).pdmp" \
	     >> ${C}/emacs-w32.list
	if [ ${NATIVE_COMP} = yes ]
	then
	    echo "usr/lib/emacs/${VERSION}/native-lisp/$(ls native-lisp)" \
		 >> ${C}/emacs-w32.list
	fi
	
	cd ${B}/../build-gtk
	cygconf
	cygmake
	echo "${EMACS_EXECDIR}/emacs-$(src/emacs --fingerprint).pdmp" \
	     >> ${C}/emacs-gtk.list
	if [ ${NATIVE_COMP} = yes ]
	then
	    echo "usr/lib/emacs/${VERSION}/native-lisp/$(ls native-lisp)" \
		 >> ${C}/emacs-gtk.list
	fi

	cd ${B}/../build-lucid
	cygconf --with-x-toolkit=lucid
	cygmake
	echo "${EMACS_EXECDIR}/emacs-$(src/emacs --fingerprint).pdmp" \
	     >> ${C}/emacs-lucid.list
	if [ ${NATIVE_COMP} = yes ]
	then
	    echo "usr/lib/emacs/${VERSION}/native-lisp/$(ls native-lisp)" \
		 >> ${C}/emacs-lucid.list
	fi

	cd ${B}/../build-basic
	cygconf --with-x=no
	cygmake
	echo "${EMACS_EXECDIR}/emacs-$(src/emacs --fingerprint).pdmp" \
	     >> ${C}/emacs-basic.list
	if [ ${NATIVE_COMP} = yes ]
	then
	    echo "usr/lib/emacs/${VERSION}/native-lisp/$(ls native-lisp)" \
		 >> ${C}/emacs-basic.list
	fi
}

src_install() {
	cd ${B}/../build-basic
	cyginstall
	rm -fv \
		${D}/usr/bin/emacs.exe \
		${D}/usr/bin/ctags.exe \
		${D}/usr/bin/etags.exe \
		${D}/usr/share/man/man1/ctags* \
		${D}/usr/share/man/man1/etags*
	mv ${D}/usr/bin/{${P}.exe,emacs-basic.exe}
	mv ${D}/usr/bin/{emacsclient.exe,emacsclient-basic.exe}
	if [ ${NATIVE_COMP} = yes ]
	then
	    dodir /usr/lib/emacs/${VERSION}/native-lisp
	fi
	insinto /${EMACS_EXECDIR}
	for t in w32 gtk lucid
	do
	    newbin ${B}/../build-${t}/src/emacs.exe emacs-${t}.exe
	    newbin ${B}/../build-${t}/lib-src/emacsclient.exe emacsclient-${t}.exe
	    fingerprint=$(${B}/../build-${t}/src/emacs --fingerprint)
	    newins ${B}/../build-${t}/src/emacs.pdmp emacs-${fingerprint}.pdmp
	    if [ ${NATIVE_COMP} = yes ]
	    then
		cp -R ${B}/../build-${t}/native-lisp/${VERSION}-* \
		   ${D}/usr/lib/emacs/${VERSION}/native-lisp
	    fi
	done

	cd ${S}
	dobin make-emacs-shortcut
	dobin set-emacs-default.sh
	insinto /usr/bin
	doins nt/icons/emacs.ico
	insinto /usr/share/${PN}/${PV}/etc
	doins src/.gdbinit

	for t in basic w32 gtk lucid
	do
	    dosym set-emacs-default.sh /usr/bin/set-emacs-default-${t}.sh
	done

	dodir /etc/postinstall
	priority=0
	for t in basic w32 gtk lucid
	do
	    priority=$(( priority + 10 ))
	    cat > ${D}/etc/postinstall/emacs-${t}.sh <<EOF
/usr/sbin/alternatives --install /usr/bin/emacs emacs \\
  /usr/bin/emacs-${t}.exe ${priority}
/usr/sbin/alternatives --install /usr/bin/emacsclient emacsclient \\
  /usr/bin/emacsclient-${t}.exe ${priority}
EOF
	    chmod +x ${D}/etc/postinstall/emacs-${t}.sh
	done

	dodir /etc/preremove
	for t in basic w32 gtk lucid
	do
	    cat > ${D}/etc/preremove/emacs-${t}.sh <<EOF
/usr/sbin/update-alternatives --remove emacs /usr/bin/emacs-${t}.exe
/usr/sbin/update-alternatives --remove emacsclient /usr/bin/emacsclient-${t}.exe
EOF
	    chmod +x ${D}/etc/preremove/emacs-${t}.sh
	done

        dodir /var/lib/rebase/dynpath.d
        echo "/usr/local/lib/emacs/${VERSION}/native-lisp" \
	     > ${D}/var/lib/rebase/dynpath.d/emacs
}

src_test() {
    cd ${B}
    make check
}

# SCALLYWAG="deploy notest"
SCALLYWAG="nobuild"
# SCALLYWAG="notest"

bug#52376: 28.0.90; libdir is missing from native-comp-eln-load-path with GTK3 build

[Eli Zaretskii]

> From: Bhavin Gandhi <bhavin7392@gmail.com>
> Date: Sat, 1 Jan 2022 22:42:08 +0530
> Cc: 52376@debbugs.gnu.org, akrl@sdf.org
> 
> When all three variants are built, the make install is run from
> build-gtk directory at the end. And during this install command, it is
> creating a emacs.pdmp again, where build says Loading foo only.
> 
> Can be reproduced with following commands:
> 
> mkdir build-gtk
> cd build-gtk/
> ln -s ../configure .
> ./configure <for GTK>
> make bootstrap
> make
> cd ..
> 
> mkdir build-nox
> cd build-nox/
> ln -s ../configure .
> ./configure <for No-X>
> make bootstrap
> make
> 
> cd ../build-gtk
> sudo make install
> ^^^ This runs configure, make all lib etc again, I can see a new
> emacs.pdmp file being created which loads foo.el from source.
> 
> Is this type of build and install flow supported?

I don't know yet.  The root cause is probably something that causes a
complete rebuild in the build-gtk tree when you say "make install",
although you already built it before.  Can you figure out why it
rebuilds it?  Then we can reason about that cause, whether it's
expected or not.

bug#52376: 28.0.90; libdir is missing from native-comp-eln-load-path with GTK3 build

[Bhavin Gandhi]

On Sat, 1 Jan 2022 at 22:57, Eli Zaretskii <eliz@gnu.org> wrote:
> I don't know yet.  The root cause is probably something that causes a
> complete rebuild in the build-gtk tree when you say "make install",
> although you already built it before.  Can you figure out why it
> rebuilds it?  Then we can reason about that cause, whether it's
> expected or not.

This is what it shows when I run make install, can you please help me
with any pointers to figure out why it rebuilds?

sudo make -O -j8 V=1 VERBOSE=1 install

<< At this point the prompt is actually blank for a while and then
shows the following output.

if [ -x ./config.status ]; then    \
     ./config.status --recheck;    \
else                \
     ../configure --cache-file=/dev/null; \
fi
running CONFIG_SHELL=/bin/sh /bin/sh ./configure
--build=x86_64-redhat-linux-gnu --host=x86_64-redhat-linux-gnu
--program-prefix= --disable-dependency-tracking --prefix=/usr
--exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin
--sysconfdir=/etc --datadir=/usr/share --includedir=/usr/include
--libdir=/usr/lib64 --libexecdir=/usr/libexec --localstatedir=/var
--sharedstatedir=/var/lib --mandir=/usr/share/man
--infodir=/usr/share/info --with-dbus --with-gif --with-jpeg
--with-png --with-rsvg --with-tiff --with-xft --with-xpm
--with-x-toolkit=gtk3 --with-gpm=no --with-xwidgets --with-modules
--with-harfbuzz --with-cairo --with-json --with-native-compilation
build_alias=x86_64-redhat-linux-gnu host_alias=x86_64-redhat-linux-gnu
--no-create --no-recursion
configure: WARNING: unrecognized options: --disable-dependency-tracking
configure: loading site script /usr/share/config.site
checking for xcrun... no
…

Configured for 'x86_64-redhat-linux-gnu'.

  Where should the build process find the source code?    ..
…

I'm not sure if sharing the full command output will be helpful or not,
because after this point I think it is all build logs.

bug#52376: 28.0.90; libdir is missing from native-comp-eln-load-path with GTK3 build

[Eli Zaretskii]

> From: Bhavin Gandhi <bhavin7392@gmail.com>
> Date: Sun, 2 Jan 2022 00:34:19 +0530
> Cc: 52376@debbugs.gnu.org
> 
> On Sat, 1 Jan 2022 at 22:57, Eli Zaretskii <eliz@gnu.org> wrote:
> > I don't know yet.  The root cause is probably something that causes a
> > complete rebuild in the build-gtk tree when you say "make install",
> > although you already built it before.  Can you figure out why it
> > rebuilds it?  Then we can reason about that cause, whether it's
> > expected or not.
> 
> This is what it shows when I run make install, can you please help me
> with any pointers to figure out why it rebuilds?
> 
> sudo make -O -j8 V=1 VERBOSE=1 install
> 
> << At this point the prompt is actually blank for a while and then
> shows the following output.
> 
> if [ -x ./config.status ]; then    \
>      ./config.status --recheck;    \
> else                \
>      ../configure --cache-file=/dev/null; \
> fi
> running CONFIG_SHELL=/bin/sh /bin/sh ./configure

So config.status either doesn't exist or isn't executable?  Why not?

And what is "." in this case -- is it the build directory or the
source directory?

bug#52376: 28.0.90; libdir is missing from native-comp-eln-load-path with GTK3 build

[Bhavin Gandhi]

On Sat, 1 Jan 2022 at 22:56, Ken Brown <kbrown@cornell.edu> wrote:
> In case it's of any use to you, I can show you how I handle this on Cygwin.  We
> use a tool called cygport, which processes a file emacs.cygport (attached).  You
> can think of this as the analogue of an rpm .spec file.  I realize that you're
> not familiar with cygport, but emacs.cygport is a bash script, and I suspect you
> can decipher most of it.  The crucial things to look at are the src_compile and
> src_install functions.

Thank you for sharing the cygport Ken. It is indeed similar to a .spec
file. I was able to understand it with the help of the Cygport Reference
Manual. The build flow being used is similar to our .spec file. The
difference is, the cygport runs the install from the basic build, which was
built last in the src_compile. And in the RPM build, make install is run
from build-gtk which is the first build.

bug#52376: 28.0.90; libdir is missing from native-comp-eln-load-path with GTK3 build

[Bhavin Gandhi]

On Sun, 2 Jan 2022 at 00:49, Eli Zaretskii <eliz@gnu.org> wrote:
>
> > << At this point the prompt is actually blank for a while and then
> > shows the following output.
> >
> > if [ -x ./config.status ]; then    \
> >      ./config.status --recheck;    \
> > else                \
> >      ../configure --cache-file=/dev/null; \
> > fi
> > running CONFIG_SHELL=/bin/sh /bin/sh ./configure
>
> So config.status either doesn't exist or isn't executable?  Why not?

config.status exists and is executable as well.

~/s/e/e/s/e/build-gtk $ ls -ll config.status
ls -ll config.status
-rwxr-xr-x. 1 bhavin bhavin 81764 Jan  2 17:54 config.status

After running the "./config.status --recheck", it is running
"MAKE='/usr/bin/make' ./config.status", which in turn starts the build
again.

This is happening for Emacs 27.2 as well, the full build log is at the
following link, install starts at line number 10298:
https://kojipkgs.fedoraproject.org//packages/emacs/27.2/9.fc35/data/logs/x86_64/build.log

> And what is "." in this case -- is it the build directory or the
> source directory?

"." in this case is build directory build-gtk, which is inside the
source directory "emacs-28.0.90".

I was trying to figure out which files change in the source directory
after "make bootstrap" is run from build-nox. I'm not sure if that's a
correct thing to look at. I even tried to look at Makefile.in, but I'm not
sure where to look at and what to look at.

cd build-gtk/
./configure <for GTK>
make bootstrap
make
cd ..

~/s/e/e/s/emacs-28.0.90 $ ls -ll configure
-rwxr-xr-x. 1 bhavin bhavin 1007725 Jan  2 17:54 configure

cd build-nox/
./configure <for No-X>
make bootstrap
make

~/s/e/e/s/emacs-28.0.90 $ ls -ll configure
-rwxr-xr-x. 1 bhavin bhavin 1007725 Jan  2 18:14 configure

The timestamps on the configure, autom4te.cache directory, etc/refcards,
src/config.in, info (.info files), lisp (.elc files), doc directory are
updated at this point. The content is the same as it was after running
"make" from build-gtk directory (I'm tracking the content with Git).

bug#52376: 28.0.90; libdir is missing from native-comp-eln-load-path with GTK3 build

[Eli Zaretskii]

> From: Bhavin Gandhi <bhavin7392@gmail.com>
> Date: Sun, 2 Jan 2022 23:09:17 +0530
> Cc: 52376@debbugs.gnu.org
> 
> After running the "./config.status --recheck", it is running
> "MAKE='/usr/bin/make' ./config.status", which in turn starts the build
> again.
> 
> This is happening for Emacs 27.2 as well, the full build log is at the
> following link, install starts at line number 10298:
> https://kojipkgs.fedoraproject.org//packages/emacs/27.2/9.fc35/data/logs/x86_64/build.log
> 
> > And what is "." in this case -- is it the build directory or the
> > source directory?
> 
> "." in this case is build directory build-gtk, which is inside the
> source directory "emacs-28.0.90".
> 
> I was trying to figure out which files change in the source directory
> after "make bootstrap" is run from build-nox. I'm not sure if that's a
> correct thing to look at. I even tried to look at Makefile.in, but I'm not
> sure where to look at and what to look at.
> 
> cd build-gtk/
> ./configure <for GTK>
> make bootstrap
> make
> cd ..
> 
> ~/s/e/e/s/emacs-28.0.90 $ ls -ll configure
> -rwxr-xr-x. 1 bhavin bhavin 1007725 Jan  2 17:54 configure
> 
> cd build-nox/
> ./configure <for No-X>
> make bootstrap
> make
> 
> ~/s/e/e/s/emacs-28.0.90 $ ls -ll configure
> -rwxr-xr-x. 1 bhavin bhavin 1007725 Jan  2 18:14 configure
> 
> The timestamps on the configure, autom4te.cache directory, etc/refcards,
> src/config.in, info (.info files), lisp (.elc files), doc directory are
> updated at this point. The content is the same as it was after running
> "make" from build-gtk directory (I'm tracking the content with Git).

I guess this is because (a) you run "make bootstrap" each time, and
(b) "make bootstrap" changes some files in the source tree.

My suggestion at this point would be to use just

   ./configure <for whatever>
   make
   make install

There should be no need for you to bootstrap when you are building a
release tarball.  Bootstrap is for building from the Git repository.

bug#52376: 28.0.90; libdir is missing from native-comp-eln-load-path with GTK3 build

[Bhavin Gandhi]

On Sun, 2 Jan 2022 at 23:52, Eli Zaretskii <eliz@gnu.org> wrote:
> >
> > The timestamps on the configure, autom4te.cache directory, etc/refcards,
> > src/config.in, info (.info files), lisp (.elc files), doc directory are
> > updated at this point. The content is the same as it was after running
> > "make" from build-gtk directory (I'm tracking the content with Git).
>
> I guess this is because (a) you run "make bootstrap" each time, and
> (b) "make bootstrap" changes some files in the source tree.
>
> My suggestion at this point would be to use just
>
>    ./configure <for whatever>
>    make
>    make install

You are right. It is working as expected after removing the bootstrap
step from all the builds.

Thank you for your help and being patient with me :)

> There should be no need for you to bootstrap when you are building a
> release tarball.  Bootstrap is for building from the Git repository.

Does NATIVE_FULL_AOT=1 require "make bootstrap NATIVE_FULL_AOT=1"?
Because the following sequence is not generating .eln for all the Lisp
files. It is just generating a few (156) .eln files:

$ ./configure <flags>
$ make NATIVE_FULL_AOT=1
$ ls -1 -R native-lisp/28.0.90-8508728b/ | wc -l
156

I think this bug report can be closed as the problem I reported as part
of this report has been solved.

bug#52376: 28.0.90; libdir is missing from native-comp-eln-load-path with GTK3 build

[Eli Zaretskii]

> From: Bhavin Gandhi <bhavin7392@gmail.com>
> Date: Tue, 4 Jan 2022 23:47:56 +0530
> Cc: 52376@debbugs.gnu.org
> 
> > There should be no need for you to bootstrap when you are building a
> > release tarball.  Bootstrap is for building from the Git repository.
> 
> Does NATIVE_FULL_AOT=1 require "make bootstrap NATIVE_FULL_AOT=1"?
> Because the following sequence is not generating .eln for all the Lisp
> files. It is just generating a few (156) .eln files:
> 
> $ ./configure <flags>
> $ make NATIVE_FULL_AOT=1
> $ ls -1 -R native-lisp/28.0.90-8508728b/ | wc -l
> 156

NATIVE_FULL_AOT=1 is not yet fully supported in Emacs 28.  I think to
get what you want you need to delete all the *.elc files before you
say "make NATIVE_FULL_AOT=1".

> I think this bug report can be closed as the problem I reported as part
> of this report has been solved.

OK, done.