bug#63742: 30.0.50; [BUG] package.el having a hard time

bug#63742: 30.0.50; [BUG] package.el having a hard time

[Ruijie Yu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Hello,

I have recompiled Emacs master yesterday, but this build has had a hard
time processing my init.el file (saying that PKG-pkg.el files are not
found, where they indeed are).  I wasn't able to reproduce this exact
problem, but upon digging further, I found one issue which I can
reproduce it reliably in -Q environment, and which is probably related
to the other unreproducible issue.

    $ cd $(mktemp -d)
    $ HOME=$PWD XDG_CONFIG_HOME=$HOME/.config emacs -Q
    M-x package-initialize RET
    M-x list-packages RET
    ;; And wait for it to finish listing packages from gnu and nongnu

    / n xr RET
    ;; find some package to install, any package suffices
    i x SPC
    ;; Install it

Observe the error message shown in the minibuffer.  The following is the
full message from trying to install the package.

--8<---------------cut here---------------start------------->8---
Packages to install: 1 (xr-1.23).  Proceed? (y or n) y
Operation [ Install 1 ] started
Setting ‘package-selected-packages’ temporarily since "emacs -q" would overwrite customizations
Contacting host: elpa.gnu.org:443 [2 times]
Parsing tar file...done
Extracting...done
  INFO     Scraping files for loaddefs...done
  GEN      xr-autoloads.el
package-unpack: Wrong type argument: package-desc, nil
--8<---------------cut here---------------end--------------->8---

In GNU Emacs 30.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version
 3.24.38, cairo version 1.17.8) of 2023-05-26 built on ruijie
Repository revision: bb7605c087006b714236165f88341545355d3673
Repository branch: makepkg
System Description: Arch Linux

Configured using:
 'configure --prefix=/usr --sysconfdir=/etc --libexecdir=/usr/lib
 --localstatedir=/var --mandir=/usr/share/man --with-gameuser=:games
 --with-modules --without-libotf --without-m17n-flt --without-gconf
 --enable-link-time-optimization --with-native-compilation=aot
 --with-xinput2 --with-pgtk --without-xaw3d --with-sound=alsa
 --with-tree-sitter '--program-transform-name=s/\([ec]tags\)/\1.emacs/'
 'CFLAGS=-march=x86-64 -mtune=generic -O2 -pipe -fno-plt -fexceptions
 -Wp,-D_FORTIFY_SOURCE=2 -Wformat -Werror=format-security
 -fstack-clash-protection -fcf-protection'
 LDFLAGS=-Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now'

Configured features:
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GPM GSETTINGS HARFBUZZ JPEG
JSON LCMS2 LIBSYSTEMD LIBXML2 MODULES NATIVE_COMP NOTIFY INOTIFY PDUMPER
PGTK PNG RSVG SECCOMP SOUND SQLITE3 THREADS TIFF TOOLKIT_SCROLL_BARS
TREE_SITTER WEBP XIM GTK3 ZLIB

Important settings:
  value of $LANG: en_US.UTF-8
  value of $XMODIFIERS: @im=fcitx
  locale-coding-system: utf-8-unix

-- 
Best,


RY

bug#63742: 30.0.50; [BUG] package.el having a hard time

[Eshel Yaron via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Ruijie Yu via "Bug reports for GNU Emacs, the Swiss army knife of text
editors" <bug-gnu-emacs@gnu.org> writes:

> package-unpack: Wrong type argument: package-desc, nil

I've had this issue as well since rebuilding Emacs yesterday from the
master branch.  AFAICT the issue is that `package-load-descriptor`
returns `nil` instead of a valid package descriptor when called in
`package-unpack`.

Curiously, when I evaluate the definition of `package-load-descriptor`
(with C-M-x) it starts behaving correctly and the issue goes away.
Perhaps this is a native compilation problem?

bug#63742: 30.0.50; [BUG] package.el having a hard time

[Ruijie Yu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eshel Yaron <me@eshelyaron.com> writes:

> Ruijie Yu via "Bug reports for GNU Emacs, the Swiss army knife of text
> editors" <bug-gnu-emacs@gnu.org> writes:
>
>> package-unpack: Wrong type argument: package-desc, nil
>
> I've had this issue as well since rebuilding Emacs yesterday from the
> master branch.  AFAICT the issue is that `package-load-descriptor`
> returns `nil` instead of a valid package descriptor when called in
> `package-unpack`.
>
> Curiously, when I evaluate the definition of `package-load-descriptor`
> (with C-M-x) it starts behaving correctly and the issue goes away.
> Perhaps this is a native compilation problem?

Probably.  Interestingly, my other issue (which waws unreproducible and
hence not described in my other message) apparently went away when I
byte-compiled the problematic file.

-- 
Best,


RY

bug#63742: 30.0.50; [BUG] package.el having a hard time

[Eli Zaretskii]

merge 63742 63679 63716 63710
thanks

> Date: Sat, 27 May 2023 11:10:24 +0800
> From:  Ruijie Yu via "Bug reports for GNU Emacs,
>  the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>
> 
> I have recompiled Emacs master yesterday, but this build has had a hard
> time processing my init.el file (saying that PKG-pkg.el files are not
> found, where they indeed are).  I wasn't able to reproduce this exact
> problem, but upon digging further, I found one issue which I can
> reproduce it reliably in -Q environment, and which is probably related
> to the other unreproducible issue.
> 
>     $ cd $(mktemp -d)
>     $ HOME=$PWD XDG_CONFIG_HOME=$HOME/.config emacs -Q
>     M-x package-initialize RET
>     M-x list-packages RET
>     ;; And wait for it to finish listing packages from gnu and nongnu
> 
>     / n xr RET
>     ;; find some package to install, any package suffices
>     i x SPC
>     ;; Install it
> 
> Observe the error message shown in the minibuffer.  The following is the
> full message from trying to install the package.
> 
> --8<---------------cut here---------------start------------->8---
> Packages to install: 1 (xr-1.23).  Proceed? (y or n) y
> Operation [ Install 1 ] started
> Setting ‘package-selected-packages’ temporarily since "emacs -q" would overwrite customizations
> Contacting host: elpa.gnu.org:443 [2 times]
> Parsing tar file...done
> Extracting...done
>   INFO     Scraping files for loaddefs...done
>   GEN      xr-autoloads.el
> package-unpack: Wrong type argument: package-desc, nil
> --8<---------------cut here---------------end--------------->8---

Does the problem go away if you rebuild without native-compilation?
If yes, this is a known problem (bug#63679), and Andrea (CC'ed) is
aware of that, I think.

bug#63742: 30.0.50; [BUG] package.el having a hard time

[Ruijie Yu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

> merge 63742 63679 63716 63710
> thanks
>
>>     $ cd $(mktemp -d)
>>     $ HOME=$PWD XDG_CONFIG_HOME=$HOME/.config emacs -Q
>>     M-x package-initialize RET
>>     M-x list-packages RET
>>     ;; And wait for it to finish listing packages from gnu and nongnu
>> 
>>     / n xr RET
>>     ;; find some package to install, any package suffices
>>     i x SPC
>>     ;; Install it
>> 
>> Observe the error message shown in the minibuffer.  The following is the
>> full message from trying to install the package.
>> 
> Does the problem go away if you rebuild without native-compilation?
> If yes, this is a known problem (bug#63679), and Andrea (CC'ed) is
> aware of that, I think.

Indeed, this problem goes away if I rebuild without native-compilation.
To minimize changes, I used the same commit (bb7605c), and used the same
config options except for native comp.

--8<---------------cut here---------------start------------->8---
 'configure --prefix=/usr --sysconfdir=/etc --libexecdir=/usr/lib
 --localstatedir=/var --mandir=/usr/share/man --with-gameuser=:games
 --with-modules --without-libotf --without-m17n-flt --without-gconf
 --enable-link-time-optimization --with-native-compilation=no
 --with-xinput2 --with-pgtk --without-xaw3d --with-sound=alsa
 --with-tree-sitter '--program-transform-name=s/\([ec]tags\)/\1.emacs/'
 'CFLAGS=-march=x86-64 -mtune=generic -O2 -pipe -fno-plt -fexceptions
 -Wp,-D_FORTIFY_SOURCE=2 -Wformat -Werror=format-security
 -fstack-clash-protection -fcf-protection'
 LDFLAGS=-Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now'
--8<---------------cut here---------------end--------------->8---

By the way, do we have a bisect for this, where we know what revision(s)
introduced this issue?  If not, (and if I have time during the weekend,)
I can try to bisect it.  And in the meantime, I'll sit on all my
known-good builds until this issue is fixed.

Some data points:

- known good version (the build I'm using to send this message):
  "fa598571adab4858282f337b45984517e197f8a9"

- known bad version (the build where I observed the reported issues):
  "bb7605c087006b714236165f88341545355d3673"

-- 
Best,


RY

bug#63742: 30.0.50; [BUG] package.el having a hard time

[Ruijie Yu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

[-- Attachment #1: Type: text/plain, Size: 2874 bytes --]


>> Does the problem go away if you rebuild without native-compilation?
>> If yes, this is a known problem (bug#63679), and Andrea (CC'ed) is
>> aware of that, I think.
>
> Indeed, this problem goes away if I rebuild without native-compilation.
> To minimize changes, I used the same commit (bb7605c), and used the same
> config options except for native comp.
>
>  'configure --prefix=/usr --sysconfdir=/etc --libexecdir=/usr/lib
>  --localstatedir=/var --mandir=/usr/share/man --with-gameuser=:games
>  --with-modules --without-libotf --without-m17n-flt --without-gconf
>  --enable-link-time-optimization --with-native-compilation=no
>  --with-xinput2 --with-pgtk --without-xaw3d --with-sound=alsa
>  --with-tree-sitter '--program-transform-name=s/\([ec]tags\)/\1.emacs/'
>  'CFLAGS=-march=x86-64 -mtune=generic -O2 -pipe -fno-plt -fexceptions
>  -Wp,-D_FORTIFY_SOURCE=2 -Wformat -Werror=format-security
>  -fstack-clash-protection -fcf-protection'
>  LDFLAGS=-Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now'
>
> By the way, do we have a bisect for this, where we know what revision(s)
> introduced this issue?  If not, (and if I have time during the weekend,)
> I can try to bisect it.  And in the meantime, I'll sit on all my
> known-good builds until this issue is fixed.
>
> Some data points:
>
> - known good version (the build I'm using to send this message):
>   "fa598571adab4858282f337b45984517e197f8a9"
>
> - known bad version (the build where I observed the reported issues):
>   "bb7605c087006b714236165f88341545355d3673"

Here is the bisect result.

--8<---------------cut here---------------start------------->8---
b2b2be98da5825e6bc130999ffe38f5ed615586a is the first bad commit
commit b2b2be98da5825e6bc130999ffe38f5ed615586a
Author: Andrea Corallo <akrl@sdf.org>
Date:   Wed May 24 17:05:26 2023 +0200

    Fix nativecomp ICE compiling structure related code (bug#63674)

    * lisp/emacs-lisp/cl-macs.el (cl-defstruct): Define the predicate to
    be effective at compile time as native comp relies on `cl-typep' that
    relies o predicates to work.

    * test/src/comp-resources/comp-test-funcs.el (cl-lib): Require.
    (comp-test-struct): Define struct.
    (comp-test-63674-1-f): Add new test.

 lisp/emacs-lisp/cl-macs.el                 | 9 ++++++---
 test/src/comp-resources/comp-test-funcs.el | 8 ++++++++
 2 files changed, 14 insertions(+), 3 deletions(-)
bisect found first bad commit
--8<---------------cut here---------------end--------------->8---

What I did:

    $ git bisect start bb7605c087006b714236165f88341545355d3673 fa598571adab4858282f337b45984517e197f8a9
    $ git bisect run sh repro.sh

With the attached files as reproducer.  Hope this helps.

Oh, by the way, while inspecting my reproducer scripts I noticed that I
forgot to add -Q -- I really hope this does not bite me since
git-bisect-run took a few hours to finish.


[-- Attachment #2: make.sh --]
[-- Type: application/x-sh, Size: 682 bytes --]

[-- Attachment #3: repro.el --]
[-- Type: text/plain, Size: 75 bytes --]

;; -*- lexical-binding: t; -*-

(package-initialize)
(package-install 'xr)

[-- Attachment #4: repro.sh --]
[-- Type: application/x-sh, Size: 31 bytes --]

[-- Attachment #5: run.sh --]
[-- Type: application/x-sh, Size: 138 bytes --]

[-- Attachment #6: Type: text/plain, Size: 16 bytes --]


-- 
Best,


RY