bug#59507: 29.0.50; Autoloads generation hangs on Windows and Emacs 29

bug#59507: 29.0.50; Autoloads generation hangs on Windows and Emacs 29

[Denys Mentiei via Bug reports for GNU Emacs, the Swiss army knife of text editors]

[-- Attachment #1: Type: text/plain, Size: 3780 bytes --]

Hello!

It looks like "loaddefs-generate--file-load-name" hangs when it receives
a path starting with "c:/".

Looking at the code, it seems like the very first loop, which
eats the path piece by piece with (file-name-directory name) never
terminates as (file-name-directory-name "c:/") returns "c:/".

In GNU Emacs 29.0.50 (build 1, x86_64-w64-mingw32)
Repository revision: c618a2258cb85e4297304156ccb4c8dbb4fc984a
Repository branch: master
Windowing system distributor 'Microsoft Corp.', version 10.0.22621
System Description: Microsoft Windows 10 Enterprise (v10.0.2009.22621.819)

Configured using:
'configure --prefix=/d/a/emacs-build/emacs-build/pkg/master-x86_64
'CFLAGS=-Ofast -fno-finite-math-only -fomit-frame-pointer'
--with-native-compilation --with-zlib --with-xpm --with-xml2
--with-tree-sitter --without-tiff --with-rsvg --with-png --with-lcms2
--with-json --with-jpeg --with-harfbuzz --with-gnutls --with-gif
--without-cairo --disable-build-details --without-dbus
--with-compress-install --with-small-ja-dic'

Configured features:
ACL GIF GMP GNUTLS HARFBUZZ JPEG JSON LCMS2 LIBXML2 MODULES NATIVE_COMP
NOTIFY W32NOTIFY PDUMPER PNG RSVG SOUND SQLITE3 THREADS
TOOLKIT_SCROLL_BARS TREE_SITTER WEBP XPM ZLIB

Important settings:
  value of $LANG: ENU
  locale-coding-system: cp1252

Major mode: Fundamental

Minor modes in effect:
  global-auto-revert-mode: t
  global-hl-line-mode: t
  global-eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  column-number-mode: t
  line-number-mode: t
  indent-tabs-mode: t
  transient-mark-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message mailcap yank-media puny dired
dired-loaddefs rfc822 mml mml-sec password-cache epa derived epg rfc6068
epg-config gnus-util text-property-search time-date mm-decode mm-bodies
mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader sendmail
rfc2047 rfc2045 ietf-drums mm-util mail-prsvr mail-utils time cl-extra
help-mode straight info autoload loaddefs-gen generate-lisp-file
radix-tree lisp-mnt easy-mmode cl-seq pcase subr-x byte-opt cl-macs gv
bytecomp byte-compile advice autorevert filenotify cl-loaddefs cl-lib
hl-line rmc iso-transl tooltip cconv eldoc paren electric uniquify
ediff-hook vc-hooks lisp-float-type elisp-mode mwheel dos-w32 ls-lisp
disp-table term/w32-win w32-win w32-vars term/common-win tool-bar dnd
fontset image regexp-opt fringe tabulated-list replace newcomment
text-mode lisp-mode prog-mode register page tab-bar menu-bar rfn-eshadow
isearch easymenu timer select scroll-bar mouse jit-lock font-lock syntax
font-core term/tty-colors frame minibuffer nadvice seq simple cl-generic
indonesian philippine cham georgian utf-8-lang misc-lang vietnamese
tibetan thai tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek
romanian slovak czech european ethiopic indian cyrillic chinese
composite emoji-zwj charscript charprop case-table epa-hook
jka-cmpr-hook help abbrev obarray oclosure cl-preloaded button loaddefs
theme-loaddefs faces cus-face macroexp files window text-properties
overlay sha1 md5 base64 format env code-pages mule custom widget keymap
hashtable-print-readable backquote threads w32notify w32 lcms2 multi-tty
make-network-process native-compile emacs)

Memory information:
((conses 16 69536 67736)
(symbols 48 8376 0)
(strings 32 24966 24386)
(string-bytes 1 827292)
(vectors 16 15567)
(vector-slots 8 303325 101908)
(floats 8 35 128)
(intervals 56 213 53)
(buffers 992 12))

[-- Attachment #2: Type: text/html, Size: 7466 bytes --]

bug#59507: 29.0.50; Autoloads generation hangs on Windows and Emacs 29

[Eli Zaretskii]

> Date: Tue, 22 Nov 2022 20:13:47 +0000
> From:  Denys Mentiei via "Bug reports for GNU Emacs,
>  the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>
> 
> It looks like "loaddefs-generate--file-load-name" hangs when it receives
> a path starting with "c:/".
>  
> Looking at the code, it seems like the very first loop, which
> eats the path piece by piece with (file-name-directory name) never
> terminates as (file-name-directory-name "c:/") returns "c:/".

Thanks.

I may be missing something, but I fail to see the difference between a
Windows file name starting with "C:/" and a Unix file name starting with
"/", as far as that function is concerned.  Either they both work or they
both fail in the same way, because (file-name-directory "/") returns "/".

So do you have a real-life recipe where loaddefs-generate--file-load-name is
called and infloops?  Can you show such a recipe, preferably in "emacs -Q"?

Alternatively, a recipe which just invokes loaddefs-generate--file-load-name
with file names which cause the loop will also suffice, but in that case
please also describe how did those file names ended up being passed to the
function, i.e. with which file/directory names did the call sequence start.

The reason I'm asking is that I'm not yet sure whether the problem, whatever
it is, is in the function you pointed to or in some other place.

bug#59507: 29.0.50; Autoloads generation hangs on Windows and Emacs 29

[Denys Mentiei via Bug reports for GNU Emacs, the Swiss army knife of text editors]

[-- Attachment #1: Type: text/plain, Size: 3256 bytes --]

> I may be missing something, but I fail to see the difference between a
> Windows file name starting with "C:/" and a Unix file name starting with
> "/", as far as that function is concerned.  Either they both work or they
> both fail in the same way, because (file-name-directory "/") returns "/".

Indeed! That is just the use case I describe further was caught on Windows.

> So do you have a real-life recipe where loaddefs-generate--file-load-name is
> called and infloops?  Can you show such a recipe, preferably in "emacs -Q"?

So, this happened for me when installing a package via Straight.el.
Under the hood it invokes autoloads generation, which can be seen in the
following backtrace:

  loaddefs-generate--file-load-name("c:/Users/d/AppData/Roaming/.emacs.d/straight/build/bind-key/bind-key.el" "d:/dotfiles/.emacs.d/straight/build/bind-key/bind-key-autoloads.el")
  autoload-file-load-name("c:/Users/d/AppData/Roaming/.emacs.d/straight/build/bind-key/bind-key.el" "d:/dotfiles/.emacs.d/straight/build/bind-key/bind-key-autoloads.el")
  autoload-generate-file-autoloads("c:/Users/d/AppData/Roaming/.emacs.d/straig..." nil "d:/dotfiles/.emacs.d/straight/build/bind-key/bind-...")
  make-directory-autoloads("c:/Users/d/AppData/Roaming/.emacs.d/straight/build/bind-key/" "c:/Users/d/AppData/Roaming/.emacs.d/straight/build/bind-key/bind-key-autoloads.el")
  straight--build-autoloads((:flavor melpa :files ("bind-key.el" "bind-key-pkg.el") :package "bind-key" :local-repo "use-package" :type git :repo "jwiegley/use-package" :host github))
  run-hook-with-args(straight--build-autoloads (:flavor melpa :files ("bind-key.el" "bind-key-pkg.el") :package "bind-key" :local-repo "use-package" :type git :repo "jwiegley/use-package" :host github))
  straight--build-package((:flavor melpa :files ("bind-key.el" "bind-key-pkg.el") :package "bind-key" :local-repo "use-package" :type git :repo "jwiegley/use-package" :host github) "Building use-package")
  #f(compiled-function () #<bytecode -0x1fce4a710bdd24dd>)()
  straight--transaction-exec(use-package-\"81e6ef700f88dfe7131843f7ca08439c\"-nil-nil :now #f(compiled-function () #<bytecode -0x1fce4a710bdd24dd>))
  straight-use-package(bind-key nil nil "Building use-package")

And then it hangs in the loop mentioned in my original message.

A recipe for “emacs -Q” might be this (not smallest thing so, but a real life one):

Paste this to the *scratch*:

(defvar bootstrap-version)
(let ((bootstrap-file
       (expand-file-name "straight/repos/straight.el/bootstrap.el" user-emacs-directory))
      (bootstrap-version 5))
  (unless (file-exists-p bootstrap-file)
    (with-current-buffer
        (url-retrieve-synchronously
        "https://raw.githubusercontent.com/raxod502/straight.el/develop/install.el"
        'silent 'inhibit-cookies)
      (goto-char (point-max))
      (eval-print-last-sexp))
    (load bootstrap-file nil 'nomessage)))
(setq straight-vc-git-default-clone-depth '(1 single-branch))

aand M-x eval-buffer :) Warning: it will download the straight.el from its GitHub official repo.
Note, that I do this on Windows, so user-emacs-directory isn’t a "~/.emacs.d", but an absolute path,
starting from "c:/".

Hope this helps!

[-- Attachment #2: Type: text/html, Size: 6573 bytes --]

bug#59507: 29.0.50; Autoloads generation hangs on Windows and Emacs 29

[Eli Zaretskii]

> From: Denys Mentiei <Denys.Mentiei@tobii.com>
> CC: "59507@debbugs.gnu.org" <59507@debbugs.gnu.org>
> Date: Wed, 23 Nov 2022 19:36:12 +0000
> 
> > I may be missing something, but I fail to see the difference between a
> > Windows file name starting with "C:/" and a Unix file name starting with
> > "/", as far as that function is concerned.  Either they both work or they
> > both fail in the same way, because (file-name-directory "/") returns "/".
> 
> Indeed! That is just the use case I describe further was caught on Windows.
> 
> > So do you have a real-life recipe where loaddefs-generate--file-load-name is
> > called and infloops?  Can you show such a recipe, preferably in "emacs -Q"?
> 
> So, this happened for me when installing a package via Straight.el.
> Under the hood it invokes autoloads generation, which can be seen in the
> following backtrace:
>  
>   loaddefs-generate--file-load-name
> ("c:/Users/d/AppData/Roaming/.emacs.d/straight/build/bind-key/bind-key.el"
> "d:/dotfiles/.emacs.d/straight/build/bind-key/bind-key-autoloads.el")

Thanks, I think I see the problem now.  Please try the patch below, which I
just installed on the master branch of the Emacs Git repository:

diff --git a/lisp/emacs-lisp/loaddefs-gen.el b/lisp/emacs-lisp/loaddefs-gen.el
index ecc5f7e..2dd0417 100644
--- a/lisp/emacs-lisp/loaddefs-gen.el
+++ b/lisp/emacs-lisp/loaddefs-gen.el
@@ -108,21 +108,26 @@ loaddefs-generate--file-load-name
   (let* ((name (file-relative-name file (file-name-directory outfile)))
          (names '())
          (dir (file-name-directory outfile)))
-    ;; If `name' has directory components, only keep the
-    ;; last few that are really needed.
-    (while name
-      (setq name (directory-file-name name))
-      (push (file-name-nondirectory name) names)
-      (setq name (file-name-directory name)))
-    (while (not name)
-      (cond
-       ((null (cdr names)) (setq name (car names)))
-       ((file-exists-p (expand-file-name "subdirs.el" dir))
-        ;; FIXME: here we only check the existence of subdirs.el,
-        ;; without checking its content.  This makes it generate wrong load
-        ;; names for cases like lisp/term which is not added to load-path.
-        (setq dir (expand-file-name (pop names) dir)))
-       (t (setq name (mapconcat #'identity names "/")))))
+    ;; If `name' lives inside an ancestor directory of OUTFILE, only
+    ;; keep the last few leading directories that are really needed.
+    ;; (It will always live in an ancestor directory of OUTFILE on
+    ;; Posix systems, but on DOS/Windows it could not be, if FILE and
+    ;; OUTFILE are on different drives.)
+    (when (not (file-name-absolute-p name))
+      (while name
+        (setq name (directory-file-name name))
+        (push (file-name-nondirectory name) names)
+        (setq name (file-name-directory name)))
+      (while (not name)
+        (cond
+         ((null (cdr names)) (setq name (car names)))
+         ((file-exists-p (expand-file-name "subdirs.el" dir))
+          ;; FIXME: here we only check the existence of subdirs.el,
+          ;; without checking its content.  This makes it generate
+          ;; wrong load names for cases like lisp/term which is not
+          ;; added to load-path.
+          (setq dir (expand-file-name (pop names) dir)))
+         (t (setq name (mapconcat #'identity names "/"))))))
     (if (string-match "\\.elc?\\(\\.\\|\\'\\)" name)
         (substring name 0 (match-beginning 0))
       name)))

bug#59507: 29.0.50; Autoloads generation hangs on Windows and Emacs 29

[Denys Mentiei via Bug reports for GNU Emacs, the Swiss army knife of text editors]

[-- Attachment #1: Type: text/plain, Size: 244 bytes --]

> Thanks, I think I see the problem now.  Please try the patch below, which I
> just installed on the master branch of the Emacs Git repository:

I’ve checked the patch on the same environment and it indeed fixes the problem!

Thank you.

[-- Attachment #2: Type: text/html, Size: 1441 bytes --]

bug#59507: 29.0.50; Autoloads generation hangs on Windows and Emacs 29

[Eli Zaretskii]

> From: Denys Mentiei <Denys.Mentiei@tobii.com>
> CC: "59507@debbugs.gnu.org" <59507@debbugs.gnu.org>
> Date: Tue, 29 Nov 2022 08:58:02 +0000
> 
> > Thanks, I think I see the problem now.  Please try the patch below, which I
> > just installed on the master branch of the Emacs Git repository:
> 
> I’ve checked the patch on the same environment and it indeed fixes the problem!

Thanks, I'm therefore closing the bug.