bug#60628: 29.0.60; apropos-documentation does not work for preloaded symbols

bug#60628: 29.0.60; apropos-documentation does not work for preloaded symbols

[Eli Zaretskii]

To reproduce:

  emacs -Q
  C-h d list.*directory RET

Note that the resulting *Apropos* buffer only lists built-in functions
and variables, i.e. those defined in C.  In particular, missing are
functions/variables from files.el, such as find-directory-functions,
parse-colon-path, and directory-files-recursively, whose doc strings
all match the regexp.

This seems to be due to

    commit 59732a83c8875c8986d2221600d559a24d8309cc
    Author: Stefan Monnier <monnier@iro.umontreal.ca>
    Date:   Thu Dec 30 23:17:45 2021 -0500

        Don't store docstrings of preloaded .el files in etc/DOC

The doc string of apropos-documentation does say:

  Note that by default this command only searches in the file specified by
  ‘internal-doc-file-name’; i.e., the etc/DOC file.  With C-u prefix,
  or if ‘apropos-do-all’ is non-nil, it searches all currently defined
  documentation strings.

However, since we no longer write the doc strings of preloaded Lisp
files to DOC, that default no longer makes sense: Emacs should not
distinguish between built-in primitives and preloaded stuff, and it
didn't up to and including Emacs 28.

Moreover, "C-u C-h d" also doesn't work for preloaded functions,
because:

  (apropos-documentation-internal 'directory-files-recursively)
    => nil

This happens because:

  (apropos-safe-documentation 'directory-files-recursively)
    => ("files.elc" . 39940)

and in that case apropos-documentation-internal calls
apropos-documentation-check-elc-file, which does something very
strange: it only looks inside strings that begin with "\n#@", and
otherwise simply ignores everything else.  Is this also a remnant from
back when preloaded files had their doc strings in etc/DOC?  Or else
why does it only look for "#@" comments preceded by a newline?

Basically, apropos-documentation is currently completely broken on the
release branch.  The following simple change seems to at least repair
"C-u C-h d", but I don't think it's sufficient, as we need to
resurrect the ability to find preloaded symbols without C-u.

diff --git a/lisp/apropos.el b/lisp/apropos.el
index b260d88..cff6f1c 100644
--- a/lisp/apropos.el
+++ b/lisp/apropos.el
@@ -1064,7 +1064,9 @@ apropos-documentation-check-elc-file
       (setq apropos-files-scanned (cons file apropos-files-scanned))
       (erase-buffer)
       (insert-file-contents file)
-      (while (search-forward "\n#@" nil t)
+      (while (search-forward "#@" nil t)
+        (or (not (looking-at-p "[0-9]+"))
+            (progn
 	;; Read the comment length, and advance over it.
 	(setq end (read)
 	      beg (1+ (point))
@@ -1110,7 +1112,7 @@ apropos-documentation-check-elc-file
 					       'face apropos-match-face doc)))
 		      (setcar (nthcdr (if this-is-a-variable 3 2)
 				      apropos-item)
-			      doc))))))))))
+			      doc))))))))))))
 

Ideas? patches? comments?



In GNU Emacs 29.0.60 (build 197, i686-pc-mingw32) of 2023-01-07 built on
 HOME-C4E4A596F7
Repository revision: 9d410f8de64e91d16999a9bb5dd884d6d06d22bd
Repository branch: emacs-29
Windowing system distributor 'Microsoft Corp.', version 5.1.2600
System Description: Microsoft Windows XP Service Pack 3 (v5.1.0.2600)

Configured using:
 'configure -C --prefix=/d/usr --with-wide-int
 --enable-checking=yes,glyphs 'CFLAGS=-O0 -gdwarf-4 -g3''

Configured features:
ACL GIF GMP GNUTLS HARFBUZZ JPEG JSON LCMS2 LIBXML2 MODULES NOTIFY
W32NOTIFY PDUMPER PNG RSVG SOUND SQLITE3 THREADS TIFF
TOOLKIT_SCROLL_BARS TREE_SITTER WEBP XPM ZLIB

Important settings:
  value of $LANG: ENU
  locale-coding-system: cp1255

Major mode: Lisp Interaction

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  line-number-mode: t
  indent-tabs-mode: t
  transient-mark-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr dabbrev emacsbug message mailcap yank-media puny
dired dired-loaddefs rfc822 mml mml-sec password-cache epa derived epg
rfc6068 epg-config gnus-util text-property-search time-date subr-x
mm-decode mm-bodies mm-encode mail-parse rfc2231 mailabbrev gmm-utils
mailheader sendmail rfc2047 rfc2045 ietf-drums mm-util mail-prsvr
mail-utils cl-extra pp cl-print byte-opt gv bytecomp byte-compile
help-fns radix-tree mule-util help-mode cl-loaddefs cl-lib apropos rmc
iso-transl tooltip cconv eldoc paren electric uniquify ediff-hook
vc-hooks lisp-float-type elisp-mode mwheel dos-w32 ls-lisp disp-table
term/w32-win w32-win w32-vars term/common-win tool-bar dnd fontset image
regexp-opt fringe tabulated-list replace newcomment text-mode lisp-mode
prog-mode register page tab-bar menu-bar rfn-eshadow isearch easymenu
timer select scroll-bar mouse jit-lock font-lock syntax font-core
term/tty-colors frame minibuffer nadvice seq simple cl-generic
indonesian philippine cham georgian utf-8-lang misc-lang vietnamese
tibetan thai tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek
romanian slovak czech european ethiopic indian cyrillic chinese
composite emoji-zwj charscript charprop case-table epa-hook
jka-cmpr-hook help abbrev obarray oclosure cl-preloaded button loaddefs
theme-loaddefs faces cus-face macroexp files window text-properties
overlay sha1 md5 base64 format env code-pages mule custom widget keymap
hashtable-print-readable backquote threads w32notify w32 lcms2 multi-tty
make-network-process emacs)

Memory information:
((conses 16 54328 11479)
 (symbols 48 7273 0)
 (strings 16 20447 2422)
 (string-bytes 1 523245)
 (vectors 16 11853)
 (vector-slots 8 170903 17702)
 (floats 8 34 42)
 (intervals 40 449 141)
 (buffers 888 13))

bug#60628: 29.0.60; apropos-documentation does not work for preloaded symbols

[Eli Zaretskii]

Ping!

> Date: Sat, 07 Jan 2023 15:20:07 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> 
> To reproduce:
> 
>   emacs -Q
>   C-h d list.*directory RET
> 
> Note that the resulting *Apropos* buffer only lists built-in functions
> and variables, i.e. those defined in C.  In particular, missing are
> functions/variables from files.el, such as find-directory-functions,
> parse-colon-path, and directory-files-recursively, whose doc strings
> all match the regexp.
> 
> This seems to be due to
> 
>     commit 59732a83c8875c8986d2221600d559a24d8309cc
>     Author: Stefan Monnier <monnier@iro.umontreal.ca>
>     Date:   Thu Dec 30 23:17:45 2021 -0500
> 
>         Don't store docstrings of preloaded .el files in etc/DOC
> 
> The doc string of apropos-documentation does say:
> 
>   Note that by default this command only searches in the file specified by
>   ‘internal-doc-file-name’; i.e., the etc/DOC file.  With C-u prefix,
>   or if ‘apropos-do-all’ is non-nil, it searches all currently defined
>   documentation strings.
> 
> However, since we no longer write the doc strings of preloaded Lisp
> files to DOC, that default no longer makes sense: Emacs should not
> distinguish between built-in primitives and preloaded stuff, and it
> didn't up to and including Emacs 28.
> 
> Moreover, "C-u C-h d" also doesn't work for preloaded functions,
> because:
> 
>   (apropos-documentation-internal 'directory-files-recursively)
>     => nil
> 
> This happens because:
> 
>   (apropos-safe-documentation 'directory-files-recursively)
>     => ("files.elc" . 39940)
> 
> and in that case apropos-documentation-internal calls
> apropos-documentation-check-elc-file, which does something very
> strange: it only looks inside strings that begin with "\n#@", and
> otherwise simply ignores everything else.  Is this also a remnant from
> back when preloaded files had their doc strings in etc/DOC?  Or else
> why does it only look for "#@" comments preceded by a newline?
> 
> Basically, apropos-documentation is currently completely broken on the
> release branch.  The following simple change seems to at least repair
> "C-u C-h d", but I don't think it's sufficient, as we need to
> resurrect the ability to find preloaded symbols without C-u.
> 
> diff --git a/lisp/apropos.el b/lisp/apropos.el
> index b260d88..cff6f1c 100644
> --- a/lisp/apropos.el
> +++ b/lisp/apropos.el
> @@ -1064,7 +1064,9 @@ apropos-documentation-check-elc-file
>        (setq apropos-files-scanned (cons file apropos-files-scanned))
>        (erase-buffer)
>        (insert-file-contents file)
> -      (while (search-forward "\n#@" nil t)
> +      (while (search-forward "#@" nil t)
> +        (or (not (looking-at-p "[0-9]+"))
> +            (progn
>  	;; Read the comment length, and advance over it.
>  	(setq end (read)
>  	      beg (1+ (point))
> @@ -1110,7 +1112,7 @@ apropos-documentation-check-elc-file
>  					       'face apropos-match-face doc)))
>  		      (setcar (nthcdr (if this-is-a-variable 3 2)
>  				      apropos-item)
> -			      doc))))))))))
> +			      doc))))))))))))
>  
> 
> Ideas? patches? comments?
> 
> 
> 
> In GNU Emacs 29.0.60 (build 197, i686-pc-mingw32) of 2023-01-07 built on
>  HOME-C4E4A596F7
> Repository revision: 9d410f8de64e91d16999a9bb5dd884d6d06d22bd
> Repository branch: emacs-29
> Windowing system distributor 'Microsoft Corp.', version 5.1.2600
> System Description: Microsoft Windows XP Service Pack 3 (v5.1.0.2600)
> 
> Configured using:
>  'configure -C --prefix=/d/usr --with-wide-int
>  --enable-checking=yes,glyphs 'CFLAGS=-O0 -gdwarf-4 -g3''
> 
> Configured features:
> ACL GIF GMP GNUTLS HARFBUZZ JPEG JSON LCMS2 LIBXML2 MODULES NOTIFY
> W32NOTIFY PDUMPER PNG RSVG SOUND SQLITE3 THREADS TIFF
> TOOLKIT_SCROLL_BARS TREE_SITTER WEBP XPM ZLIB
> 
> Important settings:
>   value of $LANG: ENU
>   locale-coding-system: cp1255
> 
> Major mode: Lisp Interaction
> 
> Minor modes in effect:
>   tooltip-mode: t
>   global-eldoc-mode: t
>   eldoc-mode: t
>   show-paren-mode: t
>   electric-indent-mode: t
>   mouse-wheel-mode: t
>   tool-bar-mode: t
>   menu-bar-mode: t
>   file-name-shadow-mode: t
>   global-font-lock-mode: t
>   font-lock-mode: t
>   blink-cursor-mode: t
>   line-number-mode: t
>   indent-tabs-mode: t
>   transient-mark-mode: t
>   auto-composition-mode: t
>   auto-encryption-mode: t
>   auto-compression-mode: t
> 
> Load-path shadows:
> None found.
> 
> Features:
> (shadow sort mail-extr dabbrev emacsbug message mailcap yank-media puny
> dired dired-loaddefs rfc822 mml mml-sec password-cache epa derived epg
> rfc6068 epg-config gnus-util text-property-search time-date subr-x
> mm-decode mm-bodies mm-encode mail-parse rfc2231 mailabbrev gmm-utils
> mailheader sendmail rfc2047 rfc2045 ietf-drums mm-util mail-prsvr
> mail-utils cl-extra pp cl-print byte-opt gv bytecomp byte-compile
> help-fns radix-tree mule-util help-mode cl-loaddefs cl-lib apropos rmc
> iso-transl tooltip cconv eldoc paren electric uniquify ediff-hook
> vc-hooks lisp-float-type elisp-mode mwheel dos-w32 ls-lisp disp-table
> term/w32-win w32-win w32-vars term/common-win tool-bar dnd fontset image
> regexp-opt fringe tabulated-list replace newcomment text-mode lisp-mode
> prog-mode register page tab-bar menu-bar rfn-eshadow isearch easymenu
> timer select scroll-bar mouse jit-lock font-lock syntax font-core
> term/tty-colors frame minibuffer nadvice seq simple cl-generic
> indonesian philippine cham georgian utf-8-lang misc-lang vietnamese
> tibetan thai tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek
> romanian slovak czech european ethiopic indian cyrillic chinese
> composite emoji-zwj charscript charprop case-table epa-hook
> jka-cmpr-hook help abbrev obarray oclosure cl-preloaded button loaddefs
> theme-loaddefs faces cus-face macroexp files window text-properties
> overlay sha1 md5 base64 format env code-pages mule custom widget keymap
> hashtable-print-readable backquote threads w32notify w32 lcms2 multi-tty
> make-network-process emacs)
> 
> Memory information:
> ((conses 16 54328 11479)
>  (symbols 48 7273 0)
>  (strings 16 20447 2422)
>  (string-bytes 1 523245)
>  (vectors 16 11853)
>  (vector-slots 8 170903 17702)
>  (floats 8 34 42)
>  (intervals 40 449 141)
>  (buffers 888 13))
> 
> 
> 
> 

bug#60628: 29.0.60; apropos-documentation does not work for preloaded symbols

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> The doc string of apropos-documentation does say:
>
>   Note that by default this command only searches in the file specified by
>   ‘internal-doc-file-name’; i.e., the etc/DOC file.  With C-u prefix,
>   or if ‘apropos-do-all’ is non-nil, it searches all currently defined
>   documentation strings.
>
> However, since we no longer write the doc strings of preloaded Lisp
> files to DOC, that default no longer makes sense: Emacs should not
> distinguish between built-in primitives and preloaded stuff, and it
> didn't up to and including Emacs 28.

Well, we still do as documented.  It's just that what is covered by
‘internal-doc-file-name’ is different now :-)

IOW, we should also fix the doc so it doesn't define the behavior based
on internal details such as ‘internal-doc-file-name’.  We should
probably say that by default we only search through the predefined
functions or something like that.

And as for actually (re)implementing that behavior, we can probably do
it by replacing `mapatoms` with a loop through the elements of
`load-history` corresponding to the `preloaded-file-list`.

> and in that case apropos-documentation-internal calls
> apropos-documentation-check-elc-file, which does something very
> strange: it only looks inside strings that begin with "\n#@", and
> otherwise simply ignores everything else.  Is this also a remnant from
> back when preloaded files had their doc strings in etc/DOC?

Hmm... oh I see, it's because of that Stefan dude:

    commit 900b09c0235d54d56ef5e88d04cca61bc71cbbb7
    Author: Stefan Monnier <monnier@iro.umontreal.ca>
    Date:   Fri Aug 5 08:18:04 2022 -0400
    
        bytecomp.el: Further simplifications enabled by commit 59732a83c8875c
        
        * lisp/emacs-lisp/bytecomp.el (byte-compile-output-docform): Don't
        insert a \n before the #@ docstrings since make-docfile doesn't scan
        .elc files any more.

But searching for "#@" (with or without a \n) is not a good idea anyway,
since there can be false positives and because once we've found such
a "docstring" in a .elc file the problem of finding who this docstring
belongs to again depends on details of how `bytecomp.el` decides to
print out the code.

For `emacs-29` the better option is probably to just revert the
above commit.  And for `master` we should probably rewrite the code so
it uses the POS info in the (FILE . POS) instead of searching for
#@, but that requires a significant rewrite.
Basically, we should loop through all the functions, collecting their
(FILE . POS) data, grouping them by FILE, so we end up for every
FILE with a list of (FUNCTION . POS) so we can just jump to POS without
having to search for #@ (we can still check the presence of #@ as
sanity check).


        Stefan

bug#60628: 29.0.60; apropos-documentation does not work for preloaded symbols

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> For `emacs-29` the better option is probably to just revert the
> above commit.

Then again, since \n#@ can also match false positives, we may as well
keep the above commit and just search for #@ instead.

The patch below should fix both problems in a kind of "minimalist" way
(i.e. meant for `emacs-29`).
Can someone confirm it also works for them while I clean it up?


        Stefan


diff --git a/lisp/apropos.el b/lisp/apropos.el
index b260d889955..b66f27b8a9e 100644
--- a/lisp/apropos.el
+++ b/lisp/apropos.el
@@ -913,8 +913,19 @@
           (apropos-sort-by-scores apropos-documentation-sort-by-scores)
           f v sf sv)
       (apropos-documentation-check-doc-file)
-      (if do-all
-          (mapatoms
+      (funcall
+       (if do-all #'mapatoms
+         (lambda (f)
+           (let ((preloaded-regexp
+                  (concat "\\`"
+                          (regexp-quote lisp-directory)
+                          (regexp-opt preloaded-file-list)
+                          "\\.elc?\\'")))
+             (dolist (x load-history)
+               (when (string-match preloaded-regexp (car x))
+                 (dolist (def (cdr x))
+                   (when (eq 'defun (car-safe def))
+                     (funcall f (cdr def)))))))))
            (lambda (symbol)
              (setq f (apropos-safe-documentation symbol)
                    v (get symbol 'variable-documentation))
@@ -939,7 +950,7 @@
                          (cons (list symbol
                                      (+ (apropos-score-symbol symbol 2) sf sv)
                                      f v)
-                               apropos-accumulator)))))))
+                           apropos-accumulator))))))
       (apropos-print nil "\n----------------\n" nil t))))
 
 \f
@@ -1064,10 +1064,12 @@
       (setq apropos-files-scanned (cons file apropos-files-scanned))
       (erase-buffer)
       (insert-file-contents file)
-      (while (search-forward "\n#@" nil t)
+      (while (search-forward "#@" nil t)
 	;; Read the comment length, and advance over it.
-	(setq end (read)
-	      beg (1+ (point))
+	;; This #@ may be a false positive, so don't get upset if
+	;; it's not followed by the expected number of bytes to skip.
+	(when (and (setq end (ignore-errors (read))) (natnump end))
+	  (setq beg (1+ (point))
 	      end (+ (point) end -1))
 	(forward-char)
 	(if (save-restriction
@@ -1110,7 +1112,7 @@
 					       'face apropos-match-face doc)))
 		      (setcar (nthcdr (if this-is-a-variable 3 2)
 				      apropos-item)
-			      doc))))))))))
+				doc)))))))))))
 
 
 

bug#60628: 29.0.60; apropos-documentation does not work for preloaded symbols

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: bug-gnu-emacs@gnu.org
> Date: Fri, 13 Jan 2023 15:10:02 -0500
> 
> > For `emacs-29` the better option is probably to just revert the
> > above commit.
> 
> Then again, since \n#@ can also match false positives, we may as well
> keep the above commit and just search for #@ instead.
> 
> The patch below should fix both problems in a kind of "minimalist" way
> (i.e. meant for `emacs-29`).
> Can someone confirm it also works for them while I clean it up?

It works much better, thanks.

However, some symbols are still not shown, for some reason.  For
example, find-directory-functions and file-name-at-point-functions
(both are defcustoms).  I indeed don't see their doc strings in
files.elc.  Using "C-u C-h d" does produce an *Apropos* buffer with
find-directory-functions in it, but file-name-at-point-functions are
still not mentioned.  I guess this is a separate issue, since Emacs 28
seems to have similar problems with those 2 defcustoms?

Thanks.

bug#60628: 29.0.60; apropos-documentation does not work for preloaded symbols

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> The patch below should fix both problems in a kind of "minimalist" way
>> (i.e. meant for `emacs-29`).
>> Can someone confirm it also works for them while I clean it up?
>
> It works much better, thanks.

OK, thanks.  I pushed a slightly better version of it to `emacs-29`.

> However, some symbols are still not shown, for some reason.  For
> example, find-directory-functions and file-name-at-point-functions
> (both are defcustoms).  I indeed don't see their doc strings in
> files.elc.  Using "C-u C-h d" does produce an *Apropos* buffer with
> find-directory-functions in it, but file-name-at-point-functions are
> still not mentioned.  I guess this is a separate issue, since Emacs 28
> seems to have similar problems with those 2 defcustoms?

I pushed to master a rewrite of the .elc scanning code which may help
(it wasn't its purpose, but it might be a side benefit).


        Stefan