bug#42753: 26.3; `info-lookup.el': Add all manuals delivered by `emacs -Q', for `emacs-lisp-mode'

bug#42753: 26.3; `info-lookup.el': Add all manuals delivered by `emacs -Q', for `emacs-lisp-mode'

[Drew Adams]

Most of the manuals delivered with Emacs are missing from
`info-look.el', for looking up Emacs-Lisp commands, non-interactive
functions, options, and other variables.

I believe that a fix for this is to replace the :doc-spec value here:

(info-lookup-maybe-add-help
 :mode 'emacs-lisp-mode
 :regexp "[^][()`''',\" \t\n]+"
 :doc-spec '(;; Commands with key sequences appear in nodes as `foo' and
             ;; those without as `M-x foo'.
             ("(emacs)Command Index"  nil "['`']\\(M-x[ \t\n]+\\)?" "['']")
             ;; Variables normally appear in nodes as just `foo'.
             ("(emacs)Variable Index" nil "['`']" "['']")
             ;; Almost all functions, variables, etc appear in nodes as
             ;; " -- Function: foo" etc.  A small number of aliases and
             ;; symbols appear only as `foo', and will miss out on exact
             ;; positions.  Allowing `foo' would hit too many false matches
             ;; for things that should go to Function: etc, and those latter
             ;; are much more important.  Perhaps this could change if some
             ;; sort of fallback match scheme existed.
             ("(elisp)Index"          nil "^ -+ .*: " "\\( \\|$\\)")
             ("(cl)Function Index"    nil "^ -+ .*: " "\\( \\|$\\)")
             ("(cl)Variable Index"    nil "^ -+ .*: " "\\( \\|$\\)")))

with this :doc-spec value:

'(;; Commands with key sequences appear in nodes as `foo' and
  ;; those without as `M-x foo'.
  ("(emacs)Command Index"                nil "['`']\\(M-x[ \t\n]+\\)?" "['']")
  ;; Variables normally appear in nodes as just `foo'.
  ("(emacs)Variable Index"               nil "['`']" "['']")
  ;; Almost all functions, variables, etc appear in nodes as
  ;; " -- Function: foo" etc.  A small number of aliases and
  ;; symbols appear only as `foo', and will miss out on exact
  ;; positions.  Allowing `foo' would hit too many false matches
  ;; for things that should go to Function: etc, and those latter
  ;; are much more important.  Perhaps this could change if some
  ;; sort of fallback match scheme existed.
  ("(elisp)Index"                        nil "^ -+ .*: " "\\( \\|$\\)")
  ("(info)Index"                         nil "^ -+ .*: " "\\( \\|$\\)")
  ("(efaq-w32)Indexes"                   nil "^ -+ .*: " "\\( \\|$\\)")
  ("(eintr)Index"                        nil "^ -+ .*: " "\\( \\|$\\)")
  ("(ccmode)Command and Function Index"  nil "^ -+ .*: " "\\( \\|$\\)")
  ("(ccmode)Variable Index"              nil "^ -+ .*: " "\\( \\|$\\)")
  ("(idlwave)Index"                      nil "^ -+ .*: " "\\( \\|$\\)")
  ("(octave-mode)Lisp Function Index"    nil "^ -+ .*: " "\\( \\|$\\)")
  ("(octave-mode)Variable Index"         nil "^ -+ .*: " "\\( \\|$\\)")
  ("(org)Command and Function Index"     nil "^ -+ .*: " "\\( \\|$\\)")
  ("(org)Variable Index"                 nil "^ -+ .*: " "\\( \\|$\\)")
  ("(vhdl-mode)Command Index"            nil "^ -+ .*: " "\\( \\|$\\)")
  ("(vhdl-mode)Variable Index"           nil "^ -+ .*: " "\\( \\|$\\)")
  ("(eudc)Index"                         nil "^ -+ .*: " "\\( \\|$\\)")
  ("(emacs-gnutls)Function Index"        nil "^ -+ .*: " "\\( \\|$\\)")
  ("(emacs-gnutls)Variable Index"        nil "^ -+ .*: " "\\( \\|$\\)")
  ("(gnus)Index"                         nil "^ -+ .*: " "\\( \\|$\\)")
  ("(mh-e)Command Index"                 nil "^ -+ .*: " "\\( \\|$\\)")
  ("(mh-e)Option Index"                  nil "^ -+ .*: " "\\( \\|$\\)")
  ("(message)Index"                      nil "^ -+ .*: " "\\( \\|$\\)")
  ("(newsticker)Index"                   nil "^ -+ .*: " "\\( \\|$\\)")
  ("(pgg)Function Index"                 nil "^ -+ .*: " "\\( \\|$\\)")
  ("(pgg)Variable Index"                 nil "^ -+ .*: " "\\( \\|$\\)")
  ("(sasl)Function Index"                nil "^ -+ .*: " "\\( \\|$\\)")
  ("(sasl)Variable Index"                nil "^ -+ .*: " "\\( \\|$\\)")
  ("(sc)Command Index"                   nil "^ -+ .*: " "\\( \\|$\\)")
  ("(sc)Variable Index"                  nil "^ -+ .*: " "\\( \\|$\\)")
  ("(sieve)Index"                        nil "^ -+ .*: " "\\( \\|$\\)")
  ("(tramp)Function Index"               nil "^ -+ .*: " "\\( \\|$\\)")
  ("(tramp)Variable Index"               nil "^ -+ .*: " "\\( \\|$\\)")
  ("(autotype)Command Index"             nil "^ -+ .*: " "\\( \\|$\\)")
  ("(autotype)Variable Index"            nil "^ -+ .*: " "\\( \\|$\\)")
  ("(calc)Lisp Function Index"           nil "^ -+ .*: " "\\( \\|$\\)")
  ("(calc)Variable Index"                nil "^ -+ .*: " "\\( \\|$\\)")
  ("(dired-x)Command Index"              nil "^ -+ .*: " "\\( \\|$\\)")
  ("(dired-x)Variable Index"             nil "^ -+ .*: " "\\( \\|$\\)")
  ("(eieio)Function Index"               nil "^ -+ .*: " "\\( \\|$\\)")
  ("(ert)Index"                          nil "^ -+ .*: " "\\( \\|$\\)")
  ("(eww)Lisp Function Index"            nil "^ -+ .*: " "\\( \\|$\\)")
  ("(eww)Variable Index"                 nil "^ -+ .*: " "\\( \\|$\\)")
  ("(epa)Function Index"                 nil "^ -+ .*: " "\\( \\|$\\)")
  ("(epa)Variable Index"                 nil "^ -+ .*: " "\\( \\|$\\)")
  ("(ediff)Index"                        nil "^ -+ .*: " "\\( \\|$\\)")
  ("(eshell)Function and Variable Index" nil "^ -+ .*: " "\\( \\|$\\)")
  ("(flymake)Index"                      nil "^ -+ .*: " "\\( \\|$\\)")
  ("(forms)Index"                        nil "^ -+ .*: " "\\( \\|$\\)")
  ("(htmlfontify)Index"                  nil "^ -+ .*: " "\\( \\|$\\)")
  ("(ido)Variable Index"                 nil "^ -+ .*: " "\\( \\|$\\)")
  ("(pl-cvs)Function and Variable Index" nil "^ -+ .*: " "\\( \\|$\\)")
  ("(reftex)Index"                       nil "^ -+ .*: " "\\( \\|$\\)")
  ("(ses)Index"                          nil "^ -+ .*: " "\\( \\|$\\)")
  ("(srecode)Index"                      nil "^ -+ .*: " "\\( \\|$\\)")
  ("(semantic)Index"                     nil "^ -+ .*: " "\\( \\|$\\)")
  ("(speedbar)Index"                     nil "^ -+ .*: " "\\( \\|$\\)")
  ("(viper)Function Index"               nil "^ -+ .*: " "\\( \\|$\\)")
  ("(viper)Variable Index"               nil "^ -+ .*: " "\\( \\|$\\)")
  ("(wisent)Index"                       nil "^ -+ .*: " "\\( \\|$\\)")
  ("(woman)Command Index"                nil "^ -+ .*: " "\\( \\|$\\)")
  ("(woman)Variable Index"               nil "^ -+ .*: " "\\( \\|$\\)")
  ("(auth)Function Index"                nil "^ -+ .*: " "\\( \\|$\\)")
  ("(auth)Variable Index"                nil "^ -+ .*: " "\\( \\|$\\)")
  ("(cl)Function Index"                  nil "^ -+ .*: " "\\( \\|$\\)")
  ("(cl)Variable Index"                  nil "^ -+ .*: " "\\( \\|$\\)")
  ("(dbus)Index"                         nil "^ -+ .*: " "\\( \\|$\\)")
  ("(mime)Index"                         nil "^ -+ .*: " "\\( \\|$\\)")
  ("(smtpmail)Index"                     nil "^ -+ .*: " "\\( \\|$\\)")
  ("(url)Function Index"                 nil "^ -+ .*: " "\\( \\|$\\)")
  ("(url)Variable Index"                 nil "^ -+ .*: " "\\( \\|$\\)")
  ("(widget)Index"                       nil "^ -+ .*: " "\\( \\|$\\)")
  )

That covers all of the manuals I get with `emacs -Q', which have indexes
with entries for Emacs-Lisp commands, functions, options, and variables.

Please do that, or something similar, so such symbols get picked up by
`info-lookup'.  I used the same order here as the manuals are presented
in the top-level menu.  You may want to change the order.  Another
possible change you might want to make is to use a value similar to the
one for "(emacs)Command Index" for other "(...)Command Index" entries.

["(efaq-w32)Indexes" should probably be included conditionally, for MS
Windows.  And perhaps on other platforms `emacs -Q' gives a different
set of manuals, so other adjustments could be made.]

Note that `info-lookup' is used by 3rd-party libraries, so this will
help them as well.

It seems like there should be an easy way to just _add_ such missing
entries after the fact, e.g., in a 3rd-party library.  For example, I
was thinking that perhaps `info-lookup-add-help' (without the `-maybe')
would take care of that.

But apparently not.  While trying that I found I had to reset the
various global values in `info-look.el' - once the alist and cache are
defined there doesn't seem to be a way (already provided) to update them.
`info-look.el' seems a bit monolithic, in this regard.  But perhaps I'm
missing something.

In GNU Emacs 26.3 (build 1, x86_64-w64-mingw32)
 of 2019-08-29
Repository revision: 96dd0196c28bc36779584e47fffcca433c9309cd
Windowing system distributor `Microsoft Corp.', version 10.0.18362
Configured using:
 `configure --without-dbus --host=x86_64-w64-mingw32
 --without-compress-install 'CFLAGS=-O2 -static -g3''

bug#42753: 26.3; `info-lookup.el': Add all manuals delivered by `emacs -Q', for `emacs-lisp-mode'

[Eli Zaretskii]

> Date: Fri, 7 Aug 2020 15:20:15 -0700 (PDT)
> From: Drew Adams <drew.adams@oracle.com>
> 
> with this :doc-spec value:
> 
> '(;; Commands with key sequences appear in nodes as `foo' and
>   ;; those without as `M-x foo'.
>   ("(emacs)Command Index"                nil "['`']\\(M-x[ \t\n]+\\)?" "['']")
>   ;; Variables normally appear in nodes as just `foo'.
>   ("(emacs)Variable Index"               nil "['`']" "['']")
>   ;; Almost all functions, variables, etc appear in nodes as
>   ;; " -- Function: foo" etc.  A small number of aliases and
>   ;; symbols appear only as `foo', and will miss out on exact
>   ;; positions.  Allowing `foo' would hit too many false matches
>   ;; for things that should go to Function: etc, and those latter
>   ;; are much more important.  Perhaps this could change if some
>   ;; sort of fallback match scheme existed.
>   ("(elisp)Index"                        nil "^ -+ .*: " "\\( \\|$\\)")
>   ("(info)Index"                         nil "^ -+ .*: " "\\( \\|$\\)")
>   ("(efaq-w32)Indexes"                   nil "^ -+ .*: " "\\( \\|$\\)")
>   ("(eintr)Index"                        nil "^ -+ .*: " "\\( \\|$\\)")
>   ("(ccmode)Command and Function Index"  nil "^ -+ .*: " "\\( \\|$\\)")
>   ("(ccmode)Variable Index"              nil "^ -+ .*: " "\\( \\|$\\)")
> [...]

Wouldn't it make much more sense to make the :regexp attribute more
specialized for the non-core manuals?  AFAIU, your proposal will
greatly slow down "C-h S" in emacs-lisp-mode, because of the need to
scan many more indices, and for no good reason AFAICT.

> ["(efaq-w32)Indexes" should probably be included conditionally, for MS
> Windows.  And perhaps on other platforms `emacs -Q' gives a different
> set of manuals, so other adjustments could be made.]

That would be sub-optimal, IMO, because questions specific to certain
platforms can be asked and answered by people who run on other
platforms.  Instead, we should be more selective in the regexps that
match symbols; for example, a symbol that starts with "w32-" should
trigger search in efaq-w32 as well.

bug#42753: 26.3; `info-lookup.el': Add all manuals delivered by `emacs -Q', for `emacs-lisp-mode'

[Drew Adams]

> Wouldn't it make much more sense to make the :regexp attribute more
> specialized for the non-core manuals?  AFAIU, your proposal will
> greatly slow down "C-h S" in emacs-lisp-mode, because of the need to
> scan many more indices, and for no good reason AFAICT.
> 
> > ["(efaq-w32)Indexes" should probably be included conditionally, for MS
> > Windows.  And perhaps on other platforms `emacs -Q' gives a different
> > set of manuals, so other adjustments could be made.]
> 
> That would be sub-optimal, IMO, because questions specific to certain
> platforms can be asked and answered by people who run on other
> platforms.  Instead, we should be more selective in the regexps that
> match symbols; for example, a symbol that starts with "w32-" should
> trigger search in efaq-w32 as well.

Feel free to make any such adjustments you feel are
appropriate.

I got into this because I discovered that functions
etc. in the Dired-X manual weren't handled.
___

FWIW and IIUC, it's not really about "`C-h S' _in_
emacs-lisp-mode".  It's about Emacs-Lisp symbols in
manuals (indexes of manuals).  The Dired-X manual,
for instance, is not at all about Emacs-Lisp mode.

A separate info-lookup feature looks at the current
mode and takes it into account.  IIUC, there's nothing
in info-look.el that requires or assumes that the
current mode determines the kind of things to look up.

For example, a 3rd-party library that provides a link
for manual look-up when in `C-h f' *Help* (and there
are a few such libraries) does not make any use of the
current mode (which is `help-mode').  `C-h f' is about
Elisp functions, so the manuals to look in are those
that have entries for Elisp functions in one or more
indexes.
___

And as I said in the bug report, I may misunderstand
but it seems like there is no easy/good way for a
user or library to simply extend the predefined
behavior to, say, cover another manual (whose index
has Elisp symbols).  It seems like the behavior is
essentially chiseled in stone when info-look.el is
loaded - a once-and-for-all configuration of which
manuals to check.

If I understand that correctly, then that's the real,
underlying bug that should be fixed (not just fix the
predefined list of manuals handled).  If it were easy
to simply "add" a manual (e.g. Dired-X) to those
handling Elisp symbols then users and 3rd-party code
could do that.  Similarly, for subtracting a manual etc.

They could, themselves optimize (or not) any choice
or conditionalization of manual inclusion, and the
same for the regexps to use.

IOW, a proper bug fix (IIUC what's happening) would
be to give users easy control over info-lookup
behavior.  I'd be happy to learn that I'm mistaken
and that's already the case - and how to do that.
I didn't find it so.

bug#42753: 26.3; `info-lookup.el': Add all manuals delivered by `emacs -Q', for `emacs-lisp-mode'

[Michael Albinus]

Drew Adams <drew.adams@oracle.com> writes:

Hi Drew,

> ___
>
> FWIW and IIUC, it's not really about "`C-h S' _in_
> emacs-lisp-mode".  It's about Emacs-Lisp symbols in
> manuals (indexes of manuals).  The Dired-X manual,
> for instance, is not at all about Emacs-Lisp mode.
>
> A separate info-lookup feature looks at the current
> mode and takes it into account.  IIUC, there's nothing
> in info-look.el that requires or assumes that the
> current mode determines the kind of things to look up.
>
> For example, a 3rd-party library that provides a link
> for manual look-up when in `C-h f' *Help* (and there
> are a few such libraries) does not make any use of the
> current mode (which is `help-mode').  `C-h f' is about
> Elisp functions, so the manuals to look in are those
> that have entries for Elisp functions in one or more
> indexes.
> ___
>
> And as I said in the bug report, I may misunderstand
> but it seems like there is no easy/good way for a
> user or library to simply extend the predefined
> behavior to, say, cover another manual (whose index
> has Elisp symbols).  It seems like the behavior is
> essentially chiseled in stone when info-look.el is
> loaded - a once-and-for-all configuration of which
> manuals to check.
>
> If I understand that correctly, then that's the real,
> underlying bug that should be fixed (not just fix the
> predefined list of manuals handled).  If it were easy
> to simply "add" a manual (e.g. Dired-X) to those
> handling Elisp symbols then users and 3rd-party code
> could do that.  Similarly, for subtracting a manual etc.
>
> They could, themselves optimize (or not) any choice
> or conditionalization of manual inclusion, and the
> same for the regexps to use.
>
> IOW, a proper bug fix (IIUC what's happening) would
> be to give users easy control over info-lookup
> behavior.  I'd be happy to learn that I'm mistaken
> and that's already the case - and how to do that.
> I didn't find it so.

Tramp does this for its own symbols in tramp-integration.el, see block
starting

(with-eval-after-load 'info-look

It's a little bit hacky; a more simple interface for self-registration
would be appreciated.

Best regards, Michael.

bug#42753: 26.3; `info-lookup.el': Add all manuals delivered by `emacs -Q', for `emacs-lisp-mode'

[Lars Ingebrigtsen]

Michael Albinus <michael.albinus@gmx.de> writes:

> (with-eval-after-load 'info-look
>
> It's a little bit hacky; a more simple interface for self-registration
> would be appreciated.

Yes, indeed.  But looking at the code, it seems like the basic info-look
infrastructure doesn't really want to support Eli's idea here.  That is,
for `emacs-lisp-mode', we want to register a bunch of regexps that map
to different manuals (i.e., if the symbol starts with rmail, look in the
rmail manual and so on), but there's just one regexp per mode.

So it seems like a bigger rewrite is necessary here, unfortunately.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#42753: 26.3; `info-lookup.el': Add all manuals delivered by `emacs -Q', for `emacs-lisp-mode'

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Yes, indeed.  But looking at the code, it seems like the basic info-look
> infrastructure doesn't really want to support Eli's idea here.  That is,
> for `emacs-lisp-mode', we want to register a bunch of regexps that map
> to different manuals (i.e., if the symbol starts with rmail, look in the
> rmail manual and so on), but there's just one regexp per mode.
>
> So it seems like a bigger rewrite is necessary here, unfortunately.

I've now made some headway here, and it basically works now for many of
the "misc" manuals.  The remaining problem is that there's not a
standardised way to name the indices (eww has "lisp function index"
instead of "function index", for instance), so I think I have to extend
info-look with a way to get all the index nodes or something.

Or rename the index nodes in the manuals to all follow the same naming
conventions...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#42753: 26.3; `info-lookup.el': Add all manuals delivered by `emacs -Q', for `emacs-lisp-mode'

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Eli Zaretskii <eliz@gnu.org>,  42753@debbugs.gnu.org,  Drew Adams
>  <drew.adams@oracle.com>
> Date: Fri, 08 Oct 2021 15:36:48 +0200
> 
> Or rename the index nodes in the manuals to all follow the same naming
> conventions...

I think this alternative cannot fly, because many manuals are not
under our control.

bug#42753: 26.3; `info-lookup.el': Add all manuals delivered by `emacs -Q', for `emacs-lisp-mode'

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

>> Or rename the index nodes in the manuals to all follow the same naming
>> conventions...
>
> I think this alternative cannot fly, because many manuals are not
> under our control.

They are integrated in the Emacs source tree, so we could negotiate with
the maintainers. Even if we cannot change all manuals this way, the
number of exceptions in info-lookup would decrease.

I could start the discussion with the Tramp maintainer, I assume he is
willing to change. Lars could discuss the eww and gnus cases, if needed :-)

Of course, first we need to document the naming conventions.

Best regards, Michael.

bug#42753: 26.3; `info-lookup.el': Add all manuals delivered by `emacs -Q', for `emacs-lisp-mode'

[Eli Zaretskii]

> From: Michael Albinus <michael.albinus@gmx.de>
> Cc: Lars Ingebrigtsen <larsi@gnus.org>,  42753@debbugs.gnu.org,
>   drew.adams@oracle.com
> Date: Fri, 08 Oct 2021 16:33:11 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> Or rename the index nodes in the manuals to all follow the same naming
> >> conventions...
> >
> > I think this alternative cannot fly, because many manuals are not
> > under our control.
> 
> They are integrated in the Emacs source tree, so we could negotiate with
> the maintainers. Even if we cannot change all manuals this way, the
> number of exceptions in info-lookup would decrease.

I meant the likes of Python manual, the glibc manual, and others.
Good luck negotiating with them.  And then there's always a
possibility that the user will want to add some manual we know nothing
about.

I think a better solution will be to be tolerant, i.e. look for nodes
that contain "index", and maybe also "function" or "variable", in them
regardless of the rest.

bug#42753: 26.3; `info-lookup.el': Add all manuals delivered by `emacs -Q', for `emacs-lisp-mode'

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

Hi Eli,

>> They are integrated in the Emacs source tree, so we could negotiate with
>> the maintainers. Even if we cannot change all manuals this way, the
>> number of exceptions in info-lookup would decrease.
>
> I meant the likes of Python manual, the glibc manual, and others.
> Good luck negotiating with them.  And then there's always a
> possibility that the user will want to add some manual we know nothing
> about.

My reply was about Emacs integrated manuals.

> I think a better solution will be to be tolerant, i.e. look for nodes
> that contain "index", and maybe also "function" or "variable", in them
> regardless of the rest.

Sure. Same for *ELPA manuals. But I believe we also should use some
conventions for manuals under our control.

Best regards, Michael.

bug#42753: 26.3; `info-lookup.el': Add all manuals delivered by `emacs -Q', for `emacs-lisp-mode'

[Eli Zaretskii]

> From: Michael Albinus <michael.albinus@gmx.de>
> Cc: larsi@gnus.org,  42753@debbugs.gnu.org,  drew.adams@oracle.com
> Date: Fri, 08 Oct 2021 18:23:38 +0200
> 
> > I think a better solution will be to be tolerant, i.e. look for nodes
> > that contain "index", and maybe also "function" or "variable", in them
> > regardless of the rest.
> 
> Sure. Same for *ELPA manuals. But I believe we also should use some
> conventions for manuals under our control.

That should be a Good Thing regardless.

bug#42753: 26.3; `info-lookup.el': Add all manuals delivered by `emacs -Q', for `emacs-lisp-mode'

[Lars Ingebrigtsen]

Michael Albinus <michael.albinus@gmx.de> writes:

> My reply was about Emacs integrated manuals.

My improvements here are only for the integrated Emacs, so we can edit
them as we wish.

But looking at this again today, I think the best solution here is to
actually do the boring job to see what the index sections are actually
called, and use that in the new info-look code, so I'll be doing that
instead.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#42753: 26.3; `info-lookup.el': Add all manuals delivered by `emacs -Q', for `emacs-lisp-mode'

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> But looking at this again today, I think the best solution here is to
> actually do the boring job to see what the index sections are actually
> called, and use that in the new info-look code, so I'll be doing that
> instead.

Now done, and boy was that boring.  But `C-h S' now seems to work fine
for all the misc manuals in Emacs.  (Well, the ones I tested, at least.)

Some of the manuals don't have index nodes, but perhaps that should be fixed.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no