Patch: perform autoloading when docs is missing from autoload object

Patch: perform autoloading when docs is missing from autoload object

[Arthur Miller]

[-- Attachment #1: Type: text/plain, Size: 1389 bytes --]

I would like help functions to autoload files when help on symbols is
requestd. Built-in help do that only in limited cases, while helpful dutifully
loads libraries when symbol is not loaded. I would actually prefer to just load
documentation not the entire library, and I think I can do that, but it is a
little bit more involved (have to find source and read the docs), unlike just
performing autoloading.

About the patch: there is already an help-enable-autoload option for built-in
help, probably meant to do exactly what I want, but it is much more savvy what
it loads (only kemaps I think?).

I have introduced new variable, help-enable-symbol-autoload, and more aggressive
autoloading will be performed only when this one is set to 't, so it is an
opt-in option. I would gladly skip that extra variable and use original one, but
that would turn it into opt-out option which might not be desired by everyone.

I am not sure if understand exact usage of built-in help functions, but the
main entry should be describe-* functions, and this should only affect those? I
am little bit unsecure here.

I have two versions, I prefer the first, help-fns.el only one, but as said,
I am a bit unsure how help functions are used, so I have attached an alternative
too. To note: this is only for functions, I am not sure if it is needed for
variables?

It is just a proposal.

Best regards
/a


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-only-help-fns.patch --]
[-- Type: text/x-patch, Size: 1351 bytes --]

From 45e387e15166c5194fe4b78ec01bb65f0db9bcb4 Mon Sep 17 00:00:00 2001
From: Arthur Miller <arthur.miller@live.com>
Date: Wed, 15 Sep 2021 15:15:30 +0200
Subject: [PATCH] Do autoload if docs are not present in autoloads

* lisp/help-fns.el ('help-enable-symbol-autoload'): New option.
(help-fns--analyze-function): Perform autoloading when docs are
missing from autoload objects.
---
 lisp/help-fns.el | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/lisp/help-fns.el b/lisp/help-fns.el
index a7219ede94..575c69638d 100644
--- a/lisp/help-fns.el
+++ b/lisp/help-fns.el
@@ -132,6 +132,12 @@ help-enable-completion-autoload
   :group 'help
   :version "26.3")
 
+(defcustom help-enable-symbol-autoload nil
+  "Perform autoload if docs are missing from autoload objects."
+  :type 'boolean
+  :group 'help
+  :version "28.1")
+
 (defun help--symbol-class (s)
   "Return symbol class characters for symbol S."
   (when (stringp s)
@@ -823,6 +829,11 @@ help-fns--analyze-function
                        f))
 		    ((subrp def) (intern (subr-name def)))
                     (t def))))
+
+    (and (autoloadp real-def) (not (nth 2 real-def))
+         help-enable-symbol-autoload
+         (autoload-do-load real-def))
+
     (list real-function def aliased real-def)))
 
 (defun help-fns-function-description-header (function)
-- 
2.33.0


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: 0002-with-help.el.patch --]
[-- Type: text/x-patch, Size: 1788 bytes --]

From 1f6fd7ee99b3807e012d366d71679ae917cac096 Mon Sep 17 00:00:00 2001
From: Arthur Miller <arthur.miller@live.com>
Date: Wed, 15 Sep 2021 12:40:34 +0200
Subject: [PATCH] Load symbols when docs are not present in autoloads

* lisp/help-fns.el ('help-enable-symbol-autoload'): New option.
* lisp/help.el (help-function-arglist): Check if docs are present in
autoload object, and perform autoload if they are not.
---
 lisp/help-fns.el | 6 ++++++
 lisp/help.el     | 4 ++++
 2 files changed, 10 insertions(+)

diff --git a/lisp/help-fns.el b/lisp/help-fns.el
index a7219ede94..842bcb39fe 100644
--- a/lisp/help-fns.el
+++ b/lisp/help-fns.el
@@ -132,6 +132,12 @@ help-enable-completion-autoload
   :group 'help
   :version "26.3")
 
+(defcustom help-enable-symbol-autoload nil
+  "Perform autoload when documentation is not present in autoload object."
+  :type 'boolean
+  :group 'help
+  :version "28.1")
+
 (defun help--symbol-class (s)
   "Return symbol class characters for symbol S."
   (when (stringp s)
diff --git a/lisp/help.el b/lisp/help.el
index 29ae340481..9b6ae0c821 100644
--- a/lisp/help.el
+++ b/lisp/help.el
@@ -1883,6 +1883,10 @@ help-function-arglist
   "Return a formal argument list for the function DEF.
 If PRESERVE-NAMES is non-nil, return a formal arglist that uses
 the same names as used in the original source code, when possible."
+  ;; Load docs for autoloads when doc is missing.
+  (if (and (autoloadp def) (not (nth 2 def)) help-enable-symbol-autoload
+           (not (and (nth 4 def) (eq (nth 4 def) 'keymap))))
+      (autoload-do-load def))
   ;; Handle symbols aliased to other symbols.
   (if (and (symbolp def) (fboundp def)) (setq def (indirect-function def)))
   ;; Advice wrappers have "catch all" args, so fetch the actual underlying
-- 
2.33.0

Re: Patch: perform autoloading when docs is missing from autoload object

[Stefan Monnier]

> loads libraries when symbol is not loaded. I would actually prefer to just load
> documentation not the entire library, and I think I can do that, but it is a
> little bit more involved (have to find source and read the docs), unlike just
> performing autoloading.

Loading a docstring from the .el file can be tricky/cumbersome (at
least if we want it to be reliable and 100% correct), but as for loading
a docstring from a .elc file this is standard (its controlled by
the variable `byte-compile-dynamic-docstrings`), although currently it
seems it's not done for autoloads.


        Stefan

Re: Patch: perform autoloading when docs is missing from autoload object

[Arthur Miller]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> loads libraries when symbol is not loaded. I would actually prefer to just load
>> documentation not the entire library, and I think I can do that, but it is a
>> little bit more involved (have to find source and read the docs), unlike just
>> performing autoloading.
>
> Loading a docstring from the .el file can be tricky/cumbersome (at
> least if we want it to be reliable and 100% correct), but as for loading
> a docstring from a .elc file this is standard (its controlled by
> the variable `byte-compile-dynamic-docstrings`), although currently it
> seems it's not done for autoloads.

Indeed. My thoughts were to read .elc when avialable, and .el only if there is
no .elc file available. I didn't want to write too much about that, I wanted to
get something usable; I see doc loading as an optimization to proposed
patch. If/when I it will just replace autoload-do-load. It is one of those
"would-be-nice-to-have todo items".

Until than, the proposed patch is functionally equivavelnt to what helpful does
in this regard. I would like to hear opinion if I should look at some other
place to implement this, and if the strategy with opt-in/opt-out and extra
defcustom option is OK.

Re: Patch: perform autoloading when docs is missing from autoload object

[Lars Ingebrigtsen]

Arthur Miller <arthur.miller@live.com> writes:

> I would like help functions to autoload files when help on symbols is
> requestd.

In which cases will this make a difference when displaying the *Help*
buffer for the symbol?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Patch: perform autoloading when docs is missing from autoload object

[Arthur Miller]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Arthur Miller <arthur.miller@live.com> writes:
>
>> I would like help functions to autoload files when help on symbols is
>> requestd.
>
> In which cases will this make a difference when displaying the *Help*
> buffer for the symbol?

In case where the symbol is not loaded, and there is no doc entry in autoload
object.

Doc entry is optional for autoload objects.

It can be useful when a handcrafts an autoload statement, and omitt doc entry in
the object, or it someone might write an application that does not omit docs in
autoload statements. I personally don't use package-quickstart.el and I don't
use built-in code to generate autoloads for my system, because it is faster and
less resource consuming to not to. I just emit symbol name and path. Helpful
autoload on deman when I ask for docs, while built-in does not, so I think it
would be useful in built-in help too.

Re: Patch: perform autoloading when docs is missing from autoload object

[Lars Ingebrigtsen]

Arthur Miller <arthur.miller@live.com> writes:

>> In which cases will this make a difference when displaying the *Help*
>> buffer for the symbol?
>
> In case where the symbol is not loaded, and there is no doc entry in autoload
> object.

Oh, right --- I was thinking about ;;;###autoload, where the doc string
is available always, not

(autoload 'foo "foo")

Yes, in those cases it would indeed be helpful if the help machinery
just went ahead and loaded the library.  However, there are some
instances where loading a library has unexpected side effects, like
enabling new modes and keymaps.  Doing so is frowned upon (loading a
library shouldn't do those things), but it happens.

If this is a concern, then we could perhaps add a button to *Help* to do
the loading instead of loading automatically?

But perhaps just loading the file automatically would be fine anyway --
it's a bug for libraries to side-effect to an annoying degree upon load.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Patch: perform autoloading when docs is missing from autoload object

[Arthur Miller]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Arthur Miller <arthur.miller@live.com> writes:
>
>>> In which cases will this make a difference when displaying the *Help*
>>> buffer for the symbol?
>>
>> In case where the symbol is not loaded, and there is no doc entry in autoload
>> object.
>
> Oh, right --- I was thinking about ;;;###autoload, where the doc string
> is available always, not
>
> (autoload 'foo "foo")
>
> Yes, in those cases it would indeed be helpful if the help machinery
> just went ahead and loaded the library.  However, there are some
> instances where loading a library has unexpected side effects, like
> enabling new modes and keymaps.  Doing so is frowned upon (loading a
> library shouldn't do those things), but it happens.
>
> If this is a concern, then we could perhaps add a button to *Help* to do
> the loading instead of loading automatically?
It would be one option. The other one would be to warn about possible
unwanting side-effects in docs.

I wanted to later switch to just fetch the docs instead of performing
load. Loading docs can be done without evaluating the code. It might be a
better solution anyway?

By the way, I had thoughts about loading the library vs just docs. I reason,
that, if I look at some symbol docs via describe-* functions, I am
probably already using that library/package, or am about to use it, so it
probably is going to be loaded anyway. At least in many cases, not always of
course.

> But perhaps just loading the file automatically would be fine anyway --
> it's a bug for libraries to side-effect to an annoying degree upon load.

Helpful does it and a lot of people seems to be happy with it, but it does not
mean it is the correct thing to do, just often acceptable, or overlooked.

Built-in help does it too in some cases too, I think just for keymaps, I am not
sure there.

Re: Patch: perform autoloading when docs is missing from autoload object

[Stefan Monnier]

> Indeed. My thoughts were to read .elc when avialable,

I'm not sure you're familiar with the way it works for functions, but
basically the docstring is replaced by a "pointer" that's a cons cell
that combines the name of the file from which the function was loaded
together with an integer indicating at which byte-offset the docstring
can be found (the filename string is the same string used in all the
docstring of that file as well as the string used in `load-history`, in
terms of memory cost, it only really costs a single con-cell).
So there'd be no need for "when available" heuristic.

And all the code is already there to do the same for autoloads.
All it takes is to start compiling the autoloads files.


        Stefan

Generic autoloading? [Was Patch: perform autoloading when docs is missing from autoload object]

[Qiantan Hong]

Just in case, is it possible to provide a generic autoload mechanism
that run arbitrary function to load the missing function, 
instead of hardcoded to basically a REQUIRE?

This can be very helpful for my object capabilities implementation
because it can be used to support importing object graph more lazily.
(Instead of needing to traverse a full reference closure of a function,
we can stop at arbitrary function and resume traversal only when
that function is called).

Currently I can fake it with a closure that knows where it is
and replace itself once called.
But it feels like reimplementing the same thing twice.
(I think current autoload can also be implemented in pure Elisp
in this way, but we already have autoload object, so why not generalize it a bit).

Best,
Qiantan

Re: Generic autoloading? [Was Patch: perform autoloading when docs is missing from autoload object]

[Stefan Monnier]

> Just in case, is it possible to provide a generic autoload mechanism
> that run arbitrary function to load the missing function, 
> instead of hardcoded to basically a REQUIRE?

I think if it doesn't end up loading a file, and isn't bound to
the `symbol-function` cell of a symbol, then it's probably too different
from our autoloads to be worth trying to shoehorn it into the existing system.

> Currently I can fake it with a closure that knows where it is
> and replace itself once called.

I think the fix is to remove the word "fake" in the description ;-)

> But it feels like reimplementing the same thing twice.

I haven't looked at your code, but if what you fetch lazily is itself
a function, then I suspect that your "faking code" is clean, short, and
reliable, and there isn't much to share with the autoload code (other
than a philosophical similarity).

> (I think current autoload can also be implemented in pure Elisp
> in this way,

More or less, yes (tho it gets more costly in terms of heap usage).
Also, we have accrued various subtle details that are inconvenient to
mimic (e.g. various places call `autoload-do-load` manually rather than
just calling the function in order to get a bit more control over what
happens when).


        Stefan

Re: Generic autoloading? [Was Patch: perform autoloading when docs is missing from autoload object]

[Qiantan Hong]

> I haven't looked at your code, but if what you fetch lazily is itself
> a function, then I suspect that your "faking code" is clean, short, and
> reliable, and there isn't much to share with the autoload code (other
> than a philosophical similarity).

Thanks for the comment. I will probably do it that way then.
Also the prototype I last sent doesn’t do this yet, it just
demonstrate the basic idea, and is not a polished implementation.
Whether and how to do it lazily is a pure implementation issue.

The issue with the Elisp prototype is, it walks the object graph recursively,
and I suspect in practice it will occasionally use up stack space.
How can we work around this?

Re: Patch: perform autoloading when docs is missing from autoload object

[Arthur Miller]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Indeed. My thoughts were to read .elc when avialable,
>
> I'm not sure you're familiar with the way it works for functions, but
> basically the docstring is replaced by a "pointer" that's a cons cell
> that combines the name of the file from which the function was loaded
> together with an integer indicating at which byte-offset the docstring
> can be found (the filename string is the same string used in all the
> docstring of that file as well as the string used in `load-history`, in
> terms of memory cost, it only really costs a single con-cell).

Actually I was quite aware of that part, and how it works there. I looked at
that very explicitly. I am quite aware that it costs just an offset there, and I
don't have anything against that. I have no intention to do anything for that
case, it works as it is. I am not sure that we speak about the same thing.

I am talking about the case when there is a nil for doc, and library needs to be
autoloaded. When I ask for the help form describe-function, if there is a 'nil
in the autoload object itself, I would prefer not to load the entire library
just to load the symbol documentation for the use in help-mode buffer. Lars
expressed some concerns about loading library just for the doc purpose, so it
might be more appropriate in that case.

> So there'd be no need for "when available" heuristic.

Yes, in case you have that pointer, which I wasn't objecting to at all, but not
when you have a nil instead of that pointer. Or am I wrong there? 

> And all the code is already there to do the same for autoloads.
You mean to start generating those pointers instead of embedding the docs? It
would be an improvement indeed :). Still the case of hand-crafted code or code
generating but something else than supplied autoloading routines leaves for a
fully valid nil to be a part of the autoload object.

> All it takes is to start compiling the autoloads files.

Ok, but it does not seem that it works the way you say, at least not atm. This
is what I see in my Emacs, correct me if I do something wrong here.

I just tested with generated package-quickstart.el file. Below is what I get
when I M-: (symbol-function 'winum-select-window-0), as both before I
byte-compiled and after I byte-compiled:

(autoload "winum" "Jump to window 0.
If prefix ARG is given, delete the window instead of selecting it.

(fn &optional ARG)" t nil)

It is also fully possible that someone prefer to omit anything optional :):

#+begin_src emacs-lisp
(autoload 'academic-phrases "academic-phrases")
(autoload 'academic-phrases-by-section "academic-phrases")
(autoload 'ace-select-window "ace-window")
(autoload 'ace-delete-window "ace-window")
(autoload 'ace-swap-window "ace-window")
(autoload 'ace-delete-other-windows "ace-window")
(autoload 'ace-display-buffer "ace-window")
(autoload 'ace-window "ace-window")
(autoload 'ace-window-display-mode "ace-window")

( ... )
#+end_src

That is a fragment of what I use instead of quickstart file. I do compile entire
autoloads file together with my init.el, but I don't get any pointers to doc by
default. After compilining my own autoload.el, byte compiling it, loading and
evaling same (symbol-function 'winum-select-window-0):

Loading /home/arthur/.emacs.d/autoloads.elc...done
Mark set
(autoload "winum" nil nil nil)
Mark set

I thought maybe I needed to tell byte compiler that those autoloads are all
functions:

#+begin_src emacs-lisp
(autoload 'academic-phrases "academic-phrases"nil nil t)
(autoload 'academic-phrases-by-section "academic-phrases"nil nil t)
(autoload 'ace-select-window "ace-window"nil nil t)
(autoload 'ace-delete-window "ace-window"nil nil t)
(autoload 'ace-swap-window "ace-window"nil nil t)
(autoload 'ace-delete-other-windows "ace-window"nil nil t)
(autoload 'ace-display-buffer "ace-window"nil nil t)
(autoload 'ace-window "ace-window"nil nil t)
(autoload 'ace-window-display-mode "ace-window"nil nil t)

( ... )
#+end_src

After recompile and reload I get this:

(autoload "winum" nil nil t)

So the entire business of docs in autoloads is due to autoload file generation,
seems to me, if I don't misunderstanding something? I certainly have no problems
if autoloads generation is fixed and built-in autoloads file generator emitted
byte offsets instead of kilo bytes of fully embedded docs. But there is the case
that nil is a valid value for doc in autoload object, and thus I think it is
still important to fix help system, to fetch docs on demand.

Re: Generic autoloading? [Was Patch: perform autoloading when docs is missing from autoload object]

[Arthur Miller]

Qiantan Hong <qhong@mit.edu> writes:

> Just in case, is it possible to provide a generic autoload mechanism
> that run arbitrary function to load the missing function, 
> instead of hardcoded to basically a REQUIRE?
>
> This can be very helpful for my object capabilities implementation
> because it can be used to support importing object graph more lazily.
> (Instead of needing to traverse a full reference closure of a function,
> we can stop at arbitrary function and resume traversal only when
> that function is called).
>
> Currently I can fake it with a closure that knows where it is
> and replace itself once called.
That is what autoloads does in principle. Autoload is just a stub in funcion
slot of a symbol that loads the library where object is placed.
> But it feels like reimplementing the same thing twice.
Yes it sounds similar in essence, but you are doing this probably from memory,
autoloads are to save cpu time and speed by delay-loading from the disk "on
demand".

Re: Patch: perform autoloading when docs is missing from autoload object

[Lars Ingebrigtsen]

Arthur Miller <arthur.miller@live.com> writes:

> I wanted to later switch to just fetch the docs instead of performing
> load. Loading docs can be done without evaluating the code. It might be a
> better solution anyway?

If that's possible, that would be preferable, but as you say:

> By the way, I had thoughts about loading the library vs just docs. I reason,
> that, if I look at some symbol docs via describe-* functions, I am
> probably already using that library/package, or am about to use it, so it
> probably is going to be loaded anyway. At least in many cases, not always of
> course.

It is quite likely that you want to use the library if you've looked up
the symbol in a *Help* buffer, so why not, indeed, just load it at that
point?  If loading the library also helps with some other things --
highlighting `package-symbol's correctly in the doc string, for
instance.

So I think I'd be fine with adding the proposed functionality.  If there
are people that don't want this, we can add a user option to switch it off.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Patch: perform autoloading when docs is missing from autoload object

[Stefan Monnier]

> I just tested with generated package-quickstart.el file. Below is what I get
> when I M-: (symbol-function 'winum-select-window-0), as both before I
> byte-compiled and after I byte-compiled:

How did you perform the test?

I tested it on my end with the `loaddefs.el` file and "it worked", but
it required several tweaks (like removing the `no-byte-compile:t` from
the file-local variables, and telling `loadup.el` to load `loaddefs`
rather than `loaddefs.el`).

So I wouldn't be surprised if you need to take extra steps to convince
Emacs to compile the file and to load the compiled version of the file.

> So the entire business of docs in autoloads is due to autoload file generation,
> seems to me, if I don't misunderstanding something?

I'm afraid I don't know what you mean by "business of docs in autoloads".

> I certainly have no problems if autoloads generation is fixed and
> built-in autoloads file generator emitted byte offsets instead of kilo
> bytes of fully embedded docs.

Indeed, I think we should be possible to change `autoload.el` so that it
directly generates the "comment + byte-offsets" form in the .el files
so we don't even need to byte-compile them.  Not sure if it's a better
than byte-compiling them.

> But there is the case that nil is a valid value for doc in autoload
> object, and thus I think it is still important to fix help system, to
> fetch docs on demand.

That's true.  We could also just always use nil and always load the
source file to find the doc.  I'm not sure in that case it'd be worth
the trouble trying to avoid loading the whole file (I think it'd take
a fair bit of effort to make it work reliably enough).


        Stefan

RE: [External] : Re: Patch: perform autoloading when docs is missing from autoload object

[Drew Adams]

> It is quite likely that you want to use the
> library if you've looked up the symbol in a
> *Help* buffer, so why not, indeed, just load
> it at that point?

It's also quite likely - even more likely IMO
- that you do NOT want to use (load) the
library - you just wanted to check the doc of
something in it.

"Just load it" isn't right.  Let _users_ load
it when they want to.  That's trivial to do.

If you added an option that does such eager
loading, that would be one thing.  But to
"just load" unconditionally would be very
wrong, IMHO.

> So I think I'd be fine with adding the proposed
> functionality.  If there are people that don't
> want this, we can add a user option to switch it off.

That's wrong.  Add an option to switch it
on, if you want to provide such an optional
behavior.

Re: Patch: perform autoloading when docs is missing from autoload object

[Arthur Miller]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Arthur Miller <arthur.miller@live.com> writes:
>
>> I wanted to later switch to just fetch the docs instead of performing
>> load. Loading docs can be done without evaluating the code. It might be a
>> better solution anyway?
>
> If that's possible, that would be preferable, but as you say:
>
>> By the way, I had thoughts about loading the library vs just docs. I reason,
>> that, if I look at some symbol docs via describe-* functions, I am
>> probably already using that library/package, or am about to use it, so it
>> probably is going to be loaded anyway. At least in many cases, not always of
>> course.
>
> It is quite likely that you want to use the library if you've looked up
> the symbol in a *Help* buffer, so why not, indeed, just load it at that
> point?  If loading the library also helps with some other things --
> highlighting `package-symbol's correctly in the doc string, for
> instance.
>
> So I think I'd be fine with adding the proposed functionality.  If there
> are people that don't want this, we can add a user option to switch it off.

It is already an opt-in option.

(defcustom help-enable-symbol-autoload nil
  "Perform autoload when documentation is not present in autoload object."
  :type 'boolean
  :group 'help
  :version "28.1")
  
There is a defcustom variable that controls this, and it is nil by default, so
this is off unless user actively put it on.

It would be nice if it is in. Thank you.

Re: [External] : Re: Patch: perform autoloading when docs is missing from autoload object

[Arthur Miller]

Drew Adams <drew.adams@oracle.com> writes:

>> It is quite likely that you want to use the
>> library if you've looked up the symbol in a
>> *Help* buffer, so why not, indeed, just load
>> it at that point?
>
> It's also quite likely - even more likely IMO
> - that you do NOT want to use (load) the
> library - you just wanted to check the doc of
> something in it.
>
> "Just load it" isn't right.  Let _users_ load
> it when they want to.  That's trivial to do.
>
> If you added an option that does such eager
> loading, that would be one thing.  But to
> "just load" unconditionally would be very
> wrong, IMHO.
>
>> So I think I'd be fine with adding the proposed
>> functionality.  If there are people that don't
>> want this, we can add a user option to switch it off.
>
> That's wrong.  Add an option to switch it
> on, if you want to provide such an optional
> behavior.

:) Yes I agree with you.

That option is in the patch already, and the default is 'nil'. Haven't you
looked at it?

Everything is "as-before" until user actively toggles that option on:

(defcustom help-enable-symbol-autoload nil
  "Perform autoload when documentation is not present in autoload object."
  :type 'boolean
  :group 'help
  :version "28.1")

Re: Patch: perform autoloading when docs is missing from autoload object

[Lars Ingebrigtsen]

Arthur Miller <arthur.miller@live.com> writes:

> It is already an opt-in option.
>
> (defcustom help-enable-symbol-autoload nil
>   "Perform autoload when documentation is not present in autoload object."
>   :type 'boolean
>   :group 'help
>   :version "28.1")

I guess I should have read the patch.  :-)  I've now pushed it to Emacs
28 (with some changes).

It's really too late in the Emacs 28 cycle to add new features, but
since it defaults to nil, I think it should be OK.  It'll default to t
in Emacs 29.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Patch: perform autoloading when docs is missing from autoload object

[Arthur Miller]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> I just tested with generated package-quickstart.el file. Below is what I get
>> when I M-: (symbol-function 'winum-select-window-0), as both before I
>> byte-compiled and after I byte-compiled:
>
> How did you perform the test?

If I remember well, we have removed the no-byte-compile form the
package-quickstart.el a while ago. I just re-created new one, and I can confirm
that is no file-local variable that forbids it to be byte compiled. When I
checked yesterday the file was truly byte-compiled. I opened .elc file and
evaled it when I was writing the mail.
>
> I tested it on my end with the `loaddefs.el` file and "it worked", but
> it required several tweaks (like removing the `no-byte-compile:t` from
> the file-local variables, and telling `loadup.el` to load `loaddefs`
> rather than `loaddefs.el`).

I tested with package-quickstart. As said, manually opened .elc and evaled the
buffer.

> So I wouldn't be surprised if you need to take extra steps to convince
> Emacs to compile the file and to load the compiled version of the file.

Aha. Maybe. Than I am not sure how to do it.

By the way, when I was reading the manual, and testing all this, I could see
that autoloads from Emacs sources were with offsets in autoload object. But I
couldn't see those from installed packages. That is why I have finally ditched
the entire package-quickstart and Emacs autoload generation.

>> So the entire business of docs in autoloads is due to autoload file generation,
>> seems to me, if I don't misunderstanding something?
>
> I'm afraid I don't know what you mean by "business of docs in autoloads".

Eh; sorry, that was a goofy expression. I meant that documentation ends up
in autoload objects, because Emacs code that scrapes for autoloads explicitly put
it there. I am no 100% sure, that is just my assumption, I hoped you would say
yes or no :).

>> I certainly have no problems if autoloads generation is fixed and
>> built-in autoloads file generator emitted byte offsets instead of kilo
>> bytes of fully embedded docs.
>
> Indeed, I think we should be possible to change `autoload.el` so that it
> directly generates the "comment + byte-offsets" form in the .el files
> so we don't even need to byte-compile them.  Not sure if it's a better
> than byte-compiling them.

I am not either, it is a lot of data that gets pulled in when docs are
compiled. Maybe left it as a user option?

>> But there is the case that nil is a valid value for doc in autoload
>> object, and thus I think it is still important to fix help system, to
>> fetch docs on demand.
>
> That's true.  We could also just always use nil and always load the
> source file to find the doc.  I'm not sure in that case it'd be worth
> the trouble trying to avoid loading the whole file (I think it'd take
> a fair bit of effort to make it work reliably enough).

Rationale was that it is faster to just parsing with 'read' without loading
structures in the memory;

But I think that you are correct; it might not be worth the trouble,
since it would take quite some parsing effort to do it well, if it is even
possible. I think it could only be those symbols that are more or less
explicitly defined as defun, defmacro, defconst, defvar etc.

Indirect definitions are almost impossible to find but I don't think those are
recognized by completion either, so they are probably not possible to request
with describe-* functions.

I have some parsing code I wrote tonight to pull in source for symbols (instead
of built-ins I used yesterday, as seen in patched help-mode.el in the bug report
I filed). What I have noticed is that I can't request from Emacs to give me docs for
'package-desc' which is a cl-defstruct in package.el, because completion does
not seem to recoginze it as a symbol. But maybe I am wrong about that one, I am
not sure there.

Re: Patch: perform autoloading when docs is missing from autoload object

[Arthur Miller]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Arthur Miller <arthur.miller@live.com> writes:
>
>> It is already an opt-in option.
>>
>> (defcustom help-enable-symbol-autoload nil
>>   "Perform autoload when documentation is not present in autoload object."
>>   :type 'boolean
>>   :group 'help
>>   :version "28.1")
>
> I guess I should have read the patch.  :-)  I've now pushed it to Emacs
> 28 (with some changes).
>
> It's really too late in the Emacs 28 cycle to add new features, but
> since it defaults to nil, I think it should be OK.  It'll default to t
> in Emacs 29.

Thanks for accepting it!

Re: Patch: perform autoloading when docs is missing from autoload object

[Stefan Monnier]

> I tested with package-quickstart. As said, manually opened .elc and
> evaled the buffer.

Hmm... Can't see an obvious reason why it didn't do "the right thing"
for you (tho `M-x eval-buffer` will result in an unusable (FILE . OFFSET)
pair because the `load-file-name` var is not setup; a better test is to
load the file, but that doesn't explain why you ended up with an actual
doc *string*).

> By the way, when I was reading the manual, and testing all this, I could see
> that autoloads from Emacs sources were with offsets in autoload object.

Yes, this a special case: they're offsets into the `etc/DOC` file,
generated by the `lib-src/make-docfile.c` auxiliary tool.
This so as to bring the cost down further: instead of (FILE . OFFSET) we
can use just OFFSET where FILE is known implicitly to be `etc/DOC`.

> But I couldn't see those from installed packages. That is why I have
> finally ditched the entire package-quickstart and Emacs
> autoload generation.

Looking at the code, everything seems to be in place such that
byte-compiling the `package-quickstart.el` file *should* end up with
(FILE . OFFSET)s, but I haven't tested it to confirm yet.
You say it doesn't work, so apparently there's something somewhere that
still gets in the way :-(

>>> So the entire business of docs in autoloads is due to autoload file generation,
>>> seems to me, if I don't misunderstanding something?
>> I'm afraid I don't know what you mean by "business of docs in autoloads".
> Eh; sorry, that was a goofy expression. I meant that documentation ends up
> in autoload objects, because Emacs code that scrapes for autoloads explicitly put
> it there. I am no 100% sure, that is just my assumption, I hoped you would say
> yes or no :).

Ah, well then "yes" ;-)

> Rationale was that it is faster to just parsing with 'read' without loading
> structures in the memory;

Most Elisp files can be loaded fairly quickly (some exceptions are
things like `org.el` that pull in a lot of dependencies) and we'd only do
it once per session an only in some particular cases (not in loops,
etc...), so it's not that terribly important.


        Stefan

Re: Patch: perform autoloading when docs is missing from autoload object

[Arthur Miller]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> I tested with package-quickstart. As said, manually opened .elc and
>> evaled the buffer.
>
> Hmm... Can't see an obvious reason why it didn't do "the right thing"
> for you (tho `M-x eval-buffer` will result in an unusable (FILE . OFFSET)
> pair because the `load-file-name` var is not setup; a better test is to
> load the file, but that doesn't explain why you ended up with an actual
> doc *string*).

Sorry for bit late answer, I was little busy with playing with help-buffer, so
wasn't have so much time for anything else. I wanted to try this with Emac -Q to
be sure it's not my init file.

I did as you describe, and just loaded the file in Emacs -Q, but it still puts
in embedded docs. I am not sure, I have just been looking now around, without
actually exectuing anything, but I think that the actual data is emitted in

(defun autoload-print-form (form)) in autoload.el, and I don't see there
anything about offsets.

But I am not sure about it, to be honest; was just quick look at the pipleine. I
have tried to trace package-quicstart-refresh calls, but I didn't come out more
informed than I was before, so I will leave it for the moment.

>> By the way, when I was reading the manual, and testing all this, I could see
>> that autoloads from Emacs sources were with offsets in autoload object.
>
> Yes, this a special case: they're offsets into the `etc/DOC` file,
> generated by the `lib-src/make-docfile.c` auxiliary tool.
> This so as to bring the cost down further: instead of (FILE . OFFSET) we
> can use just OFFSET where FILE is known implicitly to be `etc/DOC`.

Cool, thank you for the info. Can't generated autoloads use the same trick for
every file? I reason that every .elc file installe din elpa is known at compile
time and load time as well. Since file itself is recorded in autoload object,
can't the same be applied? Or are there more details in the play? 

>> But I couldn't see those from installed packages. That is why I have
>> finally ditched the entire package-quickstart and Emacs
>> autoload generation.
>
> Looking at the code, everything seems to be in place such that
> byte-compiling the `package-quickstart.el` file *should* end up with
> (FILE . OFFSET)s, but I haven't tested it to confirm yet.
> You say it doesn't work, so apparently there's something somewhere that
> still gets in the way :-(

As mentioned above, I just tried with Emacs -Q and doing as you described, but I
just see embedded docs. I am not trolling or anything, I'll be glad to help to
find why is it so if I can.

>>>> So the entire business of docs in autoloads is due to autoload file generation,
>>>> seems to me, if I don't misunderstanding something?
>>> I'm afraid I don't know what you mean by "business of docs in autoloads".
>> Eh; sorry, that was a goofy expression. I meant that documentation ends up
>> in autoload objects, because Emacs code that scrapes for autoloads explicitly put
>> it there. I am no 100% sure, that is just my assumption, I hoped you would say
>> yes or no :).
>
> Ah, well then "yes" ;-)
>
>> Rationale was that it is faster to just parsing with 'read' without loading
>> structures in the memory;
>
> Most Elisp files can be loaded fairly quickly (some exceptions are
> things like `org.el` that pull in a lot of dependencies) and we'd only do
> it once per session an only in some particular cases (not in loops,
> etc...), so it's not that terribly important.
>
Indeed require/provide mechanism is a good thing and I also think it is not that
important.

When you touch on loading files; may I ask, if you have time to anser; why
require/provide isn't automated in a sense, that it always provide a feature
equaly named as the file loaded?

When I was learning  about it, as I understand, a file can provide several
features, which not many elisp file actually do. It can also be used to name
feature differently from the file itself, which few actually do. It is all good,
flexibility is always good, but for if I would like to check if a file is
loaded, and I don't know which features it provides, if any, do I have any easy
way of knowing it? I can look at load-hist and serach in lists, but it is very
convenient to say, as an example, (require 'subr) and be confident that the file
won't be loaded if it is already loaded. 'load' loads files unconditionally,
right? It probably should since it is a low-level function I guess. Sorry if I
am asking too much, there is so much to learn about Emacs :).