Helpful in Emacs?

Helpful in Emacs?

[Arthur Miller]

Any chance of getting Helpful, https://github.com/Wilfred/helpful, into official
Emacs? In the contributor list I see some familiar names which already
contributed to Emas, so they have signed FSF papers, but also some unkown ones
too.

I would like to have it partly because it does more than built-in help, partly
because it would also be possible to speed up autoloads (and ugly quickstart
file). Since helpful consults sources upon request for docs, there is no need to
pack docs into autoloads, which on my system resulted with heavily slimmed
quickstart file too.

I remember that some time ago both helpful and whichkey were discussed on this
list, but I have no idea what was conclusion if there was any.

Re: Helpful in Emacs?

[Stefan Kangas]

Arthur Miller <arthur.miller@live.com> writes:

> Any chance of getting Helpful, https://github.com/Wilfred/helpful, into official
> Emacs? In the contributor list I see some familiar names which already
> contributed to Emas, so they have signed FSF papers, but also some unkown ones
> too.

FWIW, here's my take on this:

1. helpful.el has several useful features.  They should be evaluated
one by one with the aim to integrate them in a clean way into the
built-in help.el et al libraries.  Such an evaluation would likely
show that some features are well suited for the default help system.
Maybe others we would want as optional features.  Maybe other features
still are better left in an external package.

2. Doing the above job well would presumably leave little reason to
distribute helpful.el together with Emacs.

3. Before we can even start thinking of doing any of this, we will
need to sort the copyright assignments.  This to my mind means that
the next step to make meaningful progress here is to get the module
onto GNU ELPA.  Once that is done, points 1 and 2 are mostly "only" a
matter of copying in the code.

If you would like to work together with the helpful.el maintainers to
get the package onto GNU ELPA, that would be worthwhile and
appreciated, I think.

> I remember that some time ago both helpful and whichkey were discussed on this
> list, but I have no idea what was conclusion if there was any.

AFAIR, there was no conclusion, really.

Re: Helpful in Emacs?

[Arthur Miller]

Stefan Kangas <stefan@marxist.se> writes:

> Arthur Miller <arthur.miller@live.com> writes:
>
>> Any chance of getting Helpful, https://github.com/Wilfred/helpful, into official
>> Emacs? In the contributor list I see some familiar names which already
>> contributed to Emas, so they have signed FSF papers, but also some unkown ones
>> too.
>
> FWIW, here's my take on this:
>
> 1. helpful.el has several useful features.  They should be evaluated
> one by one with the aim to integrate them in a clean way into the
> built-in help.el et al libraries.  Such an evaluation would likely
> show that some features are well suited for the default help system.
> Maybe others we would want as optional features.  Maybe other features
> still are better left in an external package.
>
> 2. Doing the above job well would presumably leave little reason to
> distribute helpful.el together with Emacs.

A week or two ago I was actually looking at Emacs help code. I wanted to bring
in the source code as well as references into help lookup, but honestly, I would
rather prefer to just include helpful instead of re-implementing everything. If
the authors have signed the paperwork, I see no reason why not just include
it. Also original help lookup could be left as low-resource, faster solution for
people who prefer to spend less resources on help lookup, while helpful could be
enabled by a custom variable, something like show extended help or as a minor mode.

> 3. Before we can even start thinking of doing any of this, we will
> need to sort the copyright assignments.  This to my mind means that

Yes. That was my thought, I wanted to check is there is interest from emacs devs
and I have CC-ed Willfred because he is the only one that can answer the
paper-status I guess, and if there is interest from his and other helpful-devs
side to get this included.

> need to sort the copyright assignments.  This to my mind means that
> the next step to make meaningful progress here is to get the module
> onto GNU ELPA.  Once that is done, points 1 and 2 are mostly "only" a
> matter of copying in the code.
>
> If you would like to work together with the helpful.el maintainers to
> get the package onto GNU ELPA, that would be worthwhile and
> appreciated, I think.
I'll help with what I can of course, but I am probably not of much help here. I
have never contributed to Elpa myself :).

By the way, there are probably no problems of getting Helpful into "Nelpa"
(non-gnu elpa), and it already is in Melpa, so the access for new users is not
far away. I would though really like to see it as a complement to built-in help
out of the box.

Re: Helpful in Emacs?

[Daniel Martín]

Arthur Miller <arthur.miller@live.com> writes:

> A week or two ago I was actually looking at Emacs help code. I wanted to bring
> in the source code as well as references into help lookup, but honestly, I would
> rather prefer to just include helpful instead of re-implementing everything. If
> the authors have signed the paperwork, I see no reason why not just include
> it. Also original help lookup could be left as low-resource, faster solution for
> people who prefer to spend less resources on help lookup, while helpful could be
> enabled by a custom variable, something like show extended help or as
> a minor mode.

Instead of integrating the library as a whole, I'd prefer we extract
those features from Helpful that people think are useful and not already
available in Emacs.  I don't think that having two separate help systems
in core is a good idea, there'll be too much duplication and maintenance
burden.

Showing references to a symbol is an interesting feature (for example,
to learn how to use an ELisp API by looking at examples).  I see that
Helpful provides this feature via the separate package elisp-refs
(https://github.com/Wilfred/elisp-refs).  The closest package I know
that actually understands ELisp is el-search from ELPA, but elisp-refs
is a more specific package for the concrete use case of searching for
references.

If we can get copyright papers, perhaps we could start by integrating
the functionality of elisp-refs as an xref backend.  Then the help
system could use the new backend to provide a "References" link button.

Re: Helpful in Emacs?

[Arthur Miller]

Daniel Martín <mardani29@yahoo.es> writes:

> Arthur Miller <arthur.miller@live.com> writes:
>
>> A week or two ago I was actually looking at Emacs help code. I wanted to bring
>> in the source code as well as references into help lookup, but honestly, I would
>> rather prefer to just include helpful instead of re-implementing everything. If
>> the authors have signed the paperwork, I see no reason why not just include
>> it. Also original help lookup could be left as low-resource, faster solution for
>> people who prefer to spend less resources on help lookup, while helpful could be
>> enabled by a custom variable, something like show extended help or as
>> a minor mode.
>
> Instead of integrating the library as a whole, I'd prefer we extract
> those features from Helpful that people think are useful and not already
> available in Emacs.  I don't think that having two separate help systems
> in core is a good idea, there'll be too much duplication and maintenance
> burden.
I don't think there is much of duplication there. Those two libraries don't
share exactly same code, so extracting features is not as straightforward as one
might expect. At least it seem so to me. In principle, extracting stuff from
helpful, would be as easy as to re-implement helpful for Emacs in that
case. Maybe I am wrong, I am not very familiar with code bases for neither of
those, I just glanced over it a couple of weeks ago. But in general, I think
there are more fun places to hack on, than to reimplement somethign that already
works. Just bring it in :).

Anyway for extracting something from helpful, there is still need
for original author(s) to do the fsf paperwork, and to be willing to donate
their code to Emacs.

> Showing references to a symbol is an interesting feature (for example,
> to learn how to use an ELisp API by looking at examples).  I see that

When things don't go the way I expect them, I use helpful to look up the
function and see the source code.

> Helpful provides this feature via the separate package elisp-refs
> (https://github.com/Wilfred/elisp-refs).  The closest package I know
> that actually understands ELisp is el-search from ELPA, but elisp-refs
> is a more specific package for the concrete use case of searching for
> references.
I have't looked how it does, but I guess you are correct.

Re: Helpful in Emacs?

[Stefan Kangas]

Arthur Miller <arthur.miller@live.com> writes:

> Anyway for extracting something from helpful, there is still need
> for original author(s) to do the fsf paperwork, and to be willing to donate
> their code to Emacs.

Indeed, that's the best starting point.  Until we can get that sorted
out, the rest of this discussion is mostly hypothetical.

I hope the authors of helpful.el are interested.

Re: Helpful in Emacs?

[Arthur Miller]

Stefan Kangas <stefan@marxist.se> writes:

> Arthur Miller <arthur.miller@live.com> writes:
>
>> Anyway for extracting something from helpful, there is still need
>> for original author(s) to do the fsf paperwork, and to be willing to donate
>> their code to Emacs.
>
> Indeed, that's the best starting point.  Until we can get that sorted
> out, the rest of this discussion is mostly hypothetical.
>
> I hope the authors of helpful.el are interested.

After looking into the helpful code I realize it used s.el, dash.el and some
other libraries which are not part of Emacs, so I guess chances to get it into
Emacs are slim to none, since all those packages would also need inclusion. That
might also explain the silence from the author(s)?

So the alternative is to develop similar functionality ourselves into built-in help?

Re: Helpful in Emacs?

[Stefan Kangas]

Arthur Miller <arthur.miller@live.com> writes:

> After looking into the helpful code I realize it used s.el, dash.el and some
> other libraries which are not part of Emacs, so I guess chances to get it into
> Emacs are slim to none, since all those packages would also need inclusion.

From helpful.el:

;; Package-Requires: ((emacs "25") (dash "2.18.0") (s "1.11.0") (f
"0.20.0") (elisp-refs "1.2"))

IIUC, "dash", "s" and "f" are just convenience libraries that could be
replaced with built-in functions.  We have already added some things
to Emacs in recent years that have been inspired by those libraries,
and if there is something needed that is still missing, we could
consider adding that too.

I don't know what "elisp-refs" is or does, but it has the same
principal author as helpful.el.

So it seems to me that none of this is impossible to resolve, in one
way or another.

> So the alternative is to develop similar functionality ourselves into built-in help?

Until we have the copyright assignments sorted, this is indeed our only option.

Re: Helpful in Emacs?

[Arthur Miller]

Stefan Kangas <stefan@marxist.se> writes:

> Arthur Miller <arthur.miller@live.com> writes:
>
>> After looking into the helpful code I realize it used s.el, dash.el and some
>> other libraries which are not part of Emacs, so I guess chances to get it into
>> Emacs are slim to none, since all those packages would also need inclusion.
>
> From helpful.el:
>
> ;; Package-Requires: ((emacs "25") (dash "2.18.0") (s "1.11.0") (f
> "0.20.0") (elisp-refs "1.2"))
>
> IIUC, "dash", "s" and "f" are just convenience libraries that could be
> replaced with built-in functions.  We have already added some things
> to Emacs in recent years that have been inspired by those libraries,
> and if there is something needed that is still missing, we could
> consider adding that too.
>
> I don't know what "elisp-refs" is or does, but it has the same
> principal author as helpful.el.
>
> So it seems to me that none of this is impossible to resolve, in one
> way or another.
No, of course it is not impossible; I was aware of that when I wrote it, but
still, it means we have to re-work it. But someone else didn't want to have two
different code bases etc. 

What I have learned form this list, is, if I want something to have, I have to
build-it and submit patch, and not even then I am guaranteed to have it in
Emacs :-).

I also haen't seen any responses from Wifred, so I'll just see if I can
implement those things I care most for: lookup docs for autoladed functions
without docs in the stub, and not yet loaded, and code inlined in help buffer,
instead of just plain link. The former would make it possible to save some
space, for example docs does not need to be included in stubs, and can be
fetched on demand, and the latter is just plain nice to have. It saves time if I
can just scroll other (help) buffer and read the source without need ot click on
link in it to fetch the source code.

Re: Helpful in Emacs?

[Jean-Christophe Helary]

> On Sep 10, 2021, at 16:58, Arthur Miller <arthur.miller@live.com> wrote:
> 
> I also haen't seen any responses from Wifred, so I'll just see if I can
> implement those things I care most for:

The thread has just started a few hours ago...

-- 
Jean-Christophe Helary @brandelune
https://mac4translators.blogspot.com
https://sr.ht/~brandelune/omegat-as-a-book/

Re: Helpful in Emacs?

[Stefan Monnier]

> IIUC, "dash", "s" and "f" are just convenience libraries that could be
> replaced with built-in functions.

Here's a monthly reminder that `dash` is in GNU ELPA.


        Stefan

Re: Helpful in Emacs?

[Eli Zaretskii]

> From: Arthur Miller <arthur.miller@live.com>
> Date: Fri, 10 Sep 2021 09:11:19 +0200
> Cc: me@wilfred.me.uk, Emacs developers <emacs-devel@gnu.org>,
>  Daniel Martín <mardani29@yahoo.es>
> 
> So the alternative is to develop similar functionality ourselves into built-in help?

Yes, but we need first to establish which features we want to have,
which ones are opt-in and which opt-out, etc.

Re: Helpful in Emacs?

[Arthur Miller]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Arthur Miller <arthur.miller@live.com>
>> Date: Fri, 10 Sep 2021 09:11:19 +0200
>> Cc: me@wilfred.me.uk, Emacs developers <emacs-devel@gnu.org>,
>>  Daniel Martín <mardani29@yahoo.es>
>> 
>> So the alternative is to develop similar functionality ourselves into built-in help?
>
> Yes, but we need first to establish which features we want to have,
> which ones are opt-in and which opt-out, etc.

I just answered to Stefan K:

What I care most for at the moment are: doc lookup for autoladed functions
without the doc in the stub, and not yet loaded; and code inlined in help buffer,
instead of just plain link. The former would make it possible to save some
space, for example docs does not need to be included in stubs, and can be
fetched on demand. For example package-quickstart.el wouldn't need to pack 800
kb of docs when some of those will never be requested anyway.

The latter is just plain nice to have. It saves time if I can just scroll other
(help) buffer and read the source without need ot click on link in it to fetch
the source code.

Re: Helpful in Emacs?

[Eli Zaretskii]

> From: Arthur Miller <arthur.miller@live.com>
> Cc: stefan@marxist.se,  me@wilfred.me.uk,  emacs-devel@gnu.org,
>   mardani29@yahoo.es
> Date: Fri, 10 Sep 2021 10:00:47 +0200
> 
> What I care most for at the moment are: doc lookup for autoladed functions
> without the doc in the stub, and not yet loaded; and code inlined in help buffer,
> instead of just plain link. The former would make it possible to save some
> space, for example docs does not need to be included in stubs, and can be
> fetched on demand. For example package-quickstart.el wouldn't need to pack 800
> kb of docs when some of those will never be requested anyway.

If this means asking for help on an autoloaded function will load its
package, I don't think it should be the default.  We could perhaps
arrange for a button that would load the package and display the full
documentation, but that's all.

> The latter is just plain nice to have. It saves time if I can just scroll other
> (help) buffer and read the source without need ot click on link in it to fetch
> the source code.

Again, displaying the source by default is too much, IMNSHO: the code
of a function could be very large.  Doing this by default also assumes
that the user will necessarily want to look at the source, something
that is IME questionable at best.  So this should also be optional,
and even then the code should be changed to avoid showing a huge
*Help* window (since currently, we fit the window size to the text in
the *Help* buffer).

Re: Helpful in Emacs?

[Arthur Miller]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Arthur Miller <arthur.miller@live.com>
>> Cc: stefan@marxist.se,  me@wilfred.me.uk,  emacs-devel@gnu.org,
>>   mardani29@yahoo.es
>> Date: Fri, 10 Sep 2021 10:00:47 +0200
>> 
>> What I care most for at the moment are: doc lookup for autoladed functions
>> without the doc in the stub, and not yet loaded; and code inlined in help buffer,
>> instead of just plain link. The former would make it possible to save some
>> space, for example docs does not need to be included in stubs, and can be
>> fetched on demand. For example package-quickstart.el wouldn't need to pack 800
>> kb of docs when some of those will never be requested anyway.
>
> If this means asking for help on an autoloaded function will load its
> package, I don't think it should be the default.  We could perhaps
No no, not loading the package. Just docs. That can be arranged, right? Since we
are poking in sources anyway? :). Of course it would not be by default. We know
that all new options are comming with a defcustom to turn them on, since thay
are off by default :).

No, it is (relatively) cpu intensive, especially if user run with compressed
sources. User could have also opted out sources, so it can't be a default
option at all.

I remember a discussion with someone on Reddit who complained that helpful is
slow for him, and I forgott in that discussion that I am running directly from
the source dir with uncompressed sources.

> arrange for a button that would load the package and display the full
> documentation, but that's all.
>
>> The latter is just plain nice to have. It saves time if I can just scroll other
>> (help) buffer and read the source without need ot click on link in it to fetch
>> the source code.
>
> Again, displaying the source by default is too much, IMNSHO: the code
> of a function could be very large.  Doing this by default also assumes
> that the user will necessarily want to look at the source, something
> that is IME questionable at best.  So this should also be optional,
> and even then the code should be changed to avoid showing a huge
> *Help* window (since currently, we fit the window size to the text in
> the *Help* buffer).

Exactly, and same here, not by default but as an opt-in option.

Re: Helpful in Emacs?

[Eduardo Ochs]

On Thu, 9 Sept 2021 at 12:22, Daniel Martín <mardani29@yahoo.es> wrote:
>
> Arthur Miller <arthur.miller@live.com> writes:
>
> > A week or two ago I was actually looking at Emacs help code. I wanted to bring
> > in the source code as well as references into help lookup, but honestly, I would
> > rather prefer to just include helpful instead of re-implementing everything. If
> > the authors have signed the paperwork, I see no reason why not just include
> > it. Also original help lookup could be left as low-resource, faster solution for
> > people who prefer to spend less resources on help lookup, while helpful could be
> > enabled by a custom variable, something like show extended help or as
> > a minor mode.
>
> Instead of integrating the library as a whole, I'd prefer we extract
> those features from Helpful that people think are useful and not already
> available in Emacs.  I don't think that having two separate help systems
> in core is a good idea, there'll be too much duplication and maintenance
> burden.
>
> Showing references to a symbol is an interesting feature (for example,
> to learn how to use an ELisp API by looking at examples).  I see that
> Helpful provides this feature via the separate package elisp-refs
> (https://github.com/Wilfred/elisp-refs).  The closest package I know
> that actually understands ELisp is el-search from ELPA, but elisp-refs
> is a more specific package for the concrete use case of searching for
> references.
>
> If we can get copyright papers, perhaps we could start by integrating
> the functionality of elisp-refs as an xref backend.  Then the help
> system could use the new backend to provide a "References" link button.


Another vote for "extracting features"/"refactoring helpful" here...

I tried to add support for helpful to eev about a year ago and I got
lost and stuck - I only had a few hours to play with it, and I was not
super focused, but was quite frustrated at the end. I was hoping that
helpful would be structured as a library plus a front-end, and it was
not. I'll try again this weekend. =/

  Cheers,
    Eduardo Ochs
    http://angg.twu.net/#eev

Re: Helpful in Emacs?

[Arthur Miller]

Eduardo Ochs <eduardoochs@gmail.com> writes:

> On Thu, 9 Sept 2021 at 12:22, Daniel Martín <mardani29@yahoo.es> wrote:
>>
>> Arthur Miller <arthur.miller@live.com> writes:
>>
>> > A week or two ago I was actually looking at Emacs help code. I wanted to bring
>> > in the source code as well as references into help lookup, but honestly, I would
>> > rather prefer to just include helpful instead of re-implementing everything. If
>> > the authors have signed the paperwork, I see no reason why not just include
>> > it. Also original help lookup could be left as low-resource, faster solution for
>> > people who prefer to spend less resources on help lookup, while helpful could be
>> > enabled by a custom variable, something like show extended help or as
>> > a minor mode.
>>
>> Instead of integrating the library as a whole, I'd prefer we extract
>> those features from Helpful that people think are useful and not already
>> available in Emacs.  I don't think that having two separate help systems
>> in core is a good idea, there'll be too much duplication and maintenance
>> burden.
>>
>> Showing references to a symbol is an interesting feature (for example,
>> to learn how to use an ELisp API by looking at examples).  I see that
>> Helpful provides this feature via the separate package elisp-refs
>> (https://github.com/Wilfred/elisp-refs).  The closest package I know
>> that actually understands ELisp is el-search from ELPA, but elisp-refs
>> is a more specific package for the concrete use case of searching for
>> references.
>>
>> If we can get copyright papers, perhaps we could start by integrating
>> the functionality of elisp-refs as an xref backend.  Then the help
>> system could use the new backend to provide a "References" link button.
>
>
> Another vote for "extracting features"/"refactoring helpful" here...
>
> I tried to add support for helpful to eev about a year ago and I got
> lost and stuck - I only had a few hours to play with it, and I was not
> super focused, but was quite frustrated at the end. I was hoping that
> helpful would be structured as a library plus a front-end, and it was
> not.

Of course, it is not a library, it is an application :). That is why I say it is
probably as much work to extract stuff out of it as to rebuild it. Even faster
is just to include it as is and offer a defcustom to switch between old built-in
help and helpful. And once again, if you don't get helpful authors permission,
you can not just "extract", you will have to rebuild it from scratch. It might
not be so difficult to do it, but I would prefer to do some more fun stuff than
to implement funcitonality that already exists if we can just include it. Just
my 2 cents as they say.

Re: Helpful in Emacs?

[Arthur Miller]

Daniel Martín <mardani29@yahoo.es> writes:

> Showing references to a symbol is an interesting feature (for example,
> to learn how to use an ELisp API by looking at examples).  I see that
> Helpful provides this feature via the separate package elisp-refs
> (https://github.com/Wilfred/elisp-refs).  The closest package I know
> that actually understands ELisp is el-search from ELPA, but elisp-refs
> is a more specific package for the concrete use case of searching for
> references.

About elisp-refs, I have looked at that library this evening, after your
mail, and I think it does the right thing.

It uses read function to parse lisp files to find defs. I wrote a blog post just
yesterday about doing similar, but it was about collecting autoloads since I am
unhappy with built-in funcionality in autoload.el. I don't use
package-quickstart.el but build my own autoloads file that suits my use pattern
better. This request for helpful in Emacs started as I realized that helpful
lookups functions in source code rather than using docs stuffed in autoloads, so
I can have the cake and eat it too. Built in help can't show docs for a
autoloaded function that is not loaded yet.

My blog post is here: http://www.nextpoint.se/?p=890 (if you are interested and
have time :))

What do you need to build  xref backend you mention? As mentioned on elisp-refs
gh page:

"xref-find-references: This command is included in Emacs 25.1, but it's based on
a text search. It is confused by comments and strings, and cannot distinguish
between functions and variables."

Would rebuilding this command to use read instead of text search do the thing?

I have never looked at xref code, so I have no idea how much work it is to
re-build it though.

Otherwise, about the read function and reading stuff from lisp files, two days
ago I have built a small program to extract all defined symbols (defvars,
defcustoms, defuns and defmacros) from all lisp files in a directory tree, and I
save them in above mentioned database. I thought it would be slow, but it really
is fast. All symbols in Emacs lisp subdirectory are indexed in like 2 seconds
and my entire elpa dir with ~200+ packages takes about 0.5 secs. 

Re: Helpful in Emacs?

[Dmitry Gutov]

On 09.09.2021 23:23, Arthur Miller wrote:
> What do you need to build  xref backend you mention? As mentioned on elisp-refs
> gh page:
> 
> "xref-find-references: This command is included in Emacs 25.1, but it's based on
> a text search. It is confused by comments and strings, and cannot distinguish
> between functions and variables."
> 
> Would rebuilding this command to use read instead of text search do the thing?

You can easily plug another search implementation in 
xref-find-references. You can even make summaries multiline (though that 
would feel weird to me).

But is elisp-refs that much better at searching? Distinguishing 
variables from functions sounds nice, but then you might want to rename 
the function -- and you'll probably want to rename the variable with the 
same name as well.

Same about comments: if elisp-refs skips comments altogether (rather 
than smartly parsing references inside), you might miss some references 
to your function when doing a rename.

I also wonder how it compares in performance.

Re: Helpful in Emacs?

[Arthur Miller]

[-- Attachment #1: Type: text/plain, Size: 2742 bytes --]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 09.09.2021 23:23, Arthur Miller wrote:
>> What do you need to build  xref backend you mention? As mentioned on elisp-refs
>> gh page:
>> "xref-find-references: This command is included in Emacs 25.1, but it's based
>> on
>> a text search. It is confused by comments and strings, and cannot distinguish
>> between functions and variables."
>> Would rebuilding this command to use read instead of text search do the thing?
>
> You can easily plug another search implementation in xref-find-references. You
> can even make summaries multiline (though that would feel weird to me).
Ok, that sounds good.

> But is elisp-refs that much better at searching? Distinguishing variables from
> functions sounds nice, but then you might want to rename the function -- and
> you'll probably want to rename the variable with the same name as well.

I can't really tell you since I am not an expert. The quote was from the
elisp-refs project. Read does have semantic information about symbols readily
avialable, plain string searching does not have it immidiately, but strings can
always be interned and looked up, so I really don't know how it compares to
xref. I do believe that the author is correct in what it says, otherwise I don't
think he would be doing that work. Observe also that the comment is old, a
couple of years or more.

> Same about comments: if elisp-refs skips comments altogether (rather than
> smartly parsing references inside), you might miss some references to your
> function when doing a rename.

> I also wonder how it compares in performance.

I haven't benchmarked or done any deeper investigations; I haven't even had time
to look at his code yet; but I don't think the performance would be much
different than just search through buffer. Read might be somewhat faster since
it isn't as general regex matcher as regex-search is I guess, but that would be
just my wild guessing, I don't think I am qualified to speak much about
performance here without benchmarking.

However what I have experienced, in my own little experiment, is that searching for
all symbol definitions in entire Emacs lisp directory went fast on my computer,
~2 seconds, with flushing the database to the disk. Now I don't do much, I just
collect all defuns, defmacros, defvars and defcustoms, so it is not terribly
lots of work, finding references is probably much more work, but at least as an
experiment it indicates that it could be very fast. I attached the code for the
reference.

(benchmark-run (ff-build-emacs-db))

(2.037053179 0 0.0)

Now I don't know how fast is xref-find-references nor how it does what it does
yet either, so lots here is just speculations, most of them probably wrong :). 


[-- Attachment #2: func-freq.el --]
[-- Type: text/plain, Size: 3258 bytes --]

;;; func-freq.el ---                                 -*- lexical-binding: t; -*-

;; Copyright (C) 2021  Arthur Miller

;; Author: Arthur Miller <arthur.miller@live.com>
;; Keywords: 

;; This program is free software; you can redistribute it and/or modify
;; it under the terms of the GNU General Public License as published by
;; the Free Software Foundation, either version 3 of the License, or
;; (at your option) any later version.

;; This program is distributed in the hope that it will be useful,
;; but WITHOUT ANY WARRANTY; without even the implied warranty of
;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
;; GNU General Public License for more details.

;; You should have received a copy of the GNU General Public License
;; along with this program.  If not, see <https://www.gnu.org/licenses/>.

;;; Commentary:

;; 

;;; Code:
(defun ff-print (map)
  (maphash (lambda (k v) (print (format "%s: %s" k v))) map))

(defvar ff-srcs nil
  "Cash list of source files.")
(defvar ff-lmap nil
  "Load paths")
(defvar ff-pmap nil
  "Provided features")
(defvar ff-vmap nil
  "Variables")
(defvar ff-fmap nil
  "Functions and macros")

(defun ff-save-db (db-file)
  (with-temp-file db-file
    (prin1 ff-lmap (current-buffer))
    (prin1 ff-pmap (current-buffer))
    (prin1 ff-vmap (current-buffer))
    (prin1 ff-fmap (current-buffer))))

(defun ff-read-db (db-file)
  (with-temp-buffer
    (insert-file-contents db-file)
    (goto-char (point-min))
    (setq ff-lmap (read (current-buffer))
          ff-pmap (read (current-buffer))
          ff-vmap (read (current-buffer))
          ff-fmap (read (current-buffer)))))

(defun ff-collect-features (src index)
  (let (sxp)
    (with-current-buffer (get-buffer-create "*ff-buffer*")
      (erase-buffer)
      (insert-file-contents src)
      (goto-char (point-min))
      (while (setq sxp (ignore-errors (read (current-buffer))))
        (when (listp sxp)
          (cond ((or (equal (car sxp) 'defun)
                     (equal (car sxp) 'defmacro))
                 (puthash (cadr sxp) index ff-fmap))
                ((or (equal (car sxp) 'defvar)
                     (equal (car sxp) 'defcustom))
                 (puthash (cadr sxp) index ff-vmap))
                ((equal (car sxp) 'provide)
                 (puthash (cadr sxp) index ff-pmap))))))))

(defun ff-build-db (dir-tree)
  ;;(when (or (not ff-srcs) (equal (car ff-srcs) dir-tree))
      (setq ff-srcs (cons dir-tree (directory-files-recursively dir-tree "\\.el$"))
            ff-lmap (make-hash-table :test 'equal)
            ff-vmap (make-hash-table :test 'equal)
            ff-fmap (make-hash-table :test 'equal)
            ff-pmap (make-hash-table :test 'equal))
      ;; )
  (let ((index 0))
    (dolist (src (cdr ff-srcs))
      (puthash index src ff-lmap)
      (ff-collect-features src index)
      (setq index (1+ index)))))

(defun ff-build-emacs-db ()
  (ff-build-db (expand-file-name "lisp/" source-directory))
  (ff-save-db (expand-file-name "ff-db-emacs" user-emacs-directory)))

(defun ff-build-package-db ()
  (ff-build-db (expand-file-name "elpa/" user-emacs-directory))
  (ff-save-db (expand-file-name "ff-db-packages" user-emacs-directory)))

(provide 'func-freq)
;;; func-freq.el ends here

Re: Helpful in Emacs?

[Eli Zaretskii]

> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Fri, 10 Sep 2021 01:55:08 +0300
> Cc: me@wilfred.me.uk, Stefan Kangas <stefan@marxist.se>,
>  Emacs developers <emacs-devel@gnu.org>
> 
> Distinguishing variables from functions sounds nice

I wonder how it works.  If there's a function and a variable by the
same name, how does the user tell references to which one he/she wants
to see?  Does the package have 2 different commands?  Or does it ask?
And what about looking for faces -- are those variables or functions?

The commands we have in Xref look for _identifiers_.  An identifier is
a generalization of all those different syntactic and semantic
elements: it can be a variable, a function, a structure, etc.  That
IMO is a much more convenient UI: a single command with a very short
key binding for finding any identifier.  YMMV, of course.

Re: Helpful in Emacs?

[Eli Zaretskii]

> From: Arthur Miller <arthur.miller@live.com>
> Date: Thu, 09 Sep 2021 22:23:08 +0200
> Cc: me@wilfred.me.uk, Stefan Kangas <stefan@marxist.se>,
>  Emacs developers <emacs-devel@gnu.org>
> 
> "xref-find-references: This command is included in Emacs 25.1, but it's based on
> a text search. It is confused by comments and strings, and cannot distinguish
> between functions and variables."

This is inaccurate and misleading.  It might be true for the default
back-end (which AFAIR is Grep), but certainly not for all of them.
What exactly is a "reference" is determined by the back-end search
tool, and some of them understand quite a lot of the programming
language of the source files.

So please correct that blog to avoid misleading people.

Re: Helpful in Emacs?

[Arthur Miller]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Arthur Miller <arthur.miller@live.com>
>> Date: Thu, 09 Sep 2021 22:23:08 +0200
>> Cc: me@wilfred.me.uk, Stefan Kangas <stefan@marxist.se>,
>>  Emacs developers <emacs-devel@gnu.org>
>> 
>> "xref-find-references: This command is included in Emacs 25.1, but it's based on
>> a text search. It is confused by comments and strings, and cannot distinguish
>> between functions and variables."
>
> This is inaccurate and misleading.  It might be true for the default
> back-end (which AFAIR is Grep), but certainly not for all of them.
> What exactly is a "reference" is determined by the back-end search
> tool, and some of them understand quite a lot of the programming
> language of the source files.
>
> So please correct that blog to avoid misleading people.

That is not my blog, that is quoated from the elisp-refs package:

https://github.com/Wilfred/elisp-refs

I made it clear it was a quote the first time I quoted, but it seems that has
lost in further quoting of my original message.

The original author is CC:ed, so hopefully he sees this.

Re: Helpful in Emacs?

[Juri Linkov]

> Instead of integrating the library as a whole, I'd prefer we extract
> those features from Helpful that people think are useful and not already
> available in Emacs.  I don't think that having two separate help systems
> in core is a good idea, there'll be too much duplication and maintenance
> burden.

Refactoring it as a library is very much needed.  Ideally, it should return
information in a structured way, for example:

  ((type . function)
   (name . "car")
   (source-file . "src/data.c")
   (Info-node . "(elisp) List Elements")
   (docstring . ...)
   ...)

This will allow to use the extracted information in different front-ends
such as a html front-end to display Help in eww, etc.  I once attempted
to represent such information as a generated Info node, but abandoned
due to too many small details that are too ambiguous to handle.

If it's possible to refactor helpful.el and separate a library from front-end,
it would be very helpful and will solve such bugs as bug#49587 that currently
displays the Help buffer twice: for the function and the variable part
separately when a symbol for 'C-h o' is both a function and a variable.