tags for functions

tags for functions

[Ted Zlatanov]

Ted Z wrote in comp.emacs and gnu.emacs.help:

> On Sat, 17 Jan 2009 10:29:45 -0800 (PST) Xah Lee <xah...@gmail.com> wrote:
>
> XL> Note that listing related functions in a function's doc is in many
> XL> programing manuals. e.g Mathematica, MS's JScript, PHP ... they are
> XL> quite useful. Because for those not expert yet of a lang (which is
> XL> majority), often they do not know similar functions or do not know if
> XL> there's manual section that list such, and often are confused about
> XL> the differences of many functions that seems the same ....
>
> I agree this would be useful. It's best done with tags IMO, rather
> than explicitly listing the related functions. For example, motion
> commands should be tagged "motion" and then every command with that
> tag can automatically list every motion command. The key is that the
> extra work is in classification, not in tediously listing every
> command's peers.
>
> Tags I could use: motion, file, coding-system, menu, buffer, process
>
> Each package should probably tag its commands with the package name.
>
> Short tags are not always descriptive enough, but long tags get
> unpleasantly verbose so the real art is in balancing between the two.
>
> Anything more hierarchical than tags is painful to manage in the long run.

A tangential discussion about the inconsistent naming of motion commands
led me to the proposal above.

I think it would be a nice addition to Emacs.  I did a search and didn't
find prior relevant discussions.

Every package with a comment stating Keywords: at the beginning could
automatically give those keywords as tags to its functions.  That would
probably be 50% or less of the total needed tags, but it's an easy
start.

Ted

Re: tags for functions

[Lennart Borgman]

On Tue, Jan 20, 2009 at 9:36 PM, Ted Zlatanov <tzz@lifelogs.com> wrote:
> Ted Z wrote in comp.emacs and gnu.emacs.help:
>
>> On Sat, 17 Jan 2009 10:29:45 -0800 (PST) Xah Lee <xah...@gmail.com> wrote:
>>
>> XL> Note that listing related functions in a function's doc is in many
>> XL> programing manuals. e.g Mathematica, MS's JScript, PHP ... they are
>> XL> quite useful. Because for those not expert yet of a lang (which is
>> XL> majority), often they do not know similar functions or do not know if
>> XL> there's manual section that list such, and often are confused about
>> XL> the differences of many functions that seems the same ....
>>
>> I agree this would be useful. It's best done with tags IMO, rather
>> than explicitly listing the related functions. For example, motion
>> commands should be tagged "motion" and then every command with that
>> tag can automatically list every motion command. The key is that the
>> extra work is in classification, not in tediously listing every
>> command's peers.
>>
>> Tags I could use: motion, file, coding-system, menu, buffer, process
>>
>> Each package should probably tag its commands with the package name.
>>
>> Short tags are not always descriptive enough, but long tags get
>> unpleasantly verbose so the real art is in balancing between the two.
>>
>> Anything more hierarchical than tags is painful to manage in the long run.
>
> A tangential discussion about the inconsistent naming of motion commands
> led me to the proposal above.
>
> I think it would be a nice addition to Emacs.  I did a search and didn't
> find prior relevant discussions.
>
> Every package with a comment stating Keywords: at the beginning could
> automatically give those keywords as tags to its functions.  That would
> probably be 50% or less of the total needed tags, but it's an easy
> start.

I think the idea is nice, but maybe keywords should be used (like in
defcustom etc) for the implementation?

Re: tags for functions

[Ted Zlatanov]

On Tue, 20 Jan 2009 21:44:29 +0100 Lennart Borgman <lennart.borgman@gmail.com> wrote: 

LB> On Tue, Jan 20, 2009 at 9:36 PM, Ted Zlatanov <tzz@lifelogs.com> wrote:
>> Ted Z wrote in comp.emacs and gnu.emacs.help:
>>> I agree this would be useful. It's best done with tags IMO, rather
>>> than explicitly listing the related functions. For example, motion
>>> commands should be tagged "motion" and then every command with that
>>> tag can automatically list every motion command. The key is that the
>>> extra work is in classification, not in tediously listing every
>>> command's peers.
>>> 
>>> Tags I could use: motion, file, coding-system, menu, buffer, process
>>> 
>>> Each package should probably tag its commands with the package name.
>>> 
>>> Short tags are not always descriptive enough, but long tags get
>>> unpleasantly verbose so the real art is in balancing between the two.
>>> 
>>> Anything more hierarchical than tags is painful to manage in the long run.
>> 
>> A tangential discussion about the inconsistent naming of motion commands
>> led me to the proposal above.
>> 
>> I think it would be a nice addition to Emacs.  I did a search and didn't
>> find prior relevant discussions.
>> 
>> Every package with a comment stating Keywords: at the beginning could
>> automatically give those keywords as tags to its functions.  That would
>> probably be 50% or less of the total needed tags, but it's an easy
>> start.

LB> I think the idea is nice, but maybe keywords should be used (like in
LB> defcustom etc) for the implementation?

"Keywords" is an OK name for the concept; "tags" is the same thing
logically but we should stick with what's normal for Emacs Lisp.  What's
important is to:

1) build a list of command keywords quickly (using each package's
Keywords?)

2) refine the keywords into a simple taxonomy that is not too big nor
too small.

The first question I have is, how to associate keywords with a function?
Should I use the symbol-plist?

Then I can walk the namespace, find all functions, and display their
tags.  That will let me construct a taxonomy proposal.

Thanks
Ted

Re: tags for functions

[Lennart Borgman]

On Wed, Jan 21, 2009 at 11:11 PM, Ted Zlatanov <tzz@lifelogs.com> wrote:
> LB> I think the idea is nice, but maybe keywords should be used (like in
> LB> defcustom etc) for the implementation?
>
> "Keywords" is an OK name for the concept; "tags" is the same thing
> logically but we should stick with what's normal for Emacs Lisp.  What's
> important is to:

I meant something like

  (defun my-move-defun ()
    "docstring"
    :tags '(move)
    ...)

Re: tags for functions

[Glenn Morris]

Ted Zlatanov wrote:

> 1) build a list of command keywords quickly (using each package's
> Keywords?)
>
> 2) refine the keywords into a simple taxonomy that is not too big nor
> too small.

Just because I haven't seen finder mentioned yet in this dicussion,
M-x finder-list-keywords already tries to do something related to the above.

Re: tags for functions

[MON KEY]

>2) refine the keywords into a simple taxonomy that is not too big nor too small.
> The first question I have is, how to associate keywords with a function?

Keep in mind that a taxonomic description of code's content/concepts
is entirely different from a coded object class hierarchy - the
distinction is subtle and all to often overlooked... esp. by
programmers who *are* in fact quite adept at creating taxonomy in
code.

Please consider examining the following standard for a comprehensive
approach to accomplishing this task with optimal consideration of TRT.

ANSI/NISO Z39.19-2005
Guidelines for the Construction, Format, and Management of Monolingual
Controlled Vocabularies
ISBN: 1-880124-65-3

Available here: http://www.niso.org/standards/resources/Z39-19-2005.pdf

Re: tags for functions

[Ted Zlatanov]

On Wed, 21 Jan 2009 23:13:47 +0100 Lennart Borgman <lennart.borgman@gmail.com> wrote: 

LB> On Wed, Jan 21, 2009 at 11:11 PM, Ted Zlatanov <tzz@lifelogs.com> wrote:
LB> I think the idea is nice, but maybe keywords should be used (like in
LB> defcustom etc) for the implementation?
>> 
>> "Keywords" is an OK name for the concept; "tags" is the same thing
>> logically but we should stick with what's normal for Emacs Lisp.  What's
>> important is to:

LB> I meant something like

LB>   (defun my-move-defun ()
LB>     "docstring"
LB>     :tags '(move)
LB>     ...)

OK (I misunderstood you originally but it's not worth explaining :)

Let's go with

   (defun my-move-defun ()
     "docstring"
     :keywords '(move)
     ...)

to be consistent with "Keywords" in the package headers and other
places.

(defun) would need some small changes at the C level to handle this, but
my question was whether I should use the symbol's plist to store the
keywords, or a global hashtable for speed or something else.  It
seems like the symbol plist is the best place logically; are there any
issues with walking the whole namespace to find tags?  I don't want this
to be slow.

Ted

Re: tags for functions

[Ted Zlatanov]

On Thu, 22 Jan 2009 03:07:23 -0500 MON KEY <monkey@sandpframing.com> wrote: 

>> 2) refine the keywords into a simple taxonomy that is not too big nor too small.
>> The first question I have is, how to associate keywords with a function?

MK> Keep in mind that a taxonomic description of code's content/concepts
MK> is entirely different from a coded object class hierarchy - the
MK> distinction is subtle and all to often overlooked... esp. by
MK> programmers who *are* in fact quite adept at creating taxonomy in
MK> code.

MK> Please consider examining the following standard for a comprehensive
MK> approach to accomplishing this task with optimal consideration of TRT.

MK> ANSI/NISO Z39.19-2005
MK> Guidelines for the Construction, Format, and Management of Monolingual
MK> Controlled Vocabularies
MK> ISBN: 1-880124-65-3

MK> Available here: http://www.niso.org/standards/resources/Z39-19-2005.pdf

Well, at 180 pages of dense information this is a serious standard.
Also it defines "taxonomy" as a structural hierarchical classification,
whereas I've used it loosely to mean a tag space.  I'll use "tag space"
from now on to be clear.

Since we're not classifying general knowledge but a very specific domain
(Emacs Lisp functions) I don't think the standard needs to be applied.
It makes some good points, but things like a full hierarchy are
overkill, and others like proper name conventions just don't apply.
I'll try to be consistent with its naming recommendations, at least.

It brings up "synonym rings" which may be necessary: I propose 50% of
the effort is to come from each package's predeclared Keywords header.
That's determined by the package author so it's not easy to enforce a
particular tag space over all of Emacs.  With synonym rings, we can
associate synonymous keywords together.

Once the list of keywords is established, you should take a look to see
if you agree with it or have suggestions for improvements.  I'll post
here.

Ted

Re: tags for functions

[Ted Zlatanov]

On Wed, 21 Jan 2009 17:22:38 -0500 Glenn Morris <rgm@gnu.org> wrote: 

GM> Ted Zlatanov wrote:
>> 1) build a list of command keywords quickly (using each package's
>> Keywords?)
>> 
>> 2) refine the keywords into a simple taxonomy that is not too big nor
>> too small.

GM> Just because I haven't seen finder mentioned yet in this dicussion,
GM> M-x finder-list-keywords already tries to do something related to
GM> the above.

Great, that saves me a lot of work.  I can just hook into it and get the
global list of keywords and the packages which use each one.  Very
useful.  The work then becomes:

1) figure out place to store keywords (symbol plist or global)
2) add :keywords to defun and defvar, etc. as needed
3) figure out keywords for each function (and symbol?) based on package
4) refine and grow the tag space, figure out synonym rings

Ted

Re: tags for functions

[S+*n_Pe*rm*n]

Ted, I offer up the following with the caveat that I wouldn't be
taking the time to expound on this matter without the recognition your
ability to best implement the system. I apologize if my tone comes
across as overly pedantic - it is certainly not my intention.

>Well, at 180 pages of dense information this is a serious standard.

Its a serious subject matter.

>it defines "taxonomy" as a structural hierarchical classification,

I disagree - the standard doesn't make such an overt distinction -
this is your characterization/reduction.

>whereas I've used it loosely to mean a tag space.

'Tag Space/cloud' is a catch phrase for lazy cataloging - when
implemented on the scale you are proposing my intuition is that it
will fall over - do you know of such an implementation that doesn't?.

Tagging in the sense that you seem to be using the term is best left
to domains which lack a characteristic structure and/or which can't be
pre-limited/defined.

You have the opportunity *now* to reduce the number of potential core
`tags'. You appear to be suggesting that this is your intention. Such
an effort is best characterized as the production of a controlled
vocabulary.  Apropos of this, I am proposing you formalize the
production according to the best practices outlined by the standard.
Doing so will afford you the

>Since we're not classifying general knowledge but a very specific domain
>(Emacs Lisp functions).

This was my earlier point about programmers missing a very subtle distinction.
The domain of Emacs Elisp functions has specificity - to the extent
that there is a syntax and grammar for invoking those functions.

*What* the Emacs lisp code *does*, and *how* it is/may be *applied*,
and under what circumstances it may be best applied `generally' is an
entirely different matter. This is precisely why Elisp functions have
doc strings, why Emacs is self describing, and why promotes writing
code in a literate style.

>It makes some good points, but things like a full hierarchy are
>overkill

I suggest that a closer examination will reveal just the opposite.

> and others like proper name conventions just don't apply.
What about Internationalization?

>I'll try to be consistent with its naming recommendations, at least.

Good. This gist of the entire standard is relatively straightforward
and could can be reasonably reduced to a system of Suffix keywords
(terms) attached to the following `typed' Headwords:

- BT Broader Term
  BTG Broader Term (generic)  **
  BTI Broader Term (instance) **
  BTP Broader Term (partitive) **
- NT Narrower Term
  NTI Narrower Term (instance) **
  NTG Narrower Term (generic) **
  NTP Narrower Term (partitive) **
- RT Related Term
- TT Top Term
- U USE
- UF USED FOR
- HN History Note
- SN Scope Note

**These can be omitted and the system will retain its core functionality.

> It brings up "synonym rings" which may be necessary:

It brings up 4 possible domains of description - these aren't mutually
exclusive:

-Lists of controlled terms;
-Synonym rings;
-Taxonomies;
-Thesauri

>I propose 50% of the effort is to come from each package's predeclared >Keywords header.

I am suggesting that ceding this amount of effort to the author poses
an otherwise avoidable burden and is not DTRT esp. as Emacs Elisp
should provide this facility directly.

While the author of is the entity most aware of how his/her
functions/package inter-operate and she should be able to freely
assign keywords which she deems most appropriate as descriptors of
their potential domain of operation. However, what the author may not
be aware of is how his/her functions/packages inter-operate with a)
Emacs List and b) Other Packages/functions.

Placing the burden of awareness on the author is not an acceptable
solution. Emacs/Elisp should provide a controlled (extensible)
thesauri from which an package author can choose without concern for
how the entire system (extended or otherwise) operates.

> With synonym rings, we can associate synonymous keywords together.

Exaclty! This is in fact what the standard attempts to achieve and why
it was suggested as a guideline to best practices :)

So the issue then becomes how best to provide a thesauri which can
reasonably accommodate a diverse user base with diverse needs and
perspectives.

This is the domain of cataloging, *not* coding.

Your proposed implementation should also take into consideration that
the keyword structure need be robust enough to accommodate
internationalization. Again the standard addresses this in a coherent
fashion - `Tag Space' cannot accommodate internationalization issues.

Finally, I believe that in lieu of an increasing focus on FDL (Free
Documentation License) - Creative Commons - it is very important to
consider that much of what you are proposing is concerned with the
documentation side of Elisp code.  Future generations may well elect
to catalog and access this `content' as "prose" rather than as "code".

By accommodating a robust standard of thesauri construction such as
Z39.19 future GPL/FDL'd Elisp code would present an immediate
standardized mechanism by which code-librarians/catalogers might
readily access the information utilizing other standardized systems
such as Z39.50/ISO 23950 (Information Retrieval (Z39.50): Application
Service Definition and Protocol Specification).   In essence, not only
would such packages be 'self-describing/self-documenting' they would
also become 'self-cataloging' and potentially accessible via CQL
(Common Query Language) or whatever else comes along.

Stan

Re: tags for functions

[MON KEY]

Forgot to mention that the initial formulation of a controlled
vocabulary may well be already done for you.

If you consider that the emacs/elisp info manuals are *already*
formatted in a structured manner not unlike the Z39.19 you might
recognize that many of the `keywords' and relationships you need are
already in the .texi source.

I suggest you start with the the .texi source and extract the relevant
terms as a proposed hierarchy for your controlled vocabulary. This can
then be extended by with `freely assignable 'sub-tags' attached to the
core controlled headwords.

BT: Yanking::Replacing Yanks  <= post :: is the authors `freely
assigned' sub-tag
NT: Earlier Kills::Regexp on Kill Ring <= post :: is the authors
`freely assigned' sub-tag
RT: Mark Ring::Conditional Yanks <= post :: is the authors `freely
assigned' sub-tag

Such a structure might afford integration/reflection with make-info at
build time and present a more consistent interface at a higher level.

Re: tags for functions

[Stefan Monnier]

> Let's go with

>    (defun my-move-defun ()
>      "docstring"
>      :keywords '(move)
>      ...)

How 'bout

    (defun my-move-defun ()
      "docstring"
      (declare (keywords (move)))
      ...)


-- Stefan


PS: Don't take this to mean I think this is a good idea.  I haven't had
time to think about it or even read the thread for that matter.

Re: tags for functions

[Ted Zlatanov]

On Thu, 22 Jan 2009 14:39:40 -0500 "Stefan Monnier" <monnier@iro.umontreal.ca> wrote: 

>> Let's go with
>> (defun my-move-defun ()
>> "docstring"
>> :keywords '(move)
>> ...)

SM> How 'bout

SM>     (defun my-move-defun ()
SM>       "docstring"
SM>       (declare (keywords (move)))
SM>       ...)

SM> PS: Don't take this to mean I think this is a good idea.  I haven't had
SM> time to think about it or even read the thread for that matter.

The CL `declare' macro could be used based on a quick read of the
source, assuming that's what you mean, but the CL docs don't show how it
could be used generally.  Should I modify `cl-do-proclaim' to keep track
of the keywords myself when `declare' is called?

It does seem cleaner than :keywords at the defun level.

Ted

Re: tags for functions

[Ted Zlatanov]

On Thu, 22 Jan 2009 13:15:21 -0500 "S+*n_Pe*rm*n" <stan@derbycityprints.com> wrote: 

SP> Tagging in the sense that you seem to be using the term is best left
SP> to domains which lack a characteristic structure and/or which can't
SP> be pre-limited/defined.

I think Emacs Lisp lacks a characteristic structure and it can't be
pre-limited or pre-defined.  It's an amorphous tangle of functions and
symbols.  Attempting to impose any kind of structure on that tangle is
not useful IMHO.

SP> You have the opportunity *now* to reduce the number of potential core
SP> `tags'. You appear to be suggesting that this is your intention. Such
SP> an effort is best characterized as the production of a controlled
SP> vocabulary.  Apropos of this, I am proposing you formalize the
SP> production according to the best practices outlined by the standard.

I will try, but I'm not going to follow a 180 page standard to the
letter.  The goal is discovery of similar functions, not classification
of everything (see my original post).

>> and others like proper name conventions just don't apply.
SP> What about Internationalization?

If you can be specific, one of the maintainers may give you an official
statement, but AFAIK at the Emacs Lisp level that's not a concern.

I appreciate your comments but I think your vision is far more ambitious
than my proposal and has completely different goals (though it could use
the facilities I plan to implement).  Perhaps you should start a
separate thread and make a specific proposal.  The standards you have
listed, while very thorough, are not as helpful in such a proposal as a
clear classification proposal that can be seen and touched.

Ted

Re: tags for functions

[S+*n_Pe*rm*n]

>Perhaps you should start a
>separate thread and make a specific proposal.  The standards you have
>listed, while very thorough, are not as helpful in such a proposal as a
>clear classification proposal that can be seen and touched.

I'll bite :)

I've created a Mercurial repo with the a proposal as I understand it.

You can hg pull and/or download it from:
http://sp_dbc@bitbucket.org/sp_dbc/emacs-thesauri-permuted/

Example of Emacs/Elisp thesauri - permuted by hand from .info/.texi
files for Emacs and Elisp manuals

the `.el' file includes:
Top Terms
Broader Term ==> Narrower Term relationships

This should provide a good pool of keywords even if you only approach
the problem from a `tag cloud' perspective.

s_P

Re: tags for functions

[Juri Linkov]

>>> 1) build a list of command keywords quickly (using each package's
>>> Keywords?)
>>>
>>> 2) refine the keywords into a simple taxonomy that is not too big nor
>>> too small.
>
> GM> Just because I haven't seen finder mentioned yet in this dicussion,
> GM> M-x finder-list-keywords already tries to do something related to
> GM> the above.
>
> Great, that saves me a lot of work.  I can just hook into it and get the
> global list of keywords and the packages which use each one.  Very
> useful.  The work then becomes:
>
> 1) figure out place to store keywords (symbol plist or global)
> 2) add :keywords to defun and defvar, etc. as needed
> 3) figure out keywords for each function (and symbol?) based on package
> 4) refine and grow the tag space, figure out synonym rings

finder.el extracts package keywords from the comments in the file header.
Doing the same for function keywords means placing them in the comments
before the function definition.  Many modern programming languages use the
@-syntax for documentation tags.  So for Emacs Lisp they could look like:

;; @tag1
;; @tag2
(defun my-move-defun ()
  "Docstring."
  ...)

However, since Emacs uses doc strings for documentation it would be better
to add them in doc strings like:

(defun my-move-defun ()
  "Docstring.
@tag1
@tag2"
  ...)

-- 
Juri Linkov
http://www.jurta.org/emacs/

Re: tags for functions

[Ted Zlatanov]

On Sun, 25 Jan 2009 02:32:09 +0200 Juri Linkov <juri@jurta.org> wrote: 

JL> finder.el extracts package keywords from the comments in the file
JL> header.  Doing the same for function keywords means placing them in
JL> the comments before the function definition.  Many modern
JL> programming languages use the @-syntax for documentation tags.
JL> [...]  add them in doc strings like:

JL> (defun my-move-defun ()
JL>   "Docstring.
JL> @tag1
JL> @tag2"
JL>   ...)

That's probably the least intrusive approach, I like it.  So finder.el
will be the central place for getting all keywords, which works fine for
me.  It's nice that it's already called everywhere needed, so I just
need to hook into that infrastructure.  Does anyone have a problem with it?

Thanks
Ted

Re: tags for functions

[Juri Linkov]

> JL> finder.el extracts package keywords from the comments in the file
> JL> header.  Doing the same for function keywords means placing them in
> JL> the comments before the function definition.  Many modern
> JL> programming languages use the @-syntax for documentation tags.
> JL> [...]  add them in doc strings like:
>
> JL> (defun my-move-defun ()
> JL>   "Docstring.
> JL> @tag1
> JL> @tag2"
> JL>   ...)
>
> That's probably the least intrusive approach, I like it.  So finder.el
> will be the central place for getting all keywords, which works fine for
> me.  It's nice that it's already called everywhere needed, so I just
> need to hook into that infrastructure.  Does anyone have a problem with it?

finder.el scans source files for keywords in comments.  So we don't need
finder.el with tags in doc strings because it's easy to collect tags from
the `documentation' property of all function symbols.

-- 
Juri Linkov
http://www.jurta.org/emacs/

Re: tags for functions

[Ted Zlatanov]

On Tue, 27 Jan 2009 01:55:32 +0200 Juri Linkov <juri@jurta.org> wrote: 

JL> finder.el scans source files for keywords in comments.  So we don't need
JL> finder.el with tags in doc strings because it's easy to collect tags from
JL> the `documentation' property of all function symbols.

You've lost me.  Can you please explain with code or point to relevant
code?  It's not clear to me where the `documentation' property comes
from (the doc tags?  then what code parses it?) and how to access it
globally.

Thanks
Ted

Re: tags for functions

[MON KEY]

> You've lost me.  Can you please explain with code or point to relevant
> code?

(documentation 'documentation)
&
(describe-symbol 'documentation)

> It's not clear to me where the `documentation' property comes
> from (the doc tags?  then what code parses it?)

(symbol-plist 'documentation)

(documentation-property 'documentation
'some-documentation-property)

Re: tags for functions

[Juri Linkov]

> JL> finder.el scans source files for keywords in comments.  So we don't need
> JL> finder.el with tags in doc strings because it's easy to collect tags from
> JL> the `documentation' property of all function symbols.
>
> You've lost me.  Can you please explain with code or point to relevant
> code?

Something like

(defun my-move-defun ()
  "Docstring.
@keywords keyword1, keyword2")

(defun my-copy-defun ()
  "Docstring.
@keywords keyword3, keyword4")

(let (keywordlist docstr)
  (mapatoms
   (lambda (symbol)
     (when (and (functionp symbol) (setq docstr (documentation symbol t))
                (string-match "@keywords \\(.*\\)" docstr))
       (add-to-list 'keywordlist
                    (cons symbol (split-string (match-string 1 docstr)
                                               "\\s-*,\\s-*" t))))))
  keywordlist)

=> '((my-copy-defun "keyword3" "keyword4")
     (my-move-defun "keyword1" "keyword2"))

-- 
Juri Linkov
http://www.jurta.org/emacs/

Re: tags for functions

[Lennart Borgman]

On Wed, Jan 28, 2009 at 1:02 AM, Juri Linkov <juri@jurta.org> wrote:
>> JL> finder.el scans source files for keywords in comments.  So we don't need
>> JL> finder.el with tags in doc strings because it's easy to collect tags from
>> JL> the `documentation' property of all function symbols.
>>
>> You've lost me.  Can you please explain with code or point to relevant
>> code?
>
> Something like
>
> (defun my-move-defun ()
>  "Docstring.
> @keywords keyword1, keyword2")
>
> (defun my-copy-defun ()
>  "Docstring.
> @keywords keyword3, keyword4")
>
> (let (keywordlist docstr)
>  (mapatoms
>   (lambda (symbol)
>     (when (and (functionp symbol) (setq docstr (documentation symbol t))
>                (string-match "@keywords \\(.*\\)" docstr))
>       (add-to-list 'keywordlist
>                    (cons symbol (split-string (match-string 1 docstr)
>                                               "\\s-*,\\s-*" t))))))
>  keywordlist)
>
> => '((my-copy-defun "keyword3" "keyword4")
>     (my-move-defun "keyword1" "keyword2"))


8.4 seconds (evaled) on my pc.

Maybe :keywords is better?

Re: tags for functions

[Ted Zlatanov]

On Wed, 28 Jan 2009 02:02:51 +0200 Juri Linkov <juri@jurta.org> wrote: 

JL> (defun my-move-defun ()
JL>   "Docstring.
JL> @keywords keyword1, keyword2")

JL> (defun my-copy-defun ()
JL>   "Docstring.
JL> @keywords keyword3, keyword4")

JL> (let (keywordlist docstr)
JL>   (mapatoms
JL>    (lambda (symbol)
JL>      (when (and (functionp symbol) (setq docstr (documentation symbol t))
JL>                 (string-match "@keywords \\(.*\\)" docstr))
JL>        (add-to-list 'keywordlist
JL>                     (cons symbol (split-string (match-string 1 docstr)
JL>                                                "\\s-*,\\s-*" t))))))
JL>   keywordlist)

JL> => '((my-copy-defun "keyword3" "keyword4")
JL>      (my-move-defun "keyword1" "keyword2"))

I see, thank you for explaining.

On Wed, 28 Jan 2009 01:36:03 +0100 Lennart Borgman <lennart.borgman@gmail.com> wrote: 

LB> 8.4 seconds (evaled) on my pc.

LB> Maybe :keywords is better?

It seems to me that modifying (defun) to update a cache is the best
approach, since that's the only way the documentation string is normally
created or modified AFAIK.  Maybe a defun-after-hook would be
appropriate, in case people want to run other code on a defun call?  At
least for me, a defun-after-hook would solve it, and require no changes
outside of setting up defun-after-hook.  In contrast, :keywords requires
changing things significantly, and (declare) as Stefan suggested
requires code changes everywhere.

Setting things up would require, at any point:

1) fill defun-after-hook with the cache population function

2) populate the cache with all the docs for functions already defined,
using mapatoms

So it doesn't require to be set up at any particular time, and can be a
loadable package.

Also, this needs to be fast.  The cache can hold reverse mapping from
keyword to list of functions, so it would be much faster to mapatoms for
the common usage scenarios (look up all keywords and look up functions
for a keyword).  Obviously, looking up keywords for function doesn't
require mapatoms so it will be fast regardless.

Thanks
Ted

Re: tags for functions

[Stefan Monnier]

JL> (when (and (functionp symbol) (setq docstr (documentation symbol t))

Most docstrings (upon startup) are not kept in memory but in the DOC
file instead.  Calling `documentation' will load this file and look for
the corresponding docstring there.

I think the best/only way forward is to do something along the lines of
cus-load.el, i.e. generate the data offline.


        Stefan "who still doesn't know what data we're talking about"

Re: tags for functions

[Ted Zlatanov]

On Wed, 28 Jan 2009 13:40:01 -0500 Stefan Monnier <monnier@IRO.UMontreal.CA> wrote: 

JL> (when (and (functionp symbol) (setq docstr (documentation symbol t))

SM> Most docstrings (upon startup) are not kept in memory but in the DOC
SM> file instead.  Calling `documentation' will load this file and look for
SM> the corresponding docstring there.

SM> I think the best/only way forward is to do something along the lines of
SM> cus-load.el, i.e. generate the data offline.

SM>         Stefan "who still doesn't know what data we're talking about"

I'm interested in "@keyword" substrings of the docstring that indicate
the keywords for a particular function.  The goals are to:

- list all keywords
- list all keywords for a function
- list all functions for a keyword

Juri's solution is slow, as Lennart noted, due to the `documentation'
issue you mentioned.  I said I think a defun-after-hook would be the
best solution.  Maybe it should be combined with a etc/DOC scan to find
all the existing keywords quickly.

Ted

Re: tags for functions

[Stefan Monnier]

SM> Stefan "who still doesn't know what data we're talking about"
> I'm interested in "@keyword" substrings of the docstring that indicate
> the keywords for a particular function.  The goals are to:
> - list all keywords
> - list all keywords for a function
> - list all functions for a keyword

Not know what that data would be used for, it's hard to give you
good feedback.

> Juri's solution is slow, as Lennart noted, due to the `documentation'
> issue you mentioned.  I said I think a defun-after-hook would be the
> best solution.  Maybe it should be combined with a etc/DOC scan to find
> all the existing keywords quickly.

In any case, we don't want to collect/store this info all the time: we
only want to pay for it when this data is actually used.


        Stefan

Re: tags for functions

[Ted Zlatanov]

On Wed, 28 Jan 2009 20:43:33 -0500 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

SM> Stefan "who still doesn't know what data we're talking about"
>> I'm interested in "@keyword" substrings of the docstring that indicate
>> the keywords for a particular function.  The goals are to:
>> - list all keywords
>> - list all keywords for a function
>> - list all functions for a keyword

SM> Not know what that data would be used for, it's hard to give you
SM> good feedback.

The goal is to do the three operations above as an API.  Then Emacs will
use it, for example, to show related functions in C-h f

The package can also provide browsing by keyword, as finder.el does.

>> Juri's solution is slow, as Lennart noted, due to the `documentation'
>> issue you mentioned.  I said I think a defun-after-hook would be the
>> best solution.  Maybe it should be combined with a etc/DOC scan to find
>> all the existing keywords quickly.

SM> In any case, we don't want to collect/store this info all the time: we
SM> only want to pay for it when this data is actually used.

Agreed.  So here's what I am proposing:

A package which, when installed, will:

1) modify defun-after-hook so new defun calls will be noted, or install
advice on defun (I'd rather do the former)

2) scan existing docstrings over mapatoms using `documentation' (it's
slow now apparently, so it will need to be optimized for a batch scan)

3) provide functionality as listed above under "goals"

I only need defun-after-hook to be approved, I can do the rest.  Do you
agree it's useful or would you rather not provide such a hook?

Thanks
Ted

Re: tags for functions

[Lennart Borgman]

On Thu, Jan 29, 2009 at 2:43 AM, Stefan Monnier
<monnier@iro.umontreal.ca> wrote:
>> Juri's solution is slow, as Lennart noted, due to the `documentation'
>> issue you mentioned.  I said I think a defun-after-hook would be the
>> best solution.  Maybe it should be combined with a etc/DOC scan to find
>> all the existing keywords quickly.
>
> In any case, we don't want to collect/store this info all the time: we
> only want to pay for it when this data is actually used.

Is there really any reason not to just collect it while compiling
(using :keywords)? Beside the human work needed ... ;-)

(let ((n 0))
  (defun my-keyw ()
    (let (keywordlist docstr)
      (mapatoms
       (lambda (symbol)
         (when (functionp symbol)
           (setq n (1+ n))
           (put symbol 'my-keyw (list 'test 'ing)))))))

  (benchmark 1 '(my-keyw))
  (message "n=%s" n))

This takes 0.2 s for 12000 functions.

Re: tags for functions

[Stefan Monnier]

SM> Stefan "who still doesn't know what data we're talking about"
>>> I'm interested in "@keyword" substrings of the docstring that indicate
>>> the keywords for a particular function.  The goals are to:
>>> - list all keywords
>>> - list all keywords for a function
>>> - list all functions for a keyword

SM> Not know what that data would be used for, it's hard to give you
SM> good feedback.
> The goal is to do the three operations above as an API.

This is not "what it's used for", it's "how it's provided".

> Then Emacs will use it, for example, to show related functions in C-h f
> The package can also provide browsing by keyword, as finder.el does.

I see, thanks.  The list of related functions can be rather long, so
it's probably better to only add a "show related" button in C-h f and
only show the list when the user asks for it.

> 1) modify defun-after-hook so new defun calls will be noted, or install
> advice on defun (I'd rather do the former)

I'd rather to try to avoid both.

> 2) scan existing docstrings over mapatoms using `documentation' (it's
> slow now apparently, so it will need to be optimized for a batch scan)

If it's only done "once per session" and only when the user specifically
asks for this info, it's probably not that bad.

> I only need defun-after-hook to be approved, I can do the rest.  Do you
> agree it's useful or would you rather not provide such a hook?

I'm not convinced.  I'm not even sure this kind of info will turn out to
be useful/usable.  Currently, you'd spend all your time scanning
docstrings that don't contain any such keywords.  Adding those keywords
to docstrings would be a major undertaking.  So it's probably better to
start with data from elsewhere (e.g. from the elisp manual) anyway.

In other words, maybe a defun-after-hook ill be the right tool, but
we're pretty far from being in a position to judge, and it seems likely
that the end design will use a completely different approach anyway.


        Stefan

Re: tags for functions

[Ted Zlatanov]

On Thu, 29 Jan 2009 21:32:59 +0100 Lennart Borgman <lennart.borgman@gmail.com> wrote: 

LB> On Thu, Jan 29, 2009 at 2:43 AM, Stefan Monnier
LB> <monnier@iro.umontreal.ca> wrote:
>>> Juri's solution is slow, as Lennart noted, due to the `documentation'
>>> issue you mentioned.  I said I think a defun-after-hook would be the
>>> best solution.  Maybe it should be combined with a etc/DOC scan to find
>>> all the existing keywords quickly.
>> 
>> In any case, we don't want to collect/store this info all the time: we
>> only want to pay for it when this data is actually used.

LB> Is there really any reason not to just collect it while compiling
LB> (using :keywords)? Beside the human work needed ... ;-)

LB> (let ((n 0))
LB>   (defun my-keyw ()
LB>     (let (keywordlist docstr)
LB>       (mapatoms
LB>        (lambda (symbol)
LB>          (when (functionp symbol)
LB>            (setq n (1+ n))
LB>            (put symbol 'my-keyw (list 'test 'ing)))))))

LB>   (benchmark 1 '(my-keyw))
LB>   (message "n=%s" n))

LB> This takes 0.2 s for 12000 functions.

Modifying docstrings is much easier, and does not involve any code
changes anywhere, so I would prefer it as a way of declaring keywords.

Ted

Re: tags for functions

[Ted Zlatanov]

On Thu, 29 Jan 2009 16:52:04 -0500 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

>> Then Emacs will use it, for example, to show related functions in C-h f
>> The package can also provide browsing by keyword, as finder.el does.

SM> I see, thanks.  The list of related functions can be rather long, so
SM> it's probably better to only add a "show related" button in C-h f and
SM> only show the list when the user asks for it.

OK.

>> 2) scan existing docstrings over mapatoms using `documentation' (it's
>> slow now apparently, so it will need to be optimized for a batch scan)

SM> If it's only done "once per session" and only when the user specifically
SM> asks for this info, it's probably not that bad.

OK.

>> I only need defun-after-hook to be approved, I can do the rest.  Do you
>> agree it's useful or would you rather not provide such a hook?

SM> I'm not convinced.  I'm not even sure this kind of info will turn out to
SM> be useful/usable.  Currently, you'd spend all your time scanning
SM> docstrings that don't contain any such keywords.  

We have to start somewhere.  It's definitely useful to me, and two other
people that started the original thread.  The example that brought it up
is that motion functions are legion in the Emacs Lisp namespace, and
have wildly disparate names.  A "motion" tag would allow a curious
explorer to see what other motion commands exist from any starting
point.  I suggested some other tags.

SM> Adding those keywords to docstrings would be a major undertaking.
SM> So it's probably better to start with data from elsewhere (e.g. from
SM> the elisp manual) anyway.

Yes, but I'm sure it can be automated a bit regardless.  The hard part,
actually, is picking the right keywords as others have pointed out.

SM> In other words, maybe a defun-after-hook ill be the right tool, but
SM> we're pretty far from being in a position to judge, and it seems likely
SM> that the end design will use a completely different approach anyway.

Understood.  Rather than overengineer, I will provide a simple working
solution that uses docstring tags, gathering them all when it's first
loaded as a package (so autoloading will DTRT as you suggested).  If it
needs optimizations, they can be added later.

Ted

RE: tags for functions

[Drew Adams]

1. When the thing documented in *Help* is also in the manual, then we have a
link to the manual from the *Help* buffer.

2. The manual should have such a `See also' cross reference, if any place
should.

3. When both the thing doc'd in *Help* and the thing xref'd are in the manual,
#1 and #2 would take care of such cross-referencing. Admittedly, passing through
the manual represents an indirection, but typically a fruitful one.

Dunno if it's worth adding this for cases that are not covered by the manual. It
sounds like a helpful feature. I can't speak to the implementation cost. (There
is also a doc-maintenance cost to consider.)

Re: tags for functions

[Ted Zlatanov]

On Fri, 30 Jan 2009 08:06:42 -0800 "Drew Adams" <drew.adams@oracle.com> wrote: 

DA> 1. When the thing documented in *Help* is also in the manual, then we have a
DA> link to the manual from the *Help* buffer.

DA> 2. The manual should have such a `See also' cross reference, if any place
DA> should.

Agreed on both counts.  The API I'm providing can be combined with the
manual generation to do this, and it will use the manual sections to get
the first keyword seeds.  I don't want the manual to be the primary
source of keywords, that's all.

DA> 3. When both the thing doc'd in *Help* and the thing xref'd are in the manual,
DA> #1 and #2 would take care of such cross-referencing. Admittedly, passing through
DA> the manual represents an indirection, but typically a fruitful one.

DA> Dunno if it's worth adding this for cases that are not covered by the manual. It
DA> sounds like a helpful feature. I can't speak to the implementation cost. (There
DA> is also a doc-maintenance cost to consider.)

There are many many packages and functions not covered in the manual
that are nevertheless used and useful.  They can and should have
keywords.  So the docstring should be the primary source of keywords,
and looking up the function with C-h f should be the primary way to
cross-reference functions by keyword.

The goal of this API is not to index Emacs' internal functions'
keywords.  It will cover everything that `mapatoms' returns; I'm
considering symbols in addition to functions but I want to get functions
working first.

Ted

Re: tags for functions

[Stefan Monnier]

> 1. When the thing documented in *Help* is also in the manual, then we have a
> link to the manual from the *Help* buffer.

We can get pretty close to that already with info-lookup-symbol.
It'd probably be a good addition (for post-23.1, obviously).

> 2. The manual should have such a `See also' cross reference, if any place
> should.

Not sure what you're saying here.


        Stefan

RE: tags for functions

[Drew Adams]

> > 1. When the thing documented in *Help* is also in the 
> >    manual, then we have a link to the manual from the *Help* buffer.
> 
> We can get pretty close to that already with info-lookup-symbol.
> It'd probably be a good addition (for post-23.1, obviously).
> 
> > 2. The manual should have such a `See also' cross 
> >    reference, if any place should.
> 
> Not sure what you're saying here.

Only that the manual already takes care of this need, to some extent, for
functions that are documented there: *Help* gets you to where the function is
documented in the manual, and a cross-ref in the manual can then take you to doc
for related functions, when that's appropriate.

(But, as mentioned, going through the manual adds indirection, and it doesn't
help for functions that are not in the manual.)

Re: tags for functions

[Juri Linkov]

> The goal of this API is not to index Emacs' internal functions'
> keywords.  It will cover everything that `mapatoms' returns; I'm
> considering symbols in addition to functions but I want to get functions
> working first.

We might also want to find keywords of the functions whose packages are
not yet loaded.  So using `mapatoms' can't find them, and we will need
a global registry like finder-inf.el, cus-load.el, loaddefs.el.

-- 
Juri Linkov
http://www.jurta.org/emacs/