Adding a few more finder keywords

Adding a few more finder keywords

[Artur Malabarba]

Would it be acceptable for us to add a few more keywords to
`finder-known-keywords'?

There are lots of new keywords being used by the thousands of packages
out there, and some of them are a little chaotic (see below). I think
it would help a bit if we added the most common ones to
`finder-known-keywords'.

The list of keywords below was obtained with the following code (note
that it depends on map.el).

    (defvar kwd-pkg-table (make-hash-table :test #'equal :size 2000))
    (package-read-all-archive-contents)
    (package--mapc
     (lambda (desc)
       (dolist (key (package-desc--keywords desc))
         (cl-pushnew (package-desc-name desc) (gethash key accum)))))

    (defvar kwd-pkg-sorted-alist
      (sort (map-into kwd-pkg-table 'list)
            (lambda (x y) (> (length (cdr x))
                        (length (cdr y))))))

    (pp (mapcar (lambda (p) (cons (car p) (length (cdr p)))) kwd-pkg-alist)
        (current-buffer))

Note how the list contains:
- 17 "theme" and 18 "themes".
- 8 "package" and 4 "packages" (and many other instances of singular vs plural)
- "mode-line", "modeline", "mode line", and even "stausline".
- 4 "major" and 4 "major-mode"

I think this situation can be improved if we extend the list of
default keywords. So a new developer, writing his first package,
doesn't have to decide between "theme" or "themes" (one of them will
already be offerd to them in the defaults).

I'm not suggesting we impose any kind of restriction here. Everyone
will still be free to use whatever keywords they want. All I'm
suggesting is that we add a few more entries to the variable
`finder-known-keywords'


Cheers, (data below)

----------------------

(("convenience" . 387)
 ("tools" . 166)
 ("languages" . 148)
 ("lisp" . 91)
 ("extensions" . 89)
 ("faces" . 59)
 ("files" . 41)
 ("org" . 40)
 ("completion" . 39)
 ("processes" . 36)
 ("helm" . 36)
 ("matching" . 35)
 ("help" . 34)
 ("git" . 30)
 ("comm" . 30)
 ("data" . 28)
 ("frames" . 28)
 ("ruby" . 28)
 ("wp" . 25)
 ("emacs" . 24)
 ("evil" . 24)
 ("vc" . 23)
 ("search" . 22)
 ("local" . 22)
 ("hypermedia" . 21)
 ("internal" . 21)
 ("color" . 20)
 ("c" . 20)
 ("python" . 19)
 ("project" . 19)
 ("themes" . 18)
 ("abbrev" . 18)
 ("games" . 18)
 ("theme" . 17)
 ("multimedia" . 17)
 ("dired" . 17)
 ("emulation" . 16)
 ("clojure" . 16)
 ("docs" . 16)
 ("org-mode" . 16)
 ("window" . 15)
 ("unix" . 15)
 ("editing" . 15)
 ("outlines" . 15)
 ("calendar" . 14)
 ("navigation" . 14)
 ("highlight" . 14)
 ("programming" . 13)
 ("text" . 13)
 ("javascript" . 13)
 ("mode" . 13)
 ("test" . 13)
 ("language" . 12)
 ("mouse" . 11)
 ("keys" . 11)
 ("php" . 11)
 ("grep" . 10)
 ("unicode" . 10)
 ("html" . 10)
 ("buffer" . 10)
 ("perl" . 10)
 ("mode-line" . 9)
 ("cursor" . 9)
 ("region" . 9)
 ("rails" . 9)
 ("interface" . 9)
 ("package" . 8)
 ("tex" . 8)
 ("keyboard" . 8)
 ("mail" . 8)
 ("maint" . 8)
 ("news" . 8)
 ("elisp" . 8)
 ("emulations" . 8)
 ("auto-complete" . 8)
 ("testing" . 8)
 ("tests" . 8)
 ("development" . 8)
 ("flymake" . 8)
 ("frame" . 8)
 ("shell" . 8)
 ("xml" . 7)
 ("network" . 7)
 ("edit" . 7)
 ("http" . 7)
 ("cider" . 7)
 ("java" . 7)
 ("dictionary" . 7)
 ("anything" . 7)
 ("web" . 7)
 ("chinese" . 7)
 ("vim" . 7)
 ("terminal" . 7)
 ("i18n" . 7)
 ("configuration" . 7)
 ("markdown" . 6)
 ("indentation" . 6)
 ("regexp" . 6)
 ("css" . 6)
 ("latex" . 6)
 ("bibtex" . 6)
 ("github" . 6)
 ("snippets" . 6)
 ("menu" . 6)
 ("utility" . 6)
 ("c++" . 6)
 ("go" . 6)
 ("repl" . 6)
 ("haskell" . 6)
 ("characters" . 6)
 ("markup" . 6)
 ("productivity" . 6)
 ("presentation" . 6)
 ("table" . 6)
 ("window manager" . 6)
 ("tree" . 5)
 ("display" . 5)
 ("replace" . 5)
 ("projectile" . 5)
 ("compilation" . 5)
 ("face" . 5)
 ("blog" . 5)
 ("buffers" . 5)
 ("windows" . 5)
 ("nrepl" . 5)
 ("command" . 5)
 ("accessibility" . 5)
 ("popup" . 5)
 ("paste" . 5)
 ("directories" . 5)
 ("hide" . 5)
 ("repeat" . 5)
 ("erc" . 5)
 ("emms" . 5)
 ("osx" . 5)
 ("babel" . 5)
 ("oop" . 5)
 ("speed" . 5)
 ("ssh" . 4)
 ("config" . 4)
 ("packages" . 4)
 ("input" . 4)
 ("clojurescript" . 4)
 ("maven" . 4)
 ("gnus" . 4)
 ("todo" . 4)
 ("ascii" . 4)
 ("symbols" . 4)
 ("dark" . 4)
 ("rest" . 4)
 ("jump" . 4)
 ("projects" . 4)
 ("whitespace" . 4)
 ("ess" . 4)
 ("wiki" . 4)
 ("major" . 4)
 ("major-mode" . 4)
 ("database" . 4)
 ("cycle" . 4)
 ("pastebin" . 4)
 ("refactoring" . 4)
 ("eshell" . 4)
 ("etags" . 4)
 ("tags" . 4)
 ("vim-emulation" . 4)
 ("flycheck" . 4)
 ("compile" . 4)
 ("sql" . 4)
 ("minibuffer" . 4)
 ("environment" . 4)
 ("session" . 4)
 ("intellisense" . 4)
 ("sage" . 4)
 ("math" . 4)
 ("ido" . 4)
 ("ocaml" . 4)
 ("w3m" . 3)
 ("strings" . 3)
 ("hex" . 3)
 ("rgb" . 3)
 ("bbdb" . 3)
 ("highlighting" . 3)
 ("less" . 3)
 ("color-theme" . 3)
 ("learning" . 3)
 ("apropos" . 3)
 ("backup" . 3)
 ("csv" . 3)
 ("macro" . 3)
 ("terminals" . 3)
 ("cursors" . 3)
 ("graph" . 3)
 ("file" . 3)
 ("postgresql" . 3)
 ("toggle" . 3)
 ("discover" . 3)
 ("line" . 3)
 ("selection" . 3)
 ("string" . 3)
 ("elscreen" . 3)
 ("soundcloud" . 3)
 ("unittest" . 3)
 ("ert" . 3)
 ("uml" . 3)
 ("searching" . 3)
 ("textmate" . 3)
 ("comment" . 3)
 ("thing" . 3)
 ("functions" . 3)
 ("music" . 3)
 ("workspace" . 3)
 ("kill" . 3)
 ("management" . 3)
 ("images" . 3)
 ("mac" . 3)
 ("folding" . 3)
 ("refactor" . 3)
 ("log" . 3)
 ("irc" . 3)
 ("utilities" . 3)
 ("writing" . 3)
 ("simplenote" . 3)
 ("chat" . 3)
 ("tdd" . 3)
 ("button" . 3)
 ("widget" . 3)
 ("virtualenv" . 3)
 ("automation" . 3)
 ("server" . 3)
 ("persistence" . 3)
 ("async" . 3)
 ("deferred" . 3)
 ("translation" . 3)
 ("dotemacs" . 3)
 ("yasnippet" . 3)
 ("slime" . 3)
 ("startup" . 3)
 ("yaml" . 2)
 ("rpc" . 2)
 ("www" . 2)
 ("history" . 2)
 ("palette" . 2)
 ("data structures" . 2)
 ("gnuplot" . 2)
 ("plotting" . 2)
 ("lists" . 2)
 ("location" . 2)
 ("motion" . 2)
 ("parser" . 2)
 ("data structures trie" . 2)
 ("register" . 2)
 ("minor-mode" . 2)
 ("context" . 2)
 ("ada" . 2)
 ("emmet" . 2)
 ("bootstrap" . 2)
 ("comments" . 2)
 ("deftheme" . 2)
 ("occur" . 2)
 ("amd" . 2)
 ("fun" . 2)
 ("json" . 2)
 ("api" . 2)
 ("milkode" . 2)
 ("keyword" . 2)
 ("download" . 2)
 ("bittorrent" . 2)
 ("lua" . 2)
 ("fingers" . 2)
 ("modal" . 2)
 ("workman" . 2)
 ("bookmark" . 2)
 ("bookmarks" . 2)
 ("url" . 2)
 ("blogger" . 2)
 ("macros" . 2)
 ("chef" . 2)
 ("font" . 2)
 ("coffee-mode" . 2)
 ("company" . 2)
 ("documentation" . 2)
 ("grads" . 2)
 ("script" . 2)
 ("cypher" . 2)
 ("of" . 2)
 ("diredp" . 2)
 ("sort" . 2)
 ("pastie" . 2)
 ("dropbox" . 2)
 ("e2wm" . 2)
 ("eldoc" . 2)
 ("evernote" . 2)
 ("applescript" . 2)
 ("english" . 2)
 ("grammar" . 2)
 ("r" . 2)
 ("tag" . 2)
 ("tab" . 2)
 ("leader" . 2)
 ("paredit" . 2)
 ("smartparens" . 2)
 ("tabs" . 2)
 ("desktop" . 2)
 ("hsv" . 2)
 ("hexadecimal" . 2)
 ("menus" . 2)
 ("cli" . 2)
 ("zsh" . 2)
 ("d" . 2)
 ("jshint" . 2)
 ("vala" . 2)
 ("movement" . 2)
 ("fullscreen" . 2)
 ("gerrit" . 2)
 ("version control" . 2)
 ("release management" . 2)
 ("gnome" . 2)
 ("hardware" . 2)
 ("template" . 2)
 ("graphviz" . 2)
 ("dot" . 2)
 ("marking" . 2)
 ("bib" . 2)
 ("project-management" . 2)
 ("scss" . 2)
 ("google" . 2)
 ("browse" . 2)
 ("hg" . 2)
 ("show" . 2)
 ("tail" . 2)
 ("light" . 2)
 ("emulating" . 2)
 ("password" . 2)
 ("fringe" . 2)
 ("mongodb" . 2)
 ("data structure" . 2)
 ("io" . 2)
 ("isearch" . 2)
 ("invisible" . 2)
 ("libraries" . 2)
 ("mallard" . 2)
 ("workspaces" . 2)
 ("extentions" . 2)
 ("narrow" . 2)
 ("view" . 2)
 ("nyan" . 2)
 ("cat" . 2)
 ("lulz" . 2)
 ("browser" . 2)
 ("literate programming" . 2)
 ("reproducible research" . 2)
 ("notes" . 2)
 ("jekyll" . 2)
 ("filtering" . 2)
 ("asciidoc" . 2)
 ("restructuredtext" . 2)
 ("pcomplete" . 2)
 ("switch" . 2)
 ("speedbar" . 2)
 ("check" . 2)
 ("rcirc" . 2)
 ("layout" . 2)
 ("scala" . 2)
 ("yank" . 2)
 ("plain text" . 2)
 ("find" . 2)
 ("remember" . 2)
 ("spell" . 2)
 ("sphinx" . 2)
 ("comms" . 2)
 ("swoop" . 2)
 ("inner" . 2)
 ("thesaurus" . 2)
 ("spelling" . 2)
 ("timer" . 2)
 ("minor mode" . 2)
 ("time" . 2)
 ("toml" . 2)
 ("tramp" . 2)
 ("vagrant" . 2)
 ("visual" . 2)
 ("feedback" . 2)
 ("external" . 2)
 ("lines" . 2)
 ("wikipedia" . 2)
 ("cut" . 2)
 ("copy" . 2)
 ("window-configuration" . 2)
 ("ahk" . 2)
 ("autohotkey" . 2)
 ("hotkey" . 2)
 ("keyboard shortcut" . 2)
 ("elixir" . 2)
 ("message" . 2)
 ("exuberant ctags" . 2)
 ("auto" . 2)
 ("client" . 2)
 ("internet" . 2)
 ("c#" . 2)
 ("imenu" . 2)
 ("killing" . 2)
 ("code" . 2)
 ("elpa" . 2)
 ("emacswiki" . 2)
 ("emacs-lisp" . 2)
 ("mpv" . 2)
 ("profile" . 2)
 ("mnemonic" . 2)
 ("gitlab" . 2)
 ("resizing" . 2)
 ("gemfile" . 2)
 ("usability" . 2)
 ("tooltip" . 2)
 ("tool" . 2)
 ("csharp" . 2)
 ("magit" . 2)
 ("publish" . 2)
 ("twitter" . 2)
 ("library" . 2)
 ("applications" . 2)
 ("solar" . 2)
 ("sunrise" . 2)
 ("sunset" . 2)
 ("rspec" . 2)
;;; keywords with a count of 1 are omitted. Run the code to se them
)

RE: Adding a few more finder keywords

[Drew Adams]

> Would it be acceptable for us to add a few more keywords to
> `finder-known-keywords'?

Good initiative, generally, to revisit the list.  Maybe a few
"few"er, though?  I'm also not sure how helpful it is for
`finder-known-keywords' to be large.

Be careful of spelling: e.g., extensions, not extentions.

I would suggest that you remove keywords that are related only
to Emacs libraries that are not delivered as part of GNU Emacs.
Here are some such candidates (not sure of all, and there might
be others).

>  ("helm" . 36)
>  ("projectile" . 5)
>  ("emms" . 5)
>  ("maven" . 4)
>  ("ess" . 4)
>  ("pastebin" . 4)
>  ("flycheck" . 4)
>  ("intellisense" . 4)
>  ("sage" . 4)
>  ("color-theme" . 3)
>  ("postgresql" . 3)
>  ("elscreen" . 3)
>  ("soundcloud" . 3)
>  ("simplenote" . 3)
>  ("tdd" . 3)
>  ("virtualenv" . 3)
>  ("palette" . 2)
>  ("diredp" . 2)
>  ("e2wm" . 2)
>  ("nyan" . 2)
>  ("remember" . 2)
>  ("swoop" . 2)
>  ("inner" . 2)

RE: Adding a few more finder keywords

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 451 bytes --]

On Apr 25, 2015 8:51 PM, "Drew Adams" <drew.adams@oracle.com> wrote:
>
> > Would it be acceptable for us to add a few more keywords to
> > `finder-known-keywords'?
>
> Good initiative, generally, to revisit the list.  Maybe a few
> "few"er, though?  I'm also not sure how helpful it is for
> `finder-known-keywords' to be large.

Oh dear, I should have been more clear. The list at the end was just some
data I gathered. I don't mean to add them all.

[-- Attachment #2: Type: text/html, Size: 641 bytes --]

Re: Adding a few more finder keywords

[Oleh Krehel]

Artur Malabarba <bruce.connor.am@gmail.com> writes:

> Note how the list contains:
> - 17 "theme" and 18 "themes".
> - 8 "package" and 4 "packages" (and many other instances of singular vs plural)
> - "mode-line", "modeline", "mode line", and even "stausline".
> - 4 "major" and 4 "major-mode"
>
> I think this situation can be improved if we extend the list of
> default keywords. So a new developer, writing his first package,
> doesn't have to decide between "theme" or "themes" (one of them will
> already be offerd to them in the defaults).

I've just added `checkdoc-package-keywords' that checks if the current
file's keywords are in `finder-known-keywords'. It's possible to make
`checkdoc-current-buffer' call `checkdoc-package-keywords'
automatically. I didn't make this the default, since
`finder-known-keywords' still doesn't contain such common keywords as
"org", "completion", "git" etc.

What remains to do now, is to extend `finder-known-keywords', and
possibly make `checkdoc-package-keywords-flag' default to t.

Oleh

RE: Adding a few more finder keywords

[Drew Adams]

> I've just added `checkdoc-package-keywords' that checks if the
> current file's keywords are in `finder-known-keywords'.

Why? There is no reason to signal to users that there might be
a problem if they use keywords that are not in `finder-known-keywords'.

The purpose of `finder-known-keywords' is not to suggest that other
keywords are a mistake.  File-header `Keywords:' is for arbitrary
labels that any user can make use of.  It is not only for "standard"
keywords that might be "known" to Emacs at any state, let alone to
Emacs at startup.

Re: Adding a few more finder keywords

[Oleh Krehel]

Drew Adams <drew.adams@oracle.com> writes:

>> I've just added `checkdoc-package-keywords' that checks if the
>> current file's keywords are in `finder-known-keywords'.
>
> Why? There is no reason to signal to users that there might be
> a problem if they use keywords that are not in `finder-known-keywords'.

It's a matter of checking the current keywords against a list of
keywords that are known to be good. `finder-known-keywords' is a good
start for such a list.

> The purpose of `finder-known-keywords' is not to suggest that other
> keywords are a mistake.  File-header `Keywords:' is for arbitrary
> labels that any user can make use of.  It is not only for "standard"
> keywords that might be "known" to Emacs at any state, let alone to
> Emacs at startup.

I've provided a command that a package developer can voluntarily call to
make sure that the keywords he chose are sound.  It is for the benefit
of all users if >2000 known packages can be organized by a smallish list
of keywords, with no redundant synonyms, typos etc.  If a package author
doesn't want to conform to the keyword guidelines that I propose to
recommend, fine: it's just a recommendation.

Oleh

Re: Adding a few more finder keywords

[Artur Malabarba]

2015-06-08 15:56 GMT+01:00 Oleh Krehel <ohwoeowho@gmail.com>:
> I've just added `checkdoc-package-keywords' that checks if the current
> file's keywords are in `finder-known-keywords'. It's possible to make
> `checkdoc-current-buffer' call `checkdoc-package-keywords'
> automatically. I didn't make this the default, since
> `finder-known-keywords' still doesn't contain such common keywords as
> "org", "completion", "git" etc.
>
> What remains to do now, is to extend `finder-known-keywords', and
> possibly make `checkdoc-package-keywords-flag' default to t.

Besides extending `finder-known-keywords', I think checkdoc should
offer a list variable like `checkdoc-keywords-to-ignore-warnings',
which the developer can use in his personal configs to explicitly
prevent some warnings. Once both are done I would personally be fine
with defaulting the flag to `t`.

Re: Adding a few more finder keywords

[Artur Malabarba]

> Besides extending `finder-known-keywords', I think checkdoc should
> offer a list variable like `checkdoc-keywords-to-ignore-warnings',
> which the developer can use in his personal configs to explicitly
> prevent some warnings. Once both are done I would personally be fine
> with defaulting the flag to `t`.

The idea would then be:

"Hey, you ran checkdoc and it found these unknown keywords. If you
think they're not redundant with any of the keywords in
`finder-known-keywords' you can prevent this warning by adding them to
`checkdoc-keywords-to-ignore-warnings'.

RE: Adding a few more finder keywords

[Drew Adams]

> >> I've just added `checkdoc-package-keywords' that checks if the
> >> current file's keywords are in `finder-known-keywords'.
> >
> > Why? There is no reason to signal to users that there might be
> > a problem if they use keywords that are not in `finder-known-
> > keywords'.
> 
> It's a matter of checking the current keywords against a list of
> keywords that are known to be good. `finder-known-keywords' is a
> good start for such a list.

I know, but experience shows that such "checks", even when intended
to be only helpful and not restrictive, tend to be taken by some
users as indicating what is permissible.  IMHO, this is not helpful
as the default behavior.  Making it optional, and not turned on
by default, lets any user who is savvy and will not be frightened
or misled by such checking signalling phony "problems" can turn it on.

> > The purpose of `finder-known-keywords' is not to suggest that
> > other keywords are a mistake.  File-header `Keywords:' is for
> > arbitrary labels that any user can make use of.  It is not only
> > for "standard" keywords that might be "known" to Emacs at any
> > state, let alone to Emacs at startup.
> 
> I've provided a command that a package developer can voluntarily
> call to make sure that the keywords he chose are sound.

That's good.  I'm all for it.  Voluntary, customizable, and turned
off by default.  And I would probably be in favor of making
`finder-known-keywords' a user option, to facilitate and encourage
user customization of "the keywords he chose".

> It is for the benefit of all users if >2000 known packages can be
> organized by a smallish list of keywords, with no redundant
> synonyms, typos etc.

But you just acknowledged that the list is for "keywords he chose",
that is, keywords that a user can choose, and not just some
predefined "smallish list" of "standard" keywords.

> If a package author doesn't want to conform to the keyword
> guidelines

What guidelines?  File header `Keywords:' is not for package.el.
It predates it by decades, and its use is for arbitrary keywords.

If you want to invent a different file-header keyword from
`Keywords:', for example only for package.el or for your
prescribed more restrictive, guidelined use, then I'm not
opposed to that.  But please do not try to co-opt `Keywords:'
for something different from, and especially more restrictive
than, what it has been intended and used for all of these years.

> that I propose to recommend, fine: it's just a recommendation.

If it comes from GNU Emacs, such a recommendation should not
be applied to existing file-header keyword `Keywords:'.  If you
want to propose a new file-header keyword that has your
recommended use and meaning, please go right ahead.

In sum, OK to propose and check-doc whatever, but not for
the existing `Keywords:' keyword.

RE: Adding a few more finder keywords

[Drew Adams]

> The idea would then be:
> "Hey, you ran checkdoc and it found these unknown keywords. If you
> think they're not redundant with any of the keywords in
> `finder-known-keywords' you can prevent this warning by adding them
> to `checkdoc-keywords-to-ignore-warnings'.

Or to `finder-known-keywords` (which should probably be a user option,
not just a defvar).

Re: Adding a few more finder keywords

[Stefan Monnier]

> I've just added `checkdoc-package-keywords' that checks if the current
> file's keywords are in `finder-known-keywords'.  It's possible to make

A common usage is for a package's keywords to be a list of keywords
(from most generic to most specific), where the most generic should
indeed be among a predefined set, but where the most specific will often
only occur in this one package (and maybe a couple others at most).

In such a case checking that the keywords are within a know list is just
a bad idea.

We could decide that the specific keywords are unwanted, tho.
This would probably work OK if package.el offers a search option that
will search the packages's descriptions (since those specific keywords
will most likely appear in the description as well).

Of course, this search-based alternative can fall flat in some cases.
E.g. for keywords like "url" or "git" which tend to appear in lots of
package descriptions even if those packages have nothing to do with Git
or with handling URLs.


        Stefan

Re: Adding a few more finder keywords

[Stephen J. Turnbull]

Stefan Monnier writes:

 > We could decide that the specific keywords are unwanted, tho.

An "unwanted" keyword doesn't exist though.  Somebody wanted it or it
wasn't in Keywords: in the first place.  And although every human is
unique, very few humans are so unique that they'll choose a keyword
that nobody else would use to look up packages.

So I think what you mean by "unwanted" is mostly "redundant (because a
synonym)".  It seems to me that

1.  There *should* be a list of "recommended keywords" which package
    maintainers can easily access for reference when choosing keywords
    to specify for their packages and users can refer to get an idea
    of the keywords maintainers are likely to use.

2.  There *should* be a database of synonyms of recommended keywords
    for use by maintainers to discover recommended keywords, and for
    *finder* to use in user searches for keywords.  Finder should
    probably divide its report into exact matches for the user's
    keyword and matches discovered via synonyms.

    The schema for this database is unclear to me.  Should there be a
    "similarity" measure to indicate how synonymous two keywords are?
    (Probably a YAGNI.)  Should the primary key of the database be
    restricted to recommended keywords, or perhaps just be the most
    frequently used of a synonym group?  (See point 3 below.)

3.  There should be a tool to walk the libraries producing a Pareto
    distribution of keywords.  Those at the top of the distribution
    would be excellent candidates for the "recommended" list (but
    beware, it's quite possible that two popular keywords could be
    synonyms!).  Those at the bottom would be candidates for addition
    to the database of synonyms and replacement with a recommended
    keyword.

    Probably this tool only needs to be run at release time, and the
    distribution database could be included in etc.

    There's no need to be fascist about keyword maintenance and
    pruning low-frequency keywords that have synonyms, either.  There
    is quite some incentive for maintainers to use user-discoverable
    (ie, recommended) keywords, if you provide the tools so that they
    can find them easily.

Re: Adding a few more finder keywords

[Oleh Krehel]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

> 1.  There *should* be a list of "recommended keywords" which package
>     maintainers can easily access for reference when choosing keywords
>     to specify for their packages and users can refer to get an idea
>     of the keywords maintainers are likely to use.

This is what I was thinking. And it would facilitate adding an option to
filter the package list by tags. I mean not searching (that should be
there too), but browsing. 2000 packages is a lot, but if you narrow to a
"git" tag, you'll get less than 50, which means there's a chance to
browse them all. Similarly, "theme" should be a tag, enabling the user
to browse all available themes. Most common package managers have an
option to narrow to a tag. Emacs could be more flexible to allow e.g. a
"+git +file -theme" tag filter.

To give an example, Synaptic package manager has around 50 sections,
featuring separate sections for R, Haskell, OCaml, PHP, Ruby etc.
So if a user wants to see everything Emacs has to offer on Ruby, he'll
just filter to a "ruby" tag. And then it's the package's author's fault
if he didn't specify the proper tag.

These tags could be also used to make the built-in files more visible.
For instance, I've learned about cmacexp.el on this mailing list. But I
could (and would prefer to) have learned about it by looking at the "c"
tag. Here's the result of git grep:

    lisp/find-file.el:5:             ;; Keywords: c, matching, tools
    lisp/obsolete/cc-compat.el:9:;; Keywords:   c languages
    lisp/progmodes/cc-align.el:13:;; Keywords:   c languages
    lisp/progmodes/cc-bytecomp.el:8:;; Keywords:   c languages
    lisp/progmodes/cc-cmds.el:13:;; Keywords:   c languages
    lisp/progmodes/cc-defs.el:13:;; Keywords:   c languages
    lisp/progmodes/cc-engine.el:13:;; Keywords:   c languages
    lisp/progmodes/cc-fonts.el:9:;; Keywords:   c languages
    lisp/progmodes/cc-guess.el:10:;; Keywords:   c languages oop
    lisp/progmodes/cc-langs.el:13:;; Keywords:   c languages
    lisp/progmodes/cc-menus.el:12:;; Keywords:   c languages
    lisp/progmodes/cc-mode.el:13:;; Keywords:   c languages
    lisp/progmodes/cc-styles.el:13:;; Keywords:   c languages
    lisp/progmodes/cc-vars.el:13:;; Keywords:   c languages
    lisp/progmodes/cmacexp.el:8:;; Keywords: c
    lisp/progmodes/cpp.el:6:;; Keywords: c, faces, tools
    lisp/progmodes/cwarn.el:6:;; Keywords: c, languages, faces
    lisp/progmodes/ebrowse.el:7:;; Keywords: C++ tags tools
    lisp/progmodes/flymake.el:8:;; Keywords: c languages tools
    lisp/progmodes/hideif.el:8:;; Keywords: c, outlines
    lisp/progmodes/hideshow.el:7:;; Keywords: C C++ java lisp tools editing comments blocks hiding outlines
    lisp/tooltip.el:6:;; Keywords: help c mouse tools

It could look a lot more pretty with the package description and
all. And the point is that a person who appreciates C will likely check
out all 22 built-in files that have the "c" tag.

Re: Adding a few more finder keywords

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 363 bytes --]

On Jun 9, 2015 7:52 AM, "Oleh Krehel" <ohwoeowho@gmail.com> wrote:
> This is what I was thinking. And it would facilitate adding an option to
> filter the package list by tags. I mean not searching (that should be
> there too), but browsing.

Like the f key? It doesn't support the "+git +file -theme" filter you
suggest later, but it does filter by single tags.

[-- Attachment #2: Type: text/html, Size: 499 bytes --]

Re: Adding a few more finder keywords

[Oleh Krehel]

Artur Malabarba <bruce.connor.am@gmail.com> writes:

> On Jun 9, 2015 7:52 AM, "Oleh Krehel" <ohwoeowho@gmail.com> wrote:
>> This is what I was thinking. And it would facilitate adding an option to
>> filter the package list by tags. I mean not searching (that should be
>> there too), but browsing. 
>
> Like the f key? It doesn't support the "+git +file -theme" filter you suggest later, but it does filter by single tags.

Almost. Synaptic package manager does more than "f": it shows you tags
in a list (of length ~50). When you click a tag, you narrow.

To make "f" equivalent to what Synaptic offers, there should be a way to
have around 50 prominent completions for it. Currently, the completion
list for "f" on my system has a length 1000. Barely useful for
searching, unusable for browsing. Unless you know the correct tag. Then
it's good.

So two points:

1. have ~50 tags instead of 1000, preferably visually.
2. add all built-in files to the list when a tag is browsed.

The second point is important. Just today, I've read a thread on
emacs-help about iswitchb / icomplete, where it wasn't clear to people
that:

- icomplete provides similar features to iswitchb
- iswitchb is still part of Emacs

I think browsing "completion" tag could have cleared it up: a person, on
hearing iswitchb is obsolete, looks at "completion" tag:

- sees icomplete is there, built-in
- sees iswitchb is still there, built-in
- sees ivy/helm/icicles are also there

RE: Adding a few more finder keywords

[Drew Adams]

>  > We could decide that the specific keywords are unwanted, tho.
> 
> An "unwanted" keyword doesn't exist though.  Somebody wanted it or
> it wasn't in Keywords: in the first place.  And although every human
> is unique, very few humans are so unique that they'll choose a keyword
> that nobody else would use to look up packages.
> 
> So I think what you mean by "unwanted" is mostly "redundant (because
> a> synonym)". It seems to me that
> 
> 1.  There *should* be a list of "recommended keywords" which package
>     maintainers can easily access for reference...
> 2.  There *should* be a database of synonyms of recommended keywords
>     for use by maintainers...
> 3.  There should be a tool to walk the libraries...
>     Probably this tool only needs to be run at release time...
> 
>     There's no need to be fascist about keyword maintenance...

I pretty much agree with all of those points, as being good things
to have.  But possibilities that work only with a set of
"recommended", predefined keywords, e.g. for the package system,
should use a different file-header keyword from `Keywords:'.  You
want something different from what `Keywords:' has always been,
something that conflicts with its usage?  Fine, just add a new
file-header keyword for that.  Happiness all around.

However, just as we should not co-opt `Keywords:', so we should
not co-opt `finder'.  Finder works with `Keywords:' in a particular
way. It is fine to extend finder so that it does additional things,
for particular use cases or in particular environments (e.g., for
Emacs maintainers or for the package system), but what it does
with `Keywords:' should not be cannibalized for the new features.

Again, let it do those things with a new file-header keyword.
If some of the things finder will do are the same, then let it
do them with both `Keywords:' and the new file-header keyword.
IOW, to the extent that some part of the updated finder does not
conflict with the normal interpretation/use of `Keywords:', let
it be used for both.

Should be a no-brainer.  `Keywords:' ain't broke; don't "fix" it.
Feel free to add new features that do something different and
have a different motivation.  But don't bother `Keywords:' just
to implement what you need.  It's not hard for you to leave
`Keywords:' alone, for its original, more flexible, use cases.

Re: Adding a few more finder keywords

[Oleh Krehel]

Drew Adams <drew.adams@oracle.com> writes:

> Again, let it do those things with a new file-header keyword.
> If some of the things finder will do are the same, then let it
> do them with both `Keywords:' and the new file-header keyword.
> IOW, to the extent that some part of the updated finder does not
> conflict with the normal interpretation/use of `Keywords:', let
> it be used for both.

I disagree. Redundancy leads to misuse and under-use. And it's a pain to
parse. Using `Keywords:' for tagging sections is good because most files
need not be touched: they're already fine.

On the other hand, if we introduce something like:

    ;; Section: python

with a tight list of exclusive sections (a file can belong to only one
section), I'd be fine with that. The key here that it needs to be small,
with little room for misinterpretation. As I mentioned before, there are
more than 1000 unique keywords for 2000 packages. Having so many unique
keywords hampers their functionality.

But then again, being too restrictive with `Section:' would probably
lead to 800 packages filed under "convenience". That's why I think that
`Keywords:' are still better. We just need to select a group of keywords
that are deemed "important" and make it easy to see all "important"
keywords at once and browse them.


> Should be a no-brainer.  `Keywords:' ain't broke; don't "fix" it.
> Feel free to add new features that do something different and
> have a different motivation.  But don't bother `Keywords:' just
> to implement what you need.  It's not hard for you to leave
> `Keywords:' alone, for its original, more flexible, use cases.

I don't see how defining a subset of "important" ~50 keywords among the
current ~1000 keywords in use is doing anything against the "more
flexible" use cases.

Basically, I want something like `finder-list-keywords' to work for all
packages managed by package.el as well. I think it would be a great
interface to complement `package-list-packages'. Maybe call it
`package-list-categories'. It currently has 36 sections and looks great.
I wouldn't mind changing my ace-window.el from:

    ;; Keywords: window, location

to

    ;; Keywords: frames, window, location

or even just

    ;; Keywords: frames

to conform to `finder-list-keywords' convention.

RE: Adding a few more finder keywords

[Drew Adams]

> > Again, let it do those things with a new file-header keyword.
> > If some of the things finder will do are the same, then let it
> > do them with both `Keywords:' and the new file-header keyword.
> > IOW, to the extent that some part of the updated finder does not
> > conflict with the normal interpretation/use of `Keywords:', let
> > it be used for both.
> 
> I disagree. Redundancy leads to misuse and under-use.

No one is proposing redundancy.  You are trying to co-opt `Keywords:'
for a different use, effectively imposing/encouraging/warning-about
restrictions on what it should contain.  That's not the same thing -
no redundancy.

> And it's a pain to parse.

Why should parsing your new file-header keyword be any more
painful than parsing the longstanding and differently purposed
keyword `Keywords:'?

> Using `Keywords:' for tagging sections is good because most
> files need not be touched: they're already fine.

Laziness!

And anyway, nothing says that you cannot *also* parse `Keywords:'
for your particular use.  What you should not do is interpret the
meaning of its keywords as only applying to your use case.  You
should not warn users just because you encounter something in
`Keywords:' that you don't recognize or don't like.  You don't
own that field.

IOW, if you want to go ahead and look at `Keywords:', in addition
to your new file-header field, to look for certain "recommended"
keywords, that's your prerogative.

What you should not do is impose on other keywords in `Keywords:'
your use-case interpretation of them being aliens, not recognized,
or illegitimate.

For your new file-header keyword, you can impose any semantics
you like.  But not for `Keywords:'.  Blaring warning sirens
because you find something in `Keywords:' that you don't like
or recognize is a no-no, IMHO.  That's OK for your own field,
but not for the longstanding shared field `Keywords:'.  That
doesn't prevent you from examining `Keywords:', e.g., looking
for keywords that are acceptable to you.

> On the other hand, if we introduce something like:
>     ;; Section: python
> with a tight list of exclusive sections (a file can belong to only
> one section), I'd be fine with that.

Go for it.  A priori, I have no opinion about what you do with
any new file-header keyword you introduce.

> there are more than 1000 unique keywords for 2000 packages.

So what?

> Having so many unique keywords hampers their functionality.

Hampers the particular functionality you have in mind, perhaps.

There are maybe 500 or 1000 different functionalities that users
have created for those 1000 unique keywords - who knows?  And who
cares?  Users are free to use `Keywords:' however they want.

If their uses of `Keywords:' hamper the functionality you have
in mind, there is a very simple solution: you should not expect
`Keywords:' to do only what you want.  It can do anything.

If you need a restrictive, nailed-down file-header keyword then
the simple solution is obvious: come up with your own.  Don't
try to take over the one that the other boys & girls have been
happily sharing for a very long time.

> We just need to select a group of keywords that are deemed
> "important" and make it easy to see all "important"
> keywords at once and browse them.

Just don't do that to `Keywords:'.  Decide such importance for
your own shiny new file-header field.

> I don't see how defining a subset of "important" ~50 keywords among
> the current ~1000 keywords in use is doing anything against the "more
> flexible" use cases.

It's what you intend for the "unimportant" keywords that conflicts
with the longstanding use of `Keywords:'.  You intend to issue
warnings and such (whatever the means used and the effect).

That's not right.  Just create your own sandbox and leave the old,
messy long-shared sandbox alone.  Everyone will be happy.  You will
be able to completely control your new zone, keeping it squeaky
clean, and those dirty kids will still be able to play their same
old scrappy games in the old sandbox.  They won't bother you, and
you won't bother them.

> Basically, I want something like `finder-list-keywords' to work for
> all packages managed by package.el as well.

No problem; go for it.  I would like something like that too.

> I think it would be a great interface to complement
> `package-list-packages'. Maybe call it `package-list-categories'.

Again, I'm all in favor.

You know, `finder.el' is not just about `Keywords:'.  It deals also
with `Commentary:' and does some other things.

And more importantly, `Keywords:' is not just about what `finder.el'
does with it.  `finder.el' is one feature that can make use of
`Keywords:'.  It came along after `Keywords:' existed, as an idea
by ESR of one thing that could be done usefully with existing Emacs
file headers.

It is fine to add functionality to `finder.el', to more specifically
support the new package system.  (It would also be OK to do that in
some other library, instead of `finder.el'.)

What should not happen is that someone comes along and reinterprets
`Keywords:' or `Commentary:' or any longstanding file-header thingy,
imposing new restrictions on what to expect there.  That should be
verboten.

We already saw this happen, unfortunately, to the longstanding
file-header keyword `Version:'.  That free-form field has now
been compromised, by misguided influence from the new package
system.  What should have been done in that case was to add a new
file-header keyword, just for the package system, with whatever
nice, tight restrictions are appropriate for it.  That correct
approach was taken for `Package-Requires:', thank goodness, but
alas not for `Version:'.  We should learn from that mistake.

It is all too tempting to try to co-opt some existing field,
perhaps especially if `finder.el' already does something with it
that you find useful and extensible in your direction.  Ignore
the temptation.  Assume that others are using that existing
field in ways that don't necessarily fit your plans.  Leave it
alone - move on.

If you need something new, then add something new.  Don't
compromise existing constructs that others have been happily
using in ways you don't approve of or cannot make use of.
Share the road.

Re: Adding a few more finder keywords

[Stephen J. Turnbull]

Oleh Krehel writes:

 > with a tight list of exclusive sections (a file can belong to only
 > one section), I'd be fine with that. The key here that it needs to
 > be small, with little room for misinterpretation.

Unfortunately, the interpretation in this case is being done by
humans.  That means that there's enough room for an elephant, let
alone misinterpretation.  Really, Drew is right.  Just choose a
different field name and write the necessary functions to use it, and
see what happens.  I don't think you're likely to get good results,
but who knows?  If you do, and there's demand for merging that
functionality with finder, it won't be hard.  But untangling the two
features if you start by hijacking finder and the Keywords header will
be very difficult, because people will be used to using it that way.

Re: Adding a few more finder keywords

[Oleh Krehel]

Drew Adams <drew.adams@oracle.com> writes:

> If you need something new, then add something new.  Don't
> compromise existing constructs that others have been happily
> using in ways you don't approve of or cannot make use of.
> Share the road.

It seems that a misunderstanding lead you to believe that someone is
enforcing something. I ensure you that this isn't so. There will never
be a warning unless the package author specifically runs an interactive
command because he wants to check if his package will generate a
warning.

Inventing a new section is an option, but it's a cumbersome and
unnecessary path. I can have what I want with just `Keywords:' without
imposing anything on anyone, possibly offering a guideline through a
separate checkdoc utility that so far comes disabled by default.

Let me show you what I have in mind. All the code I wrote just now, I
require nothing more than extending the list of recommended keywords:

(defun finder-add-packages ()
  (let ((keys (delete-dups (package-all-keywords))))
    (dolist (key keys)
      (let ((packages (package-menu--refresh t (list key)))
            (kw (intern key)))
        (puthash kw
                 (delete-dups
                  (append
                   (gethash kw finder-keywords-hash)
                   (mapcar (lambda (x)
                             (package-desc-name (car x)))
                           packages)))
                 finder-keywords-hash)))))

(defun finder-add-keywords ()
  (setq finder-known-keywords
      (mapcar
       (lambda (x)
         (cons x (prin1-to-string x)))
       (hash-table-keys finder-keywords-hash))))

(defun finder-add-good-keywords ()
  (setq finder-known-keywords
        (delete-dups
         (append finder-known-keywords
                 '((python . "Python programming language")
                   (clojure . "Clojure programming language")
                   (ocaml . "OCaml programming language"))))))

Call `finder-add-packages' to add the `package-list-packages' content to
`finder-keywords-hash'.

Now, calling `finder-add-keywords' will cause chaos: "M-x"
`finder-list-keywords' results in a buffer with 1473 lines.

But resetting `finder-known-keywords' to its definition and calling
`finder-add-good-keywords' will extend the amount of sections by 3 to 39
- still a comfortable number.  Now, very conveniently I can browse the
packages available for Python, Clojure and OCaml.  Zero changes in
outside packages. Zero changes to `Keywords:'. Only change is that some
keywords were promoted to `finder-known-keywords', and the package list
was processed into finder (which takes quite long - around 10 seconds).

This is all the functionality that I wanted. Now, I'd like to make this
functionality available to users, with (almost) zero configuration.
What remains to do is to add more than just these 3 keywords to
`finder-add-good-keywords', but not 1200.

RE: Adding a few more finder keywords

[Drew Adams]

> > If you need something new, then add something new.  Don't
> > compromise existing constructs that others have been happily
> > using in ways you don't approve of or cannot make use of.
> > Share the road.
> 
> It seems that a misunderstanding lead you to believe that someone is
> enforcing something. I ensure you that this isn't so. There will
> never be a warning unless the package author specifically runs an
> interactive command because he wants to check if his package will
> generate a warning.

Whether it's a package author or another user, s?he should not
be asking for a test of whether `Keywords:' contains unrecognized
keywords.  S?he should be asking whether some other, new,
package.el-specific field contains unrecognized keywords.

That's the point.  There is no sense in a package author or anyone
else looking to see whether `Keywords:' is "proper".  Doing what
you suggest will only encourage package authors to restrict
`Keywords:' to "proper" keywords.  That is misguided, is what I
am arguing.

On the other hand, it is entirely useful for package authors to
check for unrecognized package keywords.  That checking should
not be done against `Keywords:'.  That's all.

The feature you want to provide is something I've already said
I am in favor of.  The need for package authors to check for
unrecognized package keywords is a real need.  And a warning
when a package author checks for that is entirely appropriate.
Your new feature will be a welcome addition.

What you do not seem to get is that it is not `Keywords:' that
you and package authors should be using for this.  That's all.

> Inventing a new section is an option, but it's a cumbersome

Tough tiddlywinks.  Others got there before you.

That part of the prairie has already been settled.  If you want
to live there too, then live by the same wild-west rules as the
longtime inhabitants.  No one has asked for a new sheriff with
new rules.  You might find this locale dirty, messy, chaotic,
and confusing.  But that's what the settlers of `Keywords:'
had in mind, and that's they way they've developed it.  Think
Rio de Janeiro, not Brasilia.  This is not virgin territory.

> and unnecessary path.

It's not unnecessary.  It's necessary, if you (as I do) want
to preserve `Keywords:' for what it's been all along: a place
for arbitrary keywords, invented by anyone, for any purpose
whatsoever.

> I can have what I want with just `Keywords:' without imposing
> anything on anyone,

In my book, discouraging and warning people about "improper"
keywords in `Keywords:' is imposing.  That kind of policing (or
kindly "suggesting") does not belong in `Keywords:'.  Please
take it elsewhere.  That's all I'm asking.

> possibly offering a guideline through a separate checkdoc utility
> that so far comes disabled by default.

All well and good.  Just please take it elsewhere from `Keywords:'.