Improving browsing and discoverability in the Packages Menu

Improving browsing and discoverability in the Packages Menu

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 1060 bytes --]

The package menu is fantastic when you know what you're looking for. You
just search for the package name or for a phrase, and you're likely to find
it.

It's not that great when you just want to browse, specially if you have
Melpa and Marmalade configured, simply because the menu is overwhelming. A
few items that stand out to me:

1. The package list is ginormous and growing.
2. There are duplicate entries galore.
3. There's no useful sense of categories. (we have keywords,but see below).
4. Are there other points that people feel can be improved?

I can see myself writing something to improve 2.

Any solution for 3 would also be a solution for 1, but I'm not sure what
that would be.

Also regarding 3: We have keywords but they are not the same thing as
categories.
I'm not saying we should come up with yet another classification, but I'm
saying that keywords need some serious work. It's not obvious to the user
what keywords are available or how to browse them. And they're sort of all
over the place with a lot of overlap.

Cheers to all,
Artur

[-- Attachment #2: Type: text/html, Size: 1250 bytes --]

Re: Improving browsing and discoverability in the Packages Menu

[Stefan Monnier]

> 1. The package list is ginormous and growing.
> 2. There are duplicate entries galore.
> 3. There's no useful sense of categories. (we have keywords,but see below).
> 4. Are there other points that people feel can be improved?

Indeed, that's a known problem.

> I can see myself writing something to improve 2.

Great, go for it.

> Any solution for 3 would also be a solution for 1, but I'm not sure what
> that would be.

1 and 3 are pretty closely related and are the main issue for me.

My best idea so far would be to write a "filter" command which would
work as follow:
- you type in the minibuffer some word
- the word is matched against the packages's descriptions and keywords.
- as you type the word, the list of packages gets filtered to only those
  that match.
- as you type the word, a list of possible completions is
  displayed, in icompletion-mode style.


        Stefan

Re: Improving browsing and discoverability in the Packages Menu

[raman]

Many ways of filtering -- over and above the present "filter by
keywords" would be nice.

1. Hide all installed/builtin packages 
2. Hide packages from one or more repositories 
3. Hide helper packages eg packages  that would only be installed as a
dependency of something else that is user-facing 

Re: Improving browsing and discoverability in the Packages Menu

[Ted Zlatanov]

On Wed, 18 Feb 2015 08:23:51 -0800 raman <raman@google.com> wrote: 

r> Many ways of filtering -- over and above the present "filter by
r> keywords" would be nice.

r> 1. Hide all installed/builtin packages 
r> 2. Hide packages from one or more repositories 

(Sorry for the late comment, I've been away.  I don't think this has
happened, though?)

It would be easiest, I think, to make the source repo, the package
status, and other metadata (signed, etc.) into keywords and then enhance
the keyword browsing.  The "filter" functionality Stefan suggested makes
sense across keywords *and* package names and descriptions, and can
easily be plugged into the current "Filter" function in the menu.

r> 3. Hide helper packages eg packages  that would only be installed as a
r> dependency of something else that is user-facing 

I don't think it's good to ever hide those, but maybe they could be
given the "required-by=user-level-package" tags (multiples would be OK)?
And then in the UI they could be presented in a visually different way.
But hiding them would be a problem if you end up installing things you
didn't want.

Ted

Re: Improving browsing and discoverability in the Packages Menu

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 1160 bytes --]

> r> Many ways of filtering -- over and above the present "filter by
> r> keywords" would be nice.
>
> r> 1. Hide all installed/builtin packages
> r> 2. Hide packages from one or more repositories
>
> (Sorry for the late comment, I've been away.  I don't think this has
> happened, though?)

It hasn't.

> It would be easiest, I think, to make the source repo, the package
> status, and other metadata (signed, etc.) into keywords and then enhance
> the keyword browsing.  The "filter" functionality Stefan suggested makes
> sense across keywords *and* package names and descriptions, and can
> easily be plugged into the current "Filter" function in the menu.

Sounds very reasonable. We could also implement excluding keywords. So it
would be easy to hide nil built-ins, for instance.

> r> 3. Hide helper packages eg packages  that would only be installed as a
> r> dependency of something else that is user-facing
>
> I don't think it's good to ever hide those,

This part is already done. Dependencies are not hidden, but are given a
comment face and sorted after the installed packages.

If the above is implemented, it should also be easy to hide them.

[-- Attachment #2: Type: text/html, Size: 1479 bytes --]

Re: Improving browsing and discoverability in the Packages Menu

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 1160 bytes --]

> r> Many ways of filtering -- over and above the present "filter by
> r> keywords" would be nice.
>
> r> 1. Hide all installed/builtin packages
> r> 2. Hide packages from one or more repositories
>
> (Sorry for the late comment, I've been away.  I don't think this has
> happened, though?)

It hasn't.

> It would be easiest, I think, to make the source repo, the package
> status, and other metadata (signed, etc.) into keywords and then enhance
> the keyword browsing.  The "filter" functionality Stefan suggested makes
> sense across keywords *and* package names and descriptions, and can
> easily be plugged into the current "Filter" function in the menu.

Sounds very reasonable. We could also implement excluding keywords. So it
would be easy to hide nil built-ins, for instance.

> r> 3. Hide helper packages eg packages  that would only be installed as a
> r> dependency of something else that is user-facing
>
> I don't think it's good to ever hide those,

This part is already done. Dependencies are not hidden, but are given a
comment face and sorted after the installed packages.

If the above is implemented, it should also be easy to hide them.

[-- Attachment #2: Type: text/html, Size: 1479 bytes --]

Re: Improving browsing and discoverability in the Packages Menu

[raman]

Another valuable filtering capability would be to group and filter by
"bundles" e.g. the large collection o f"helm-<xxx>", "copany-<xxx>" etc
could collapse into a single "bundle" that could be expanded and collapsed.
-- 

Re: Improving browsing and discoverability in the Packages Menu

[Ted Zlatanov]

On Sat, 18 Apr 2015 14:11:20 -0700 raman <raman@google.com> wrote: 

r> Another valuable filtering capability would be to group and filter by
r> "bundles" e.g. the large collection o f"helm-<xxx>", "copany-<xxx>" etc
r> could collapse into a single "bundle" that could be expanded and collapsed.

Who would define these bundles, maintain their definitions, pin packages
to each others, and test them to ensure they are not broken?

Or do you mean simply everything that begins with "helm-" is in the helm
bundle?  I don't think that's very useful, you'd end up installing a
boatload of useless packages.

Ted

Re: Improving browsing and discoverability in the Packages Menu

[Alexis]

Ted Zlatanov <tzz@lifelogs.com> writes:

> Or do you mean simply everything that begins with "helm-" is in 
> the helm bundle?

That's what i assumed was being proposed.

> I don't think that's very useful, you'd end up installing a 
> boatload of useless packages.

i would find it very useful just to be able to visually 'hide' 
things i consider to be clutter whilst browsing. For instance: i 
no longer use the `autocomplete` package, so all the packages 
beginning with ac- are of no interest to me.

(Bonus points for functionality to allow avoiding searching in 
such 'hidden' sections.)


Alexis.

Re: Improving browsing and discoverability in the Packages Menu

[Eric Abrahamsen]

Artur Malabarba <bruce.connor.am@gmail.com> writes:


[...]

> Also regarding 3: We have keywords but they are not the same thing as
> categories. 
> I'm not saying we should come up with yet another classification, but
> I'm saying that keywords need some serious work. It's not obvious to
> the user what keywords are available or how to browse them. And
> they're sort of all over the place with a lot of overlap.

This would be nice to fix. `finder-known-keywords' looks like the place
to start. Package filtering is only indirectly connected to the value of
that variable, though.

Filtering is done on keywords actually found in the packages. If package
authors used `auto-insert' when creating their packages, they would be
prompted to add keywords from `finder-known-keywords'. That leaves a lot
of wiggle room for the insertion of random keywords.

In order to make filtering useful, it seems like it would be worthwhile
fleshing out the taxonomy of `finder-known-keywords', and enforcing it
-- ie, keywords not in that variable would be ignored by package
filtering.

Eric

Re: Improving browsing and discoverability in the Packages Menu

[Stefan Monnier]

> i would find it very useful just to be able to visually 'hide' things
> i consider to be clutter whilst browsing. For instance: i no longer use the
> `autocomplete` package, so all the packages beginning with ac- are of no
> interest to me.

Maybe one way to hide those "ac-*" packages is to let the user state "I
won't install package <foo>" so that all packages which (transitively)
require <foo> get greyed out or hidden.

We already have something similar in the treatment of packages which
require a version of Emacs higher than the current one.


        Stefan

Re: Improving browsing and discoverability in the Packages Menu

[Alexis]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

>> i would find it very useful just to be able to visually 'hide' 
>> things i consider to be clutter whilst browsing. For instance: 
>> i no longer use the `autocomplete` package, so all the packages 
>> beginning with ac- are of no interest to me.
>
> Maybe one way to hide those "ac-*" packages is to let the user 
> state "I won't install package <foo>" so that all packages which 
> (transitively) require <foo> get greyed out or hidden.

i'd certainly be happy with that also. :-)


Alexis.

RE: Improving browsing and discoverability in the Packages Menu

[Drew Adams]

> This would be nice to fix. `finder-known-keywords' looks like the
> place to start. Package filtering is only indirectly connected to the
> value of that variable, though.
> 
> Filtering is done on keywords actually found in the packages. If
> package authors used `auto-insert' when creating their packages,
> they would be prompted to add keywords from `finder-known-keywords'.
> That leaves a lot of wiggle room for the insertion of random keywords.

I'm not sure what you meant by the last part, so this comment might
be unrelated to your post.  But I would like to remind us all that
`finder' functionality is not limited to the keywords found in
`finder-known-keywords'.

Users can use any keywords they like in their library headers.
I would not like to see `finder.el' gravitate toward imposing some
particular set of keywords or expecting only some particular set.

> In order to make filtering useful, it seems like it would be
> worthwhile fleshing out the taxonomy of `finder-known-keywords',
> and enforcing it -- ie, keywords not in that variable would be
> ignored by package filtering.

I'm not sure I would object to something like that being done only
"by package filtering".  But I would object to changing the point
of `finder.el' so that it generally becomes more rigid in that way.

Even for only "package filtering", I have the question of why.
Why should package filtering be limited to some predefined list
of keywords?  Just why should "keywords not in that variable...
be ignored by package filtering"?

Re: Improving browsing and discoverability in the Packages Menu

[raman]

I meant something light-weight like "helm-" as a prefix. I didn't mean
we'd use these "bundles" to install a bunch of packages at one shot --
that would creat bloat as you say. I meant it purely as a means for
reducing the number of entries in the packages buffer.
-- 

Re: Improving browsing and discoverability in the Packages Menu

[raman]

transitively hiding dependencies of uninteresting packages is harder --
A might be uninteresting and depend on A_d; but hiding A_d  should not
confuse B if B also depends on A_d.
-- 

Re: Improving browsing and discoverability in the Packages Menu

[Ted Zlatanov]

On Sun, 19 Apr 2015 09:12:28 -0700 raman <raman@google.com> wrote: 

r> I meant something light-weight like "helm-" as a prefix. I didn't mean
r> we'd use these "bundles" to install a bunch of packages at one shot --
r> that would creat bloat as you say. I meant it purely as a means for
r> reducing the number of entries in the packages buffer.

Oh, that's just a tree widget then, with the branching determined by
node names.

I don't think the current `tabulated-list-mode' supports a tree, and
writing a new `tabulated-tree-mode' is probably better than trying to
retrofit trees into lists.

I'm still not sure hiding helm-* under a single helm header would be
useful.  It feels artificial.  It's probably better to distinguish
between a "plugin" for a package (which you do want to group under it)
and an "extension" to a package (which may or may not merit its own
top-level position).

Ted

Re: Improving browsing and discoverability in the Packages Menu

[Stefan Monnier]

> transitively hiding dependencies of uninteresting packages is harder --

I'm talking about the reverse: hiding packages that depend on FOO,
rather than hiding dependencies of FOO.


        Stefan

Re: Improving browsing and discoverability in the Packages Menu

[Eric Abrahamsen]

Drew Adams <drew.adams@oracle.com> writes:

>> This would be nice to fix. `finder-known-keywords' looks like the
>> place to start. Package filtering is only indirectly connected to the
>> value of that variable, though.
>> 
>> Filtering is done on keywords actually found in the packages. If
>> package authors used `auto-insert' when creating their packages,
>> they would be prompted to add keywords from `finder-known-keywords'.
>> That leaves a lot of wiggle room for the insertion of random keywords.
>
> I'm not sure what you meant by the last part, so this comment might
> be unrelated to your post.  But I would like to remind us all that
> `finder' functionality is not limited to the keywords found in
> `finder-known-keywords'.
>
> Users can use any keywords they like in their library headers.
> I would not like to see `finder.el' gravitate toward imposing some
> particular set of keywords or expecting only some particular set.
>
>> In order to make filtering useful, it seems like it would be
>> worthwhile fleshing out the taxonomy of `finder-known-keywords',
>> and enforcing it -- ie, keywords not in that variable would be
>> ignored by package filtering.
>
> I'm not sure I would object to something like that being done only
> "by package filtering".  But I would object to changing the point
> of `finder.el' so that it generally becomes more rigid in that way.
>
> Even for only "package filtering", I have the question of why.
> Why should package filtering be limited to some predefined list
> of keywords?  Just why should "keywords not in that variable...
> be ignored by package filtering"?

You're right, "enforcing" anything is probably a bad idea. But looking
at the lists presented to you when filtering packages (or when doing an
auto-insert in a new elisp file), it looks a lot like tag-rot: a whole
lot of random overlapping categories and mis-matched types. I guess I
agree that package authors should be able to put anything they want in
the Keyword header. On the other hand, I want to be able to search for
packages related to sending email messages from Emacs (for instance),
and know that I'm not missing packages because their author tagged them
as "email" instead of "mail". That sort of thing just makes the whole
process feel broken.

Maybe just expand the canonical list of curated keywords, then let
package authors add their own on top of that?

Eric

Re: Improving browsing and discoverability in the Packages Menu

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 297 bytes --]

> I don't think the current `tabulated-list-mode' supports a tree, and
> writing a new `tabulated-tree-mode' is probably better than trying to
> retrofit trees into lists.

This would be very nice to have. It would also allow us to display a single
entry for packages which are on multiple repos.

[-- Attachment #2: Type: text/html, Size: 359 bytes --]

Re: Improving browsing and discoverability in the Packages Menu

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 769 bytes --]

> On the other hand, I want to be able to search for
> packages related to sending email messages from Emacs (for instance),
> and know that I'm not missing packages because their author tagged them
> as "email" instead of "mail".

Agreed. This is precisely why the current state is a mess.

> Maybe just expand the canonical list of curated keywords, then let
> package authors add their own on top of that?

This by itself will take a very long time, if ever, to have any effect. It
would have to be coupled with something else.
For instance, there could be a compiler warning if the package has unknown
keywords. This wouldn't prevent the keyword from working, but it would
inform the developer to either change the keyword or ask us to add it in.
Is that too much?

[-- Attachment #2: Type: text/html, Size: 909 bytes --]

Re: Improving browsing and discoverability in the Packages Menu

[Alexis]

Artur Malabarba <bruce.connor.am@gmail.com> writes:

>> Maybe just expand the canonical list of curated keywords, then 
>> let package authors add their own on top of that?
>
> This by itself will take a very long time, if ever, to have any 
> effect. It would have to be coupled with something else.  For 
> instance, there could be a compiler warning if the package has 
> unknown keywords. This wouldn't prevent the keyword from 
> working, but it would inform the developer to either change the 
> keyword or ask us to add it in.  Is that too much?

Hmm .... i'm not sure how practical that last bit is. For example: 
should "vcard" be added as an 'official' keyword? (i currently 
include it in the list of keywords for `org-vcard', the other two 
being "outline" and "org".) Should i instead replace the "vcard" 
keyword with "contacts", given that people could plausibly find 
`org-vcard' via searching package names instead of keywords for 
the term 'vcard', but currently wouldn't find it were they 
thinking in terms of "contacts management"? Who would be the 
person or people making the decision on whether a keyword should 
be added, modified or removed? Might such a process result in 
emacs-devel being flooded with interminable ontological 
bikeshedding?


Alexis.

Re: Improving browsing and discoverability in the Packages Menu

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 1666 bytes --]

On Apr 20, 2015 12:17 PM, "Alexis" <flexibeast@gmail.com> wrote:
>
>
>> This by itself will take a very long time, if ever, to have any effect.
It would have to be coupled with something else.  For instance, there could
be a compiler warning if the package has unknown keywords. This wouldn't
prevent the keyword from working, but it would inform the developer to
either change the keyword or ask us to add it in.  Is that too much?
>
> Hmm .... i'm not sure how practical that last bit is.

Me neither. :-) just throwing ideas around.

> For example: should "vcard" be added as an 'official' keyword? (i
currently include it in the list of keywords for `org-vcard', the other two
being "outline" and "org".)

I'd say yes.

>  Should i instead replace the "vcard" keyword with "contacts", given that
people could plausibly find `org-vcard' via searching package names instead
of keywords for the term 'vcard', but currently wouldn't find it were they
thinking in terms of "contacts management"?

Use both.

> Who would be the person or people making the decision on whether a
keyword should be added, modified or removed?

We would just accept any keyword that doesn't already have an obvious
equivalent. The point is not to be restrictive, but to prevent duplication
like mail/email. We could also have some restriction like accepting at most
1 (or 2? 3?) new keyword request per package.

> Might such a process result in emacs-devel being flooded with
interminable ontological bikeshedding?

Possibly. New packages are created at a rate of a few per day. It's hard to
say how many new keywords that entails. Somebody could estimate it based on
Melpa git history.

[-- Attachment #2: Type: text/html, Size: 2067 bytes --]

Re: Improving browsing and discoverability in the Packages Menu

[Eric Abrahamsen]

Alexis <flexibeast@gmail.com> writes:

> Artur Malabarba <bruce.connor.am@gmail.com> writes:
>
>>> Maybe just expand the canonical list of curated keywords, then let
>>> package authors add their own on top of that?
>>
>> This by itself will take a very long time, if ever, to have any
>> effect. It would have to be coupled with something else.  For
>> instance, there could be a compiler warning if the package has
>> unknown keywords. This wouldn't prevent the keyword from working,
>> but it would inform the developer to either change the keyword or
>> ask us to add it in.  Is that too much?
>
> Hmm .... i'm not sure how practical that last bit is. For example:
> should "vcard" be added as an 'official' keyword? (i currently include
> it in the list of keywords for `org-vcard', the other two being
> "outline" and "org".) Should i instead replace the "vcard" keyword
> with "contacts", given that people could plausibly find `org-vcard'
> via searching package names instead of keywords for the term 'vcard',
> but currently wouldn't find it were they thinking in terms of
> "contacts management"? Who would be the person or people making the
> decision on whether a keyword should be added, modified or removed?
> Might such a process result in emacs-devel being flooded with
> interminable ontological bikeshedding?

The danger is great. I don't have a wonderful solution, either, but I'm
seeing a vague vision of some sort of a two-tiered system: the first a
limited list of curated categories set by someone in a position of
authority, and another free-for-all keyword mashup. In your example,
you'd categorize your package as "contacts" under tier one, and put
"vcard" and "org" and whatever else in tier two. Package filtering would
happen on both tiers, perhaps with completion only for the first tier.

I can imagine two different package headers: Category and Keywords,
corresponding to the two tiers. I can also imagine tier one coming from
Keywords, and tier two coming from searching the Commentary section.

That's all I've got.

Eric

RE: Improving browsing and discoverability in the Packages Menu

[Drew Adams]

> We would just accept any keyword that doesn't already have...

"We" is what here, exactly?  Just the use of keywords by
`list-packages' (or other package viewing/filtering code)?

If so, I don't really care what you do in that regard.  I assume
it could be an improvement.

But if "we" here means Emacs in general or hints at a redirection
of the intention and flexible Emacs design of header keywords
and finder.el, then I'm not in favor.  Please solve your viewing
or filtering problems without touching file keywords or finder.el.

Do what you like on the viewing/filtering end, for package listing.
But please don't mess with the intention of file-header keywords,
which is *any* intention that anyone wants to give them, and which
at least includes *all* that finder.el can do with them.

> The point is not to be restrictive, but to prevent duplication
> like mail/email.

And what if someone has code that *wants* to distinguish keyword
`mail' from keyword `email'?  One person's (or one app's) handy
duplicate prevention can block another from taking advantage of
useful distinctions.  Eliminate what you consider duplication
at the consumer end, by filtering.  Please do not attempt to
impose a limitation on the source (`Keywords').  Filter; do not
"prevent".

Please don't restrict your thinking to your immediate task of
improving the `list-packages' UI.  Not if you are proposing
restrictions on header keywords or on what finder.el can do.

Filtering should not require redefining and redesigning the
things that are being filtered.  If the packages UI can't
figure out a way to easily let users match both `mail' and
`email', instead of preventing such "duplication" at the
source, then we're in deep trouble.  There should be no
requirement that the source code limit the keywords that
can be handled.

> We could also have some restriction like accepting at most
> 1 (or 2? 3?) new keyword request per package. 

What is a "keyword request" here?  Are you speaking only of
what a `list-packages' UI user can request?  Who or what is
requesting a keyword from who or what?

> Might such a process result in emacs-devel being flooded
> with interminable ontological bikeshedding?

Why should what can end up in a file-header `Keywords' list
concern emacs-devel at all?

Leave it alone.  Figure out whatever you like for users to 
access/search/filter/view keywords for a list of packages,
but please do not try to impose restrictions on what users
and other code can use for file-header keywords.

Emacs is always much bigger than the part of it that you are
currently thinking of improving.

(And yes, it is bigger than what is developed by emacs-devel
and distributed as part of GNU Emacs.)

> Possibly. New packages are created at a rate of a few per
> day. It's hard to say how many new keywords that entails.

It shouldn't matter.  At all.  Even just for filtering for
the UI that lists packages, IMO.  But especially beyond that
*one* use of file-header keywords.

File headers are not only for package.el (and not only for
finder.el, for that matter).  Package.el has come late to
the file-headers (including `Keywords') party.

But it is welcome to join, as long as it respects the other
partygoers.  And it doesn't matter how many packagers it
brings through the front door.  They should realize that
the party is not only for them.

They could of course start their own party elsewhere, using
their own `Package Keywords' or some such.  Or they can just
learn to get along with others at the same `Keywords' party.

(There have already been attempts to give `Version' a new,
restricted interpretation.  Fortunately, so far such
restrictions apply only to use by package.el, and not to
some general redefinition of what `Version' might be for.)

> Somebody could estimate it based on Melpa git history.

Waste of time, IMO.  And likely to misguide.

Re: Improving browsing and discoverability in the Packages Menu

[raman]

Ability to handle trees within tabulated lists would be nice. An
intermediate solution that might be quicker to build would be dired-like
marking by regexp and hiding/exposing. 
-- 

RE: Improving browsing and discoverability in the Packages Menu

[Drew Adams]

> Ability to handle trees within tabulated lists would be nice. An
> intermediate solution that might be quicker to build would be dired-
> like marking by regexp and hiding/exposing.

Not just intermediate, but better, IMO.

Markings can be used for lots of things, just as they are in Dired.

Currently, IIUC, the `list-packages' UI provides only for markings
that mean "install" and "delete".  A more general marking facility,
a la Dired, would be an improvement, and would invite future
enhancements that use the same (and familiar) UI.

RE: Improving browsing and discoverability in the Packages Menu

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 1090 bytes --]

> > We would just accept any keyword that doesn't already have...
>
> "We" is what here, exactly?  Just the use of keywords by
> `list-packages' (or other package viewing/filtering code)?

The idea is not to restrict or prevent anything, just recommend.
We would extend the list of known finder.el keywords. When the byte
compiler is compiling a package, it could issue a warning if it notices
keywords that are not part of the known list.
All keywords would keep working just the same (even those not on the list).
This warning would be the only change here.

The intention was to nudge developers towards avoiding useless duplicates.
We wouldn't enforce anything. Obviously this list of keywords that don't
issue warnings would need to grow with time. Which is the most cumbersome
aspect here.

Let me clarify, I don't think this idea should be implemented as is. I'm
just throwing thoughts out there in the hopes we can converge to something
useful.
I agree the keywords system should never become restricted, but I do think
we need to help developers towards standardising it a little.

[-- Attachment #2: Type: text/html, Size: 1240 bytes --]

RE: Improving browsing and discoverability in the Packages Menu

[Drew Adams]

(Sure wish you would send your emails as plain text, BTW.)

> > > We would just accept any keyword that doesn't already have...
> >
> > "We" is what here, exactly?  Just the use of keywords by
> > `list-packages' (or other package viewing/filtering code)?
> The idea is not to restrict or prevent anything, just recommend.
> We would extend the list of known finder.el keywords. When the byte
> compiler is compiling a package, it could issue a warning if it
> notices keywords that are not part of the known list.

I for one (perhaps the only one) would object to that.  There is no
reason to "warn" users about libraries that are not doing anything
even potentially wrong - and that includes using `Keywords' in a
file header that you might never have heard of.

Again.  File headers with keyword `Keywords' have been around for
decades.  `Keywords' is for any keywords whatsoever.  It does not
belong to package.el.

I suggested, and suggest again, that if you are going to take such
a proprietary and controlling attitude toward package.el-recognizable
keywords then package.el should add and use its own file-header
keyword, e.g., `Package Keywords' - and leave the long-existing (and
used) keyword `Keywords' alone.

> All keywords would keep working just the same (even those not on
> the list).

"Warning" users about keywords you don't recognize does not mean
that things "keep working just the same".  Forget about the warning,
if you want to pretend that things will "keep working just the same".

> This warning would be the only change here.

That's one change too much.  Have package.el do it in its own
backyard: `Package Keywords'.  Or don't have it do it at all.

> The intention was to nudge developers towards avoiding useless
> duplicates. 

Duplicates as far as package.el is concerned.  That doesn't mean
duplicates as far as every use/program is concerned.

I have no problem with package.el doing whatever it wants, in its
own territory.  But if it wants to start using existing keywords,
such as `Keywords', then it needs to play well with the others
who are already in the same sandbox.  It can't just move in and
claim the sandbox for its own.

> We wouldn't enforce anything.

The warning is uncalled for, if this is about `Keywords'.  It
can be appropriate, if it is about some new file-header keyword.

> I agree the keywords system should never become restricted, but
> I do think we need to help developers towards standardising it
> a little.

The problem is that you are referring to "the keywords system".
And you are trying to shove it into a particular use case.

File-header fields (keywords) that have been around for quite
some time should be left as is (and this should have included
`Version').  You can add whatever other fields you like, and
make their use as restrictive or helpful with as many warnings
or preventions as you like.  No problem.

Re: Improving browsing and discoverability in the Packages Menu

[Artur Malabarba]

2015-04-20 19:17 GMT+01:00 Drew Adams <drew.adams@oracle.com>:
> (Sure wish you would send your emails as plain text, BTW.)

Haven't figured out how to that on my phone yet (so I'm switching to
the laptop just for you ;-). If it's any consolation, your pre-filled
emails look terrible on my phone too. :-)

> I for one (perhaps the only one) would object to that.  There is no
> reason to "warn" users about libraries that are not doing anything
> even potentially wrong - and that includes using `Keywords' in a
> file header that you might never have heard of.
> [...]
> The problem is that you are referring to "the keywords system".
> And you are trying to shove it into a particular use case.
>
> File-header fields (keywords) that have been around for quite
> some time should be left as is (and this should have included
> `Version').  You can add whatever other fields you like, and
> make their use as restrictive or helpful with as many warnings
> or preventions as you like.  No problem.

Thanks. I do see your point in all of that.I just wish we had some way
to polish things out a little bit.f This not even for the sake of
package.el, but for the keywords as a whole.

For instance, there are 3 different mode-line keywords: mode-line,
modeline, and mode line. The last two contain a single package each,
both of which will not show up if the user looks for the "mode-line"
keword.
A similar thing happens with `package' vs `packages', application vs
applications, buffer vs buffers, bookmark vs bookmarks, etc.
Not to mention a number of keywords which are just plain senseless
like: "$", "$date:", "/", "09:41:07", "2015/01/12", and ":".

These don't just affect the package menu. These also pollute the
finder keywords menu, and pretty much anywhere else they show up.

I understand a warning may not be the way of doing that, but it would
be nice if we could help developers standardize these a little. And
I'll appreciate further suggestions.
It's clear that whoever used "modeline" as a keyword simply didn't
know that mode-line is the commonly used one (and why should they
know, it's not in `finder-list-keywords'). And if they do (for
whatever reason) have the intention of using modeline instead of
mode-line, that's fine. I just want to avoid the accidental duplicates
by making developers more aware of the already existing keywords. I'm
don't want to prohibit duplicates or anything.

RE: Improving browsing and discoverability in the Packages Menu

[Drew Adams]

> > (Sure wish you would send your emails as plain text, BTW.)
> 
> Haven't figured out how to that on my phone yet (so I'm switching to
> the laptop just for you ;-). 

Thank you.  (And I'll bet I'm not the only one who appreciates it.)

> If it's any consolation, your pre-filled emails look terrible on
> my phone too. :-)

If it's any consolation, they're not even prefilled.  I do it myself.

> Thanks. I do see your point in all of that.I just wish we had some way
> to polish things out a little bit.f This not even for the sake of
> package.el, but for the keywords as a whole.

I would say to concentrate on what package.el needs.  And in particular,
to consider having it do whatever it thinks helps it, but in its own
header keyword (field), not in `Keywords'.  IOW, start clean, from
scratch, and impose any rules and checking you want.

> For instance, there are 3 different mode-line keywords: mode-line,
> modeline, and mode line. The last two contain a single package each,
> both of which will not show up if the user looks for the "mode-line"
> keword...

Yes, it's a bother.  But there is no way to know what is a typo or
negligence and what is intentional difference, e.g. required by some
particular code or context.

That's why I suggest that you start out with a new, fresh field,
which you announce as pretty much dedicated to package.el and which
will be controlled by it.

With that announcement, you are free to ignore or correct or raise
an error for anything that doesn't fit the mold.

And with a little encouragement from such checks (e.g. errors),
library authors will learn to DTRT.

My only point was that it is not wise to try to usurp `Keywords'
for this.  Too much history, too long.  Too much out there already,
at least potentially, in the form of various usages and expectations.

> I understand a warning may not be the way of doing that, but it would
> be nice if we could help developers standardize these a little. And
> I'll appreciate further suggestions.

See above.  Hands off `Keywords' and you're free to do whatever
you think helps in this regard.

Re: Improving browsing and discoverability in the Packages Menu

[raman]

:-) I'd love to be able to apply custom marks -- simple regexp on
package name  or repo would do -- then one could hide, unhide, install
delete or whatever else.
-- 

RE: Improving browsing and discoverability in the Packages Menu

[Drew Adams]

> :-) I'd love to be able to apply custom marks -- simple regexp on
> package name  or repo would do -- then one could hide, unhide,
> install delete or whatever else.

Yes, and not just apply custom marks (as in Dired, where you can
use any marks you like, not just `*').  But much of the rest of
Dired's handling of marks (and deletion flags) too.

This includes `M-DEL' (and `C-u M-DEL'), which removes a specific
mark (or any marks) from each entry.

Dired should (in general) be a good model to follow, for general
stuff that is not file-specific.  This includes omitting,
navigating, operating on marked entries in various ways, hiding
details, etc.

If you end up grouping packages, a Dired analogy for that could
be directories (which group files, and even do so at any number
of levels).

Let users include ("insert") group listings, jump back and forth
between the group name and its listing, etc.

The Dired UI is pretty good - tried & true.

Re: Improving browsing and discoverability in the Packages Menu

[Tassilo Horn]

Drew Adams <drew.adams@oracle.com> writes:

>> For instance, there are 3 different mode-line keywords: mode-line,
>> modeline, and mode line. The last two contain a single package each,
>> both of which will not show up if the user looks for the "mode-line"
>> keword...
>
> Yes, it's a bother.  But there is no way to know what is a typo or
> negligence and what is intentional difference, e.g. required by some
> particular code or context.

If it were intended by the package author, where is the problem of
adding

  ;; byte-compile-warnings: (not unknown-keyword)

to the package's local variables section?

> That's why I suggest that you start out with a new, fresh field, which
> you announce as pretty much dedicated to package.el and which will be
> controlled by it.

I don't see why `Keywords' shouldn't be appropriate for elisp programs
installable via package.el.  It's not that it re-defines its meaning.
The benefit of using it is that it's already there most of the time.  A
new field would take some time to get picked up.  And I'm pretty sure
you won't see any occurrence of

  ;; Keywords: modeline, electronic mail
  ;; Package Keywords: mode-line, email

even if there was a new field.

Bye,
Tassilo

RE: Improving browsing and discoverability in the Packages Menu

[Drew Adams]

> >> For instance, there are 3 different mode-line keywords: mode-line,
> >> modeline, and mode line. The last two contain a single package each,
> >> both of which will not show up if the user looks for the "mode-line"
> >> keword...
> >
> > Yes, it's a bother.  But there is no way to know what is a typo or
> > negligence and what is intentional difference, e.g. required by some
> > particular code or context.
> 
> If it were intended by the package author, where is the problem of
> adding
>   ;; byte-compile-warnings: (not unknown-keyword)
> to the package's local variables section?

So to some file written by someone somewhere somewhen, perhaps
quite some time ago, you say: "All ya gotta do is jump through
this leetle hoop here."

I say, "All ya gotta do is stay away from `Keywords', which has
been around forever.  Just go and roll yer own!  Problem solved."

Ya name it `Package Keywords' or `Kate's Categories' or whatever.
End of story - poof, no problem.

Just don't try to hijack `Keywords', which Emacs has been using
for a long long time without additional rules, and stomp on its
users with a fancy set of patterns and relations that fit your
shiny new app (`package.el').  There's no call for that, no need.

It's pretty simple really.  Create your own sandbox.

(I don't mean you personally, of course.  I hope you get the idea.)

> > That's why I suggest that you start out with a new, fresh field, which
> > you announce as pretty much dedicated to package.el and which will be
> > controlled by it.
> 
> I don't see why `Keywords' shouldn't be appropriate for elisp programs
> installable via package.el.

It could be, if package.el behaved and played well with others.

But if it comes storming into the sandbox and cries that everyone
needs to play by the rules of a new game it's invented, that's
not fair.  It is only because it menaces now to start shoving
others around, tossing sand in faces, that one is inclined to
suggest politely, "Please go play over there, nicely, in your
own sandbox, if you can't get along here."

> It's not that it re-defines its meaning.

Sure it is.  If it is starting to say what's a duplicate and
what's not.  And it's even trying to "prevent" duplication (as
it sees it).

It's imposing a new world order on the little sandbox.  And there
is no need - plenty of room for it to play on its own in its own
shiny new sandbox.  It can invent and refine rules to fit its own
uses perfectly, with no noise or interference from anyone else.

> The benefit of using it is that it's already there most of the
> time.  

Ay, but there's the rub.  Instead of building (and how much
effort does it really take?) its own sandbox, it'd rather usurp
the community one in Old Town.  Step in, set some rules, issue
some warnings if they're not respected...  There's a new sheriff
in town - move along.

> A new field would take some time to get picked up.

Oh, please.  You underestimate the power of Emacs Dev.
We're already hearing that users should be told to no
longer put `require' in their init files.

`package.el' has already taken over most of the playground.
It won't be a big deal for it to build a new sandbox in one
corner that lots of kids will want to play in - especially
the new kids.  It won't be playing alone for long, trust me.

> And I'm pretty sure you won't see any occurrence of
>   ;; Keywords: modeline, electronic mail
>   ;; Package Keywords: mode-line, email
> even if there was a new field.

Sorry, I didn't catch your last point.  Perhaps you mean
to say that `Keywords' will disappear?  All the kids will
run to play in the new sandbox?

Maybe so.  But the old one will pass with a whimper in that
case, not a bang.  No sand in faces. No confusion in the
meantime, no adapting, no messy cleanup.  Peaceful coexistence.

I'd even go so far as to suggest that *each* file-header
keyword that package.el wants to use, and possibly give
special some semantics (order) to, should use the same prefix:
`Package Version', `Package Requires', `Package Keywords',...

Why not?

No confusion, complete control over its own needs and wishes.
Clean and simple.  No wrestling with what might have been
used here or there for `Version' or `Keywords' or...  Easily
recognizable for what it is: something to do with, uh, packages.

Try it.  I'll bet you a pint of whatever that if you create
the `Package Keywords' sandbox today, write up its Robert's
Rules of Order, and implement whatever enforcements or
reminders/warnings you think might be helpful...  By the
time you can say `Kate's Categories' it will be the most
popular act on the file-header circuit.  Better mousetrap,
beaten path.

(Oh, it's `Package-Requires', not `Package Requires'?
No special need for that hyphen, but so be it:
`Package-Version', `Package-Keywords',...)

Re: Improving browsing and discoverability in the Packages Menu

[Tassilo Horn]

Drew Adams <drew.adams@oracle.com> writes:

>> If it were intended by the package author, where is the problem of
>> adding
>>   ;; byte-compile-warnings: (not unknown-keyword)
>> to the package's local variables section?
>
> So to some file written by someone somewhere somewhen, perhaps quite
> some time ago, you say: "All ya gotta do is jump through this leetle
> hoop here."

Yes, exactly.  Because if it is available via ELPA, MELPA, or some other
package archive, then it seems to be actively maintained.  And in the
very unlikely case that the author/maintainer insists on "modeline"
instead of "mode-line", she can jump through the little hoop.

BTW, an `unknown-keyword' warning was not really what I meant in my
previous reply.  Of course, package.el cannot know all sensible
keywords, so it should work with any keyword.  But in the case of
keywords that are likely just synonyms or typos of some commonly used
keyword (modeline vs. mode-line), it could suggest the commonly used one
for achieving a bit standardization.

>> And I'm pretty sure you won't see any occurrence of
>>   ;; Keywords: modeline, electronic mail
>>   ;; Package Keywords: mode-line, email
>> even if there was a new field.
>
> Sorry, I didn't catch your last point.  Perhaps you mean to say that
> `Keywords' will disappear?  All the kids will run to play in the new
> sandbox?

No, what I've meant to say is that when a maintainer adds the new
`Package Keywords' header duplicating the existing `Keywords' header and
then sees that byte-compilation suggests to use the more commonly used
"mode-line" and "email" instead of "modeline" and "electronic mail", why
should she change only `Package Keywords` accordingly and not
`Keywords', too?  (Assuming that she agrees that "mode-line" and "email"
express what she intended to express, too.)

She doesn't know that there are some people that don't want `Keywords'
to be changed.  Or should the warning say something like that?

    The Package Keywords entry "modeline" is likely to mean "mode-line".
    If you agree, change it accordingly but please let the Keywords
    header as-is for the Olt Town's habitants peace of mind.

Bye,
Tassilo

Re: Improving browsing and discoverability in the Packages Menu

[Eric Abrahamsen]

Tassilo Horn <tsdh@gnu.org> writes:

> Drew Adams <drew.adams@oracle.com> writes:
>
>>> If it were intended by the package author, where is the problem of
>>> adding
>>>   ;; byte-compile-warnings: (not unknown-keyword)
>>> to the package's local variables section?
>>
>> So to some file written by someone somewhere somewhen, perhaps quite
>> some time ago, you say: "All ya gotta do is jump through this leetle
>> hoop here."
>
> Yes, exactly.  Because if it is available via ELPA, MELPA, or some other
> package archive, then it seems to be actively maintained.  And in the
> very unlikely case that the author/maintainer insists on "modeline"
> instead of "mode-line", she can jump through the little hoop.
>
> BTW, an `unknown-keyword' warning was not really what I meant in my
> previous reply.  Of course, package.el cannot know all sensible
> keywords, so it should work with any keyword.  But in the case of
> keywords that are likely just synonyms or typos of some commonly used
> keyword (modeline vs. mode-line), it could suggest the commonly used one
> for achieving a bit standardization.
>
>>> And I'm pretty sure you won't see any occurrence of
>>>   ;; Keywords: modeline, electronic mail
>>>   ;; Package Keywords: mode-line, email
>>> even if there was a new field.
>>
>> Sorry, I didn't catch your last point.  Perhaps you mean to say that
>> `Keywords' will disappear?  All the kids will run to play in the new
>> sandbox?
>
> No, what I've meant to say is that when a maintainer adds the new
> `Package Keywords' header duplicating the existing `Keywords' header and
> then sees that byte-compilation suggests to use the more commonly used
> "mode-line" and "email" instead of "modeline" and "electronic mail", why
> should she change only `Package Keywords` accordingly and not
> `Keywords', too?  (Assuming that she agrees that "mode-line" and "email"
> express what she intended to express, too.)

If we go this route I humbly suggest Package Category (or just Category)
as an alternative. It's not as immediately confusable with "keyword",
and I think semantically speaking pairs with it quite nicely.

RE: Improving browsing and discoverability in the Packages Menu

[Drew Adams]

> >> And I'm pretty sure you won't see any occurrence of
> >>   ;; Keywords: modeline, electronic mail
> >>   ;; Package Keywords: mode-line, email
> >> even if there was a new field.
> >
> > Sorry, I didn't catch your last point.  Perhaps you mean to say that
> > `Keywords' will disappear?  All the kids will run to play in the new
> > sandbox?
> 
> No, what I've meant to say is that when a maintainer adds the new
> `Package Keywords' header duplicating the existing `Keywords' header and
> then sees that byte-compilation suggests to use the more commonly used
> "mode-line" and "email" instead of "modeline" and "electronic mail", why
> should she change only `Package Keywords` accordingly and not
> `Keywords', too?  (Assuming that she agrees that "mode-line" and "email"
> express what she intended to express, too.)

If package.el continues to issue warnings for stuff that is in `Keywords'
(not `Package Keywords'), then it is not playing fair.  In that case it
would have created its own sandbox (good) but continue to poke its nose
into the old community sandbox, kick sand around, and annoy its occupants
with bullying warnings and such (bad).

If not, then there should be no problem.  Keywords inserted in `Package
Keywords' should obey the new rules - or else warnings & helpful
electroshock will duly ensue, per spec.

If someone blindly copies existing non-package keywords from `Keywords'
to `Package Keywords', and if some of those flaunt the clean, Streng
Verboten rules of package.el, then the copier gets what s?he deserves:
appropriate insults (er, guidance) from package.el.

> She doesn't know that there are some people that don't want `Keywords'
> to be changed.  Or should the warning say something like that?

Yes, `Keywords' is not for package keywords.  That should be part of 
the message for users (the doc).  `Package Keywords', and only `Package
Keywords', is for package keywords, which have (whatever) strict
semantics.

Not a big deal.  Clean, tidy.  Should make parsing and controlling
by package.el easy, without stepping on any toes - no sand in eyes.

`Keywords' need not (& should not) be mentioned in the package doc.
But if it is, just say that it is lax - no special semantics.  The
message should be only that keywords related to packages belong in
`Package Keywords' - that's all.

There is no mention of `Keywords' in the current Emacs manual, and no
need to add any mention of it (IMO).  Search case-sensitively for
`Keywords' in the Emacs manual and - guess what? - you find hits only
for (drum roll...) `Package Keywords'.  Seems that Ms Manual has
already paved the way for my suggestion.

Oh, but the Elisp manual mentions `Keywords' in its discussion of
packages.  What it says there (node `Simple Packages') is only
specific to use of `Keywords' by package.el.  This text should
refer instead to `Package Keywords', of course.

The Elisp manual mentions `Keywords' also in node `Library Headers'.
And (guess what?) there the text mentions the use of `Keywords' by
`finder-by-keyword' - which is NOT and should NOT be limited to
packages as defined by package.el.  That is all as it should be.

Unfortunately, the description of `Keywords' in node `Library
Headers' has been updated to mention that it is now used also by
package.el.  That is inappropriate and should be cleaned up.
Instead of saying this:

  The name of this field is unfortunate, since people often assume
  it is the place to write arbitrary keywords that describe their
  package, rather than just the relevant Finder keywords.

The correct approach here is to use a new, NOT-unfortunate name for
package keywords, and continue to let `finder-by-keyword' and any
other longstanding code use `Keywords' as originally intended (no
special rules, syntax, or relations).

Just as finder.el is not (& should not be made to be) limited to
what is good for package.el (both `finder-known-keywords' and
`finder-list-keywords' predate package.el), so should file-header
keyword `Keywords' not be limited to package.el relevance.

>     The Package Keywords entry "modeline" is likely to mean "mode-line".
>     If you agree, change it accordingly but please let the Keywords
>     header as-is for the Olt Town's habitants peace of mind.

I'd say just mention what `Package Keywords' is for in the doc, and
not repeat the doc in each warning about package keywords.  But you
know better than I what warnings people should be scared with. ;-)